Cron Background Scheduler Thread Loop & Crash Self-Healing
Decrypting cron/scheduler.py multi-thread scheduling loop, task retry, and failover
📊 Figure 8-2-1: Cron scheduled task trigger, multi-thread pool execution, and database state write flow
🔄 1. Background Scheduler Loop
In cron/scheduler.py, the system creates an independent daemon thread for background polling. This thread checks the task list every second, computes which tasks have reached their trigger point, and pushes them into a dedicated batch execution pool.
📊 Figure 8-2-2: Scheduled task timeout detection, crash capture, and dead letter queue (DLQ) retry flow
🩹 2. Exception Retry & Dead Letter Queue (DLQ) Failover
During task execution, API failures may occur due to network timeouts. The scheduler catches such Operational/Request exceptions and performs exponential backoff retries; tasks that fail after multiple retries are sent to a dead letter queue with error logs, never blocking the main thread.