Scheduled Tasks & Cron Scheduling Engine

⏰ 1. Role of Scheduled Tasks in the Project

To support periodic auto-inspection, daily report generation, and multi-platform heartbeat detection, Hermes includes a lightweight Cron scheduler. Code assets are centralized in the cron/ directory:

• cron/jobs.py: Task registrator. All automated tasks (e.g., "CEO Hourly Engine", "Strategy Calibration") define their scheduling parameters and execution periods here using standard 5-field Cron syntax (e.g., */5 * * * * for every 5 minutes).
• cron/scheduler.py: Background core daemon engine. On system startup, it maintains an infinite Tick polling heartbeat in a separate daemon thread.

🔒 2. Background Task & Frontend Interaction state.db Concurrency Conflict Avoidance

Since Cron tasks often run silently in the background, they may encounter **write conflicts** with frontend user queries from TUI or Feishu message gateway events. If both sides write to the session state table in state.db simultaneously, SQLite Busy Locked crashes can occur. The scheduler uses three lock-avoidance strategies:

1. Application-Level Lock Detection: Before updating task execution status to state.db, Cron attempts to acquire the global compression_locks or read the current write transaction.
2. Jitter Retry Noise Reduction: When a write conflict lock is encountered, the Cron thread automatically releases the Python-level lock and sleeps randomly between 20ms - 150ms before retrying, spreading lock timing and reducing contention to near zero.
3. Priority Degradation: If conflicts are severe, Cron temporarily skips non-critical state write-back and directly pushes task results to the gateway, preventing log write-backs from rolling back the LLM's core task.

🩹 3. Multi-Level Fault Tolerance & Log Persistence

Background scheduled tasks typically run unattended, requiring stringent robustness. cron/scheduler.py implements multi-level fault tolerance:

# cron/scheduler.py task scheduling fault tolerance logic
def run_job(self, job):
    try:
        # Step 1: Verify API connectivity and token budget
        self.verify_model_connectivity()
        
        # Step 2: Start task execution
        job.execute()
        
        # Step 3: Update success log
        self.update_job_status(job.id, "SUCCESS")
    except Exception as exc:
        # Step 4: Catch exception, log detailed stack trace
        logger.error(f"Job '{job.name}' failed: {exc}", exc_info=True)
        # Persist error info to state.db
        self.update_job_status(job.id, "FAILED", error_detail=str(exc))
        # Step 5: Send emergency alert notification to admin gateway channel
        self.broadcast_admin_alert(job.name, exc)

Through this self-healing mechanism, even if a Cron task crashes due to network timeout, the entire daemon engine maintains high availability and automatically rebuilds its heartbeat in the next Tick cycle.

🔗 In-Depth Subtopic Analysis

For a deeper understanding of the Cron scheduler, we recommend the following technical deep-dives:

• Scheduled Task Definition & Registration (cron/jobs.py job configuration registration details)
• Background Scheduler Thread Loop & Crash Failover (scheduler.py & DLQ fault tolerance mechanism)