Compute Layer: Parallel Batch Engine & Concurrency Control

⚡ 1. batch_runner.py Responsibilities & Parallel Architecture

When facing a large number of evaluation tasks or massive auto-push processing, single-threaded sequential execution results in extremely long delays. To address this, the project provides the batch_runner.py parallel execution engine:

• Task Decomposition: Breaks down a single large data stream or task configuration into independent sub-task lists.
• ThreadPoolExecutor Concurrent Pool: The system controls concurrency through concurrent.futures.ThreadPoolExecutor. The pool size is dynamically determined at startup based on host CPU and API rate limits, ensuring concurrency stays within a safe range.

📈 2. Token Metric Accounting & Concurrency Cost Calculation

The hardest challenge in concurrent execution is the centralized statistical accounting of Token usage and costs across various interfaces. batch_runner.py provides thread-safe shared state metering:

# Shared metering logic in batch_runner.py
from dataclasses import dataclass
import threading

@dataclass
class TokenCounter:
    input_tokens: int = 0
    output_tokens: int = 0
    total_cost: float = 0.0
    _lock = threading.Lock()

    def add_usage(self, input_t, output_t, cost):
        # Force concurrent locking to avoid race condition metering vulnerabilities
        with self._lock:
            self.input_tokens += input_t
            self.output_tokens += output_t
            self.total_cost += cost

With thread-level shared locks, even when dozens of sub-agents complete and return simultaneously, the total cost data maintains strong consistency and can be output as a polished JSON/Markdown metrics report.

🛡️ 3. Fault-Tolerant Backoff & SQLite Concurrent Transaction Protection

Under high concurrency, two problems are the most severe: frequent 429 Rate Limit Exceeded errors from LLM APIs, and multiple threads simultaneously writing execution results to SessionDB, triggering SQLite's Busy Locked exception. The system has elegant designs for both:

1. Exponential Backoff with Jitter: For Rate Limit errors, the system uses the formula delay = min(max_delay, backoff_factor * (2 ** attempt)) + random.uniform(0, jitter) for staggered retries, avoiding "synchronous concurrent shockwaves."
2. Database Write Transaction Queuing: When multiple threads update state in the underlying hermes_state.py, the base transaction forces session lock detection with queuing, converting concurrent write transactions into a single-linked list, ensuring data safety without locking the database main process.

🔗 Sub-Chapter Deep Dives

For a deeper understanding of batch engine concurrency, we recommend the following technical sub-topics:

• Large Task Partitioning Algorithm & ThreadPoolExecutor Concurrency Throttling (batch_runner.py task partitioning)
• Shared Cost Accounting Lock & Rate Limit Exponential Backoff Retry (batch_runner.py mutex lock synchronization)