Batch & Cost/Retry

Token Cost Shared Lock Accounting & Rate Limit Backoff Retry

Decrypting batch_runner.py multi-threaded cost lock synchronization & Exponential Backoff algorithm
Retry backoff sequence diagram
📊 Fig 12-2-1: Batch concurrent API error capture, backoff delay calculation & task channel retry sequence diagram

🔒 1. Thread-Safe Multi-Thread Accounting Cost Lock (self._lock)

During batch processing, multiple threads write input/output token counts and billing statistics in parallel. In batch_runner.py, the system equips the multi-threaded accumulation with a shared lock:

with self._lock:
    self.input_tokens += input_t
    self.output_tokens += output_t
    self.total_cost += cost

This ensures that even in high concurrency, the statistical cost data remains bit-perfectly consistent.

Concurrent counter sync
📊 Fig 12-2-2: Batch data slicing, thread pool allocation & task channel concurrency limiting structure diagram

⏳ 2. API Rate Limiting & Jitter Backoff Retry

When the LLM server returns Rate Limit Exceeded (429) or Broken pipe (32) network read/write errors, the batch engine calculates sleep delay using exponential backoff with random jitter, successfully avoiding continuous collisions.