CredentialPool Multi-API Key Health Monitoring & Rotation
Decrypting agent/credential_pool.py failover switching, cooldown avoidance & quota locking algorithms
📊 Figure 2-4-1: Multi-API Credential Probing, Request Distribution & Session Loop Interaction Sequence
🔑 1. CredentialPool Multi-Key Concurrency Isolation Design
To ensure stable operation of multi-tenant gateways and high-concurrency batch requests, a credential connection pool concept is introduced in agent/credential_pool.py:
The credential pool maintains a health queue for each key. When a key receives 401 (Unauthorized) or 403 (Forbidden) from the LLM service, it is immediately demoted and moved to a Cooldown slot, dynamically activating the next valid key to take over.
📊 Figure 2-4-2: External Credential Pool Loading, Dynamic Handshake Confirmation & API Authentication Interaction
🚦 2. Concurrent Rate-Limit Quota Locking
Each key has a corresponding requests-per-minute (RPM) limit. The credential pool maintains a sliding window counter for each key. Keys that exceed their quota are temporarily locked until their window refreshes, completely avoiding 429 errors under high concurrency.