Model Tools & Sanitization

JSON Payload Self-Healing & Surrogate Pair Cleaning

Decrypting agent/message_sanitization.py parameter correction, control character escaping & image tag removal details
JSON Self-Healing Flow
📊 Figure 3-2-1: Damaged JSON Payload Self-Healing, Brace Matching & Surrogate Cleaning Flow

🩹 1. JSON Damage Auto-Repair (_repair_tool_call_arguments)

The LLM may produce malformed JSON argument segments due to sudden truncation or framework defects. In the _repair_tool_call_arguments() method of agent/message_sanitization.py, the system provides multi-level defense repair: it calculates the bracket count difference between opening and closing braces/ brackets and automatically appends the corresponding number of } and ] at the end of the string.

Tool Execution and Parameter Sanitization
📊 Figure 3-2-2: Tool Execution Flow, Parameter Sanitization & Error Retry Control Architecture

🔒 2. Surrogate Pair Cleaning & Image Processing

  • Surrogate Pair Cleaning (_sanitize_surrogates): Some models' byte-level output may emit orphaned surrogate pairs (lone surrogates like \ud800 to \udfff) when encountering Chinese or special characters. The system forcibly replaces these, ensuring system stability.
  • Image-Unsupported Degradation (_strip_images_from_messages): When the LLM returns an error indicating image upload is unsupported, the system automatically removes image portions from the message list.