The CPython 3.10 Recursion Trap and Synthetic Recovery in TokenGuard, an LLM Circuit Breaker

TokenGuard is an open-source local circuit-breaker sidecar for LLM agents. Its author published a technical retrospective on Reddit about a recursion crash caused by differences across CPython versions, as well as the "synthetic recovery" mechanism introduced in v0.3.0. This article organizes the technical details from the original material and adds some reflections on LLM agent reliability.

A RecursionError That Only Appears on Python 3.10

The project's CI matrix passed reliably on Python 3.11 and 3.12, but kept crashing on 3.10 due to a RecursionError in ExpiringLRUDict. The team's initial hypothesis was that there was a problem with TTL eviction and sliding-window logic, but the real culprit was a difference in CPython's internal OrderedDict implementation.

In Python 3.10, OrderedDict.pop relies on the Python object protocol and validates membership via __contains__ before deleting a key. Because the custom __contains__ called super().pop() on expired entries, checking an expired key would trigger super().pop(), which would immediately re-enter __contains__ before the node was detached, resulting in an instantaneous infinite recursion loop. In 3.11 and later, CPython rewrote OrderedDict.pop to bypass Python-level dunder methods and directly manipulate internal C nodes, thereby completely masking the bug.

This "new versions mask old-version defects" phenomenon is not uncommon in cross-version compatibility. It reminds us that code that depends on language internals may behave correctly on modern runtimes while still exposing hidden recursion or state problems on older versions.

The Fix: Return to Primitive Operations

The solution was to strip out the high-level pop calls and change all pruning logic to use the primitive super().__delitem__ operation. This change passed all 135 unit and integration tests, and the problem was completely resolved.

OrderedDict and the contains Protocol

OrderedDict is a dict subclass in the collections module that guarantees key insertion order. In Python 3.10 and earlier, part of OrderedDict.pop is implemented in Python code, and before deleting a key it calls __contains__ to check whether the key exists. Because subclasses (such as ExpiringLRUDict) can override __contains__, this check triggers the subclass's custom logic. If the subclass calls super().pop() on an expired entry within __contains__, it re-enters __contains__ before deletion completes, forming recursion. Starting with Python 3.11, CPython changed OrderedDict.pop to directly operate on internal C structures, bypassing Python-level protocols such as __contains__, so the same subclass code no longer triggers recursion. Understanding this difference helps avoid relying on internal call order when writing container subclasses.

The Pain Point of Traditional Circuit Breakers: 429 Kills Execution Directly

When an LLM agent repeatedly calls a tool, a standard proxy typically returns a blunt HTTP 429 status code. But in frameworks like CrewAI or LangGraph, an unhandled 429 immediately terminates the entire execution run, destroying accumulated context and wasting token costs that have already been consumed.

TokenGuard introduced a "Loop Nudge" mode in v0.3.0 to address this failure point. It intercepts the initial loop violation and synthesizes a valid 200 OK assistant response instructing the model to change strategy; only when the agent repeats the exact same violation in a subsequent turn does it escalate to a hard 429 cutoff.

This "guide first, cut off later" strategy essentially introduces a layer of soft feedback into agent behavior management. Compared with returning an error directly, the synthetic response gives the model a chance to self-correct, thereby preserving context and reducing wasted overhead.

Other Capabilities Added in This Version

v0.3.0 also introduced inline DLP scanning to redact API keys before requests leave localhost; added mid-stream token repetition abort for runaway streaming calls; and provided a terminal TUI control center as well as a local web dashboard.

429 Semantics in LLM Agent Frameworks

In traditional HTTP clients, 429 Too Many Requests is usually treated as a retryable transient error, and the caller can wait and retry. But in agent frameworks such as CrewAI and LangGraph, a 429 returned by a tool call is often regarded by the framework as an unrecoverable failure, directly terminating the entire agent run. This is because agent frameworks usually do not have built-in 429 retry policies, and the run state (including conversation history and intermediate reasoning steps) is tightly coupled to a single tool call; once an exception is thrown, the accumulated context is hard to preserve. TokenGuard's Loop Nudge exploits exactly this semantic difference by replacing 429 with a synthetic 200 response at the framework level, giving the agent a chance to adjust its strategy instead of crashing directly.

The Next Step for Multi-Agent Workflows

Currently, multi-agent workflows still require manually configuring environment-variable routing or process wrappers. The author has set the next milestone as building native plug-and-play callback handlers for LangGraph and CrewAI, making session tracking and circuit breaking completely zero-configuration.

The project is fully open source and contains no external telemetry.

Implications for LLM Agent Reliability

TokenGuard's retrospective reveals problems at two levels. The first is how low-level runtime differences can inadvertently cause severe failures; this is exactly where the value of a cross-version test matrix lies. The second is that circuit-breaker design needs to understand the execution semantics of the higher-level framework. A 429 response that is reasonable in a traditional HTTP service may mean context loss and wasted cost in an agent framework. The synthetic recovery mechanism offers a gentler form of intervention and is worth the attention of developers focused on agent reliability.

Project URL: https://github.com/Shvirlok/TokenGuard