Routed Continual Learning: Letting AI Agents Call Up Memories on Demand

AI agents deployed at scale repeatedly encounter similar problems during continuous operation. Letting them learn from each other's experiences rather than starting from scratch every time is a natural efficiency optimization. But the difficulty lies in this: no single way of retaining memory fits all scenarios, and it is impossible to predict which piece of experience will be useful for which future problem. A global history record retains all details, but it incurs high storage and invocation costs and mixes potentially irrelevant experiences together; compressing history into a single fixed representation controls cost but may lose critical details.

To address this tension, researchers propose Routed Continual Learning (Routed CL), an architecture that decides how to partition and retain experience. Its core idea is to treat continual learning as a routing problem—selecting not only the learning mechanism but also the state cluster, rather than committing globally to a single representation.

Core Mechanism: Clustering and LLM Routing

Routed CL groups past trajectories into multiple clusters, each maintained by a memory system drawn from a fixed library of memory systems. Before each run, an LLM-based router decides—based on the problem statement, memory system descriptions, and cluster summaries—whether to reuse an existing cluster or create a new one.

The key to this design is that it does not presuppose a single memory scheme. The memory system library can contain memory mechanisms with different characteristics, and the router dynamically matches the most suitable memory approach to the current task requirements, thereby striking a balance between preserving detail and controlling cost.

The Bridging Role of the Context Memory Adapter

The architecture also introduces a Context Memory Adapter, which mediates between the memory system and the agent loop. This adapter is responsible for converting the output of the memory system into context information usable by the agent, ensuring that different memory mechanisms can seamlessly plug into the agent's decision-making process.

Continual Learning is the branch of machine learning that studies how agents can learn new tasks sequentially without forgetting old knowledge. Its core challenge is "catastrophic forgetting": after a model updates its parameters on a new task, it overwrites the representations of old tasks. Traditional continual learning methods (such as regularization, replay buffers, and parameter isolation) are mostly designed for model weights, whereas this article discusses continual learning at the context level—the agent's historical experience is stored as text in external memory or context, without involving gradient updates. This distinction is crucial: it means that "forgetting" manifests as context being truncated or compressed, leading to loss of detail, while "learning" manifests as reinjecting useful experience into the prompt. Routed CL operates precisely in this non-parametric setting, using retrieval and routing in place of parameter consolidation, so its cost comes mainly from context length and API calls rather than training compute.

In-Context Learning (ICL) refers to a mechanism in which the model completes a task without modifying its parameters, relying solely on examples or historical trajectories provided in the prompt. In agent scenarios, ICL history typically means concatenating complete previous successful or failed trajectories into the context so the model can imitate or avoid them. Its advantages are simple implementation, no training required, and preservation of original details; its disadvantages are a limited context window, cost that grows linearly with history, and irrelevant experiences that dilute attention and introduce noise. The Global ICL baseline in this article stuffs all history into the context, hence the highest cost; the "ICL-only clustering method" first clusters history and takes only the raw trajectories of relevant clusters, serving as an intermediate comparison. Understanding the cost–information trade-offs of these three ways of constructing context is a prerequisite for understanding where Routed CL's gains come from.

Experimental Design and Key Data

The research team evaluated the architecture on a heterogeneous task stream that interleaves 301 instances from six CL-Bench tasks with 97 banking-domain instances from τ³-Bench. The experiments covered four language models and conducted five matched rollouts.

The results show that, compared with a single global in-context learning (ICL) history, Routed Continual Learning improves normalized reward by 15-30% of the maximum achievable normalized reward. At the same time, its average normalized reward is higher than that of the ICL-only clustering method, and it reduces estimated API cost by 65-72% relative to Global ICL.

These data indicate that routed continual learning delivers measurable improvements on both effectiveness and cost dimensions.

Why Routing Beats Global Representation

Traditional methods either retain all history (expensive and mixed) or compress it into a single representation (cheap but lossy). The insight of Routed CL is that different problems may require experience at different granularities. Through clustering, each cluster can focus on a specific type of experience, while the LLM router acts as a "dispatcher" that selects the most relevant experience cluster based on problem characteristics.

This approach avoids forcing a unified global representation and also avoids the cost inflation brought by full history. The magnitude of API cost reduction in the experiments shows that calling memory systems on demand can significantly reduce unnecessary context length.

Implications for the Continual Learning Paradigm

The significance of this work lies in redefining continual learning as a routing problem, rather than merely a problem of memory compression or expansion. It suggests that an agent's experience management can be dynamically allocated according to the task, much like scheduling computational resources.

Of course, the method also depends on the judgment accuracy of the LLM router and the design quality of the memory system library. How to automatically expand the memory system library and how to handle transfer learning between clusters remain directions worth exploring. But based on current results, Routed CL offers a new path for experience reuse in large-scale agents that balances both effectiveness and cost.

One-line Summary

Routed CL reframes continual learning as a routing problem, selecting memory systems and experience clusters on demand while balancing effectiveness and cost.

Paragraph Summary

The article introduces Routed Continual Learning (Routed CL), an experience management architecture for long-running AI agents. To address the tension between "retaining global history is costly and mixed" and "compressing into a single representation loses detail," the method treats continual learning as a routing problem: first cluster past trajectories into multiple state clusters, each bound to one memory mechanism from a fixed memory system library; before each run, an LLM-based router decides—based on the problem statement, memory system descriptions, and cluster summaries—whether to reuse an existing cluster or create a new one; a Context Memory Adapter bridges memory outputs into the agent loop. On a heterogeneous task stream containing 301 CL-Bench instances and 97 banking instances, across four language models and five matched rollouts, normalized reward improves by 15-30% over global ICL, average reward is higher than the ICL-only clustering method, and API cost is reduced by 65-72%. The authors note that the paradigm still depends on router judgment and memory library design, and that inter-cluster transfer and automatic library expansion are future directions.