CPU-ARC: Reuse LLM Reasoning with Caching, Cutting Costs by Up to 291x

Large language models approach every new problem as if seeing it for the first time, reasoning from scratch—a "reinventing the wheel" approach that is both expensive and inefficient. A paper titled CPU-ARC: Agentic Reasoning Caching for on-CPU Reasoning Reuse proposes a new idea: transform LLM reasoning traces into caches that can be efficiently executed on CPU, thereby reusing existing reasoning when solving new problems and dramatically reducing costs.

The Problem: Why LLMs Keep "Reinventing the Wheel"

The research points out that LLMs often conflate two stages: "task solving" and "solution execution." When faced with a new problem instance—even one highly similar to a previously solved problem—the model still reasons from scratch. This mechanism causes substantial computational resources to be spent on repetitive chains of thought, when in reality many problem-solving patterns are reusable.

More critically, LLM reasoning processes are often uninterpretable, difficult to train continuously, and sensitive to input perturbations. A slight change in how a problem is phrased can lead the model to give a different answer. The researchers argue that if reasoning traces could be "solidified" into executable programs, these pain points could be bypassed.

Chain-of-Thought (CoT) is key background for understanding this problem. CoT refers to a prompting technique that has models explicitly generate intermediate reasoning steps before giving a final answer, introduced by Wei et al. in 2022. It significantly improves LLM performance on math, logic, and multi-step reasoning tasks, but comes with a side effect: every new problem requires regenerating the full reasoning chain. Since autoregressive generation proceeds token by token serially, a long chain of thought means hundreds or even thousands of forward computations, with cost growing linearly with reasoning length. Worse, CoT intermediate steps are unstable—temperature sampling, prompt wording, and subtle changes in context order can all cause reasoning paths to diverge, with final answers drifting accordingly. This is precisely the technical root of what the paper calls "uninterpretable, difficult to train continuously, and sensitive to input perturbations," and the motivation for solidifying reasoning traces into executable programs.

The Method: From Reasoning Traces to Executable Caches

The authors propose "agentic reasoning caching," whose core is to build robust, CPU-executable Agentic Reasoning Caches (ARCs) from LLM reasoning traces. Specifically, ARCs extract the reasoning steps an LLM produces when solving a class of tasks and convert them into a structured representation that can run directly on CPU.

This way, when a new problem instance arrives, the system can directly invoke the ARC to apply the existing solution without waking up the LLM again. The entire process requires minimal model involvement—or even none at all. The paper emphasizes that ARCs not only preserve the logic of the solution but are also fully interpretable—each step corresponds to explicit execution logic.

Agents and tool calling are prerequisites for ARCs to execute on CPU. Here, "agent" does not mean a continuously running LLM, but rather a program structure that decomposes reasoning steps into schedulable operations: each step corresponds to a deterministic function call, a retrieval, an arithmetic operation, or a control-flow branch. When generating reasoning traces, the LLM is essentially implicitly planning these operations; what ARCs do is make this implicit planning explicit, transcribing it into ordinary code or a workflow graph. Since these operations do not rely on neural network forward propagation, a regular CPU can execute them, thereby bypassing GPU inference. This approach aligns with "program-aided prompting" and tool-augmented reasoning, with the difference being that ARCs persist the entire solution path as a reusable asset rather than orchestrating it anew each time.

Results: On-Par Performance with Frontier Models at Drastically Lower Cost

On a variety of difficult math and science tasks, ARCs achieve on-par performance with frontier LLMs. This means that reusing reasoning through caching does not sacrifice accuracy. At the same time, ARCs demonstrate several advantages:

• Fully interpretable: the reasoning process is transparent, with every step traceable;
• Continuously trainable: caches can be continuously optimized with new data;
• Robust to input perturbations: even if problem phrasing changes, the ARC still executes stably;
• Extremely low cost: invoking an ARC is 16 to 291 times cheaper than calling the source model.

This cost advantage stems mainly from ARCs executing on CPU, requiring no GPU inference and avoiding redundant LLM computation. For scenarios requiring frequent handling of similar tasks, this reuse mechanism could bring order-of-magnitude cost reductions.

Prompt caching and KV Cache are important references for understanding the cost comparison. Prompt caching offered by major cloud providers reduces the overhead of repeated input by reusing key-value caches for identical prefixes, but it only accelerates the prefill phase for "identical prefixes"; the model still needs to generate subsequent reasoning tokens for each new problem, and GPU compute consumption remains. ARCs go further: they reuse the execution path of the entire solution, completely skipping token generation. Thus the 16x to 291x cost gap in the paper is not an optimization on the same dimension, but a structural change from "GPU inference" to "CPU execution." The reason the multiplier range is so wide is that the benefit depends on the length of the original task's reasoning chain and the cache hit rate—the longer the reasoning chain and the denser the similar problems, the more significant the savings.

Implications: A New Paradigm for Reasoning Reuse

The CPU-ARC research suggests that LLM reasoning capabilities need not "run in full" every time. By precipitating reasoning traces into executable caches, dependence on model compute can be significantly reduced while maintaining performance. This approach differs from traditional model distillation and prompt caching—it focuses more on "executing solutions" rather than "regenerating solutions."

Of course, the method has its scope of applicability: it depends on task reusability, and for highly novel or loosely structured problems, caches may struggle to cover them. Nonetheless, this research offers a noteworthy technical path for reducing LLM application costs and improving reasoning efficiency. As agent systems become widespread, reasoning caches may become standard infrastructure.

Knowledge distillation is often compared with reasoning caching, but the two have different mechanisms. Distillation has a small model imitate a large model's output distribution, yielding a neural network that still requires forward computation—just with fewer parameters; it compresses the "model," not the "solution." Reasoning caching compresses the "solving process": the solution logic for a class of problems is extracted once and frozen, with subsequent instances executed directly. The two can be combined—using a distilled model to generate initial reasoning traces, then having ARCs solidify and execute them. It is worth noting that the applicable boundary of caching depends on the homogeneity of the task family: math problems often have stable solving templates with high cache hit rates, whereas loosely structured tasks such as open-ended writing and multi-turn dialogue are difficult to parameterize into executable steps, and cache benefits decay rapidly.

One-line Summary

CPU-ARC solidifies LLM reasoning traces into CPU-executable caches, matching frontier model performance while cutting costs by 16x to 291x.

Paragraph Summary

Addressing the inefficiency of LLMs reasoning from scratch on every new problem, the CPU-ARC study proposes "Agentic Reasoning Caching": extracting solution patterns from LLM reasoning traces to build Agentic Reasoning Caches (ARCs) that execute directly on CPU. When new problems arrive, the cache is invoked to apply existing solutions, with little to no need to wake the model. On difficult math and science tasks, ARCs match frontier LLM performance while being fully interpretable, continuously trainable, and robust to input perturbations, with invocation costs 16 to 291 times lower than the source model—mainly thanks to avoiding GPU inference and redundant computation. The work suggests reasoning capabilities need not run in full every time, offering a new paradigm for reducing LLM application costs, though its effectiveness depends on task reusability.