ReTrace: Distilling Tool-Calling Agents into Planner-Free Workflows

Business process automation requires repeatedly executing the same standard operating procedure (SOP) on new inputs. LLM agents can act according to SOPs written in natural language, but every run and every step requires calling the LLM to make decisions, which significantly increases latency and cost at scale. ReTrace's approach is to transform an SOP and a small number of recorded agent trajectories into a reusable workflow—where code, not the LLM, selects the next step.

Core Problem: "Thinking It Through" on Every Run

The cost of using an LLM agent to execute an SOP lies in "re-reasoning at every step." For high-frequency, repetitive business processes, this pattern is not economical: each run pays inference costs for the same decisions, and latency accumulates with the number of steps.

ReTrace focuses on an overlooked signal—the agent's reasoning process. The explanations an agent writes at each tool call often reveal why it took that action; the tool call sequence alone does not show the conditions. ReTrace extracts the conditions behind each step precisely from these reasoning texts.

The inference cost structure of LLM agents: In mainstream agent paradigms such as ReAct, each decision step includes a full LLM forward pass, and the input context grows linearly with the number of steps (historical tool calls and observations are continually appended), so the per-step cost is not constant but rises progressively. For an SOP with N steps, the total inference cost grows roughly superlinearly, and latency accumulates accordingly. This makes the per-task cost of agents in high-frequency business processes far higher than traditional scripted automation, and it is also the economic motivation behind ReTrace's proposal to separate the planning phase from the execution phase.

Method: Extract Conditions from Reasoning Texts and Compile Them into Code Workflows

ReTrace's process can be summarized in three steps:

1. Extract conditions: From the agent's reasoning and tool call records, extract the judgment conditions behind each step.
2. Merge into a rule document: Merge the conditions extracted from multiple runs into a compact rule document.
3. Generate an executable workflow: Generate a workflow from the rule document, with code deciding the next step. Only this last step depends on a specific platform, so the rule document can be compiled separately for different orchestration platforms.

This layered design gives the "distilled" process a degree of platform portability rather than binding it to a single agent framework.

τ²-Bench and SOP-Bench: τ²-Bench is an evaluation benchmark for tool-calling agents, typically simulating two-party (user and agent) dialogue scenarios such as customer service, requiring the agent to call databases and APIs to complete tasks while complying with business rules, and judging success by whether the final state satisfies the goal. SOP-Bench, on the other hand, focuses on business processes constrained by standard operating procedures, with many procedures and clear rules, making it suitable for measuring the performance difference of workflow methods relative to few-shot generation and the original agent. What both have in common is that their tasks have deterministic evaluation criteria, making them suitable for comparing three types of metrics: completion rate, per-task cost, and latency.

Results: Completion Rates Close to Agents, Lower Cost and Latency

According to the paper's report, on SOPs from a production platform, ReTrace workflows have a slightly lower completion rate than the original agent, but the cost per task is significantly lower. On τ²-Bench, the task success rate is likewise slightly lower than the agent's, but runtime is reduced from second-scale substantial waits to within a second; after three rounds of repair on failed training samples, the k=2 workflow shows significant improvement.

On several procedures in SOP-Bench, un-repaired ReTrace is slightly inferior to few-shot generation, but after repair using labeled training failures, the success rate surpasses it, and the per-task time is far lower than the agent's. These results point to the same conclusion: delegating planning to code can significantly reduce inference overhead while retaining most of the success rate.

Few-shot generation baseline: Few-shot generation refers to the practice of directly placing a small number of examples (SOP fragments or historical trajectories) into the prompt and letting the LLM produce a workflow or code in one go. It does not require additional condition extraction and rule merging, is simple to implement, but lacks systematic induction of the implicit conditions across multiple runs, and is prone to omissions or inconsistencies at boundary conditions. ReTrace's slight inferiority to this baseline when un-repaired indicates that condition extraction and rule merging themselves introduce information loss; the fact that it surpasses the baseline after repair using labeled training failure samples shows that this loss can be compensated for through supervised iterative correction.

Implications: A Pragmatic Path for Agent Distillation

ReTrace's value is not in making agents disappear entirely, but in re-dividing responsibilities—the LLM is responsible for "teaching" the process, and code is responsible for "executing" it. For business scenarios with stable SOPs and frequent runs, this planner-free workflow form is closer to production requirements than calling the LLM every time.

It also suggests a more general observation: an agent's reasoning text is not just an explanation for humans to read, but also structured knowledge that can be extracted, compressed, and solidified into rules. How to stably extract conditions and how to handle condition conflicts across multiple runs remain engineering problems that such methods must face when being deployed.

One-line Summary

ReTrace extracts conditions from agent reasoning texts and compiles SOPs into code-driven workflows to reduce inference costs.

Paragraph Summary

ReTrace addresses the problem that when LLM agents execute standard operating procedures (SOPs), every step requires calling the LLM for inference, causing cost and latency to rise with scale. It proposes transforming an SOP and a small number of recorded agent trajectories into a reusable workflow, where code rather than the LLM decides the next step. Its key signal is the agent's reasoning text: it reveals the judgment conditions behind tool calls, which the call sequence itself does not contain. The method has three steps: extract conditions, merge into a rule document, and generate an executable workflow. Only the last step depends on a specific orchestration platform, so the rule document has a degree of platform portability. In experiments, ReTrace workflows have a slightly lower completion rate than the original agent, but lower per-task cost and greatly reduced latency; after repair on failed training samples, the success rate surpasses few-shot generation on SOP-Bench. This method is suitable for business scenarios with stable SOPs and frequent runs.