When an Agent Fails, Should We Blame the Model or the Harness?

Introduction: When an Agent Fails, Who Takes the Blame?

When an LLM Agent performs poorly on a benchmark, we habitually blame insufficient model capability. But a study from rss raises a key question that has been overlooked: an Agent's performance is actually the combined result of the model and the execution harness. In other words, when an Agent fails, the responsibility may not rest entirely with the model.

The researchers point out that an LLM Agent is composed of a model and an execution harness, so an Agent's benchmark performance reflects the joint behavior of both. Based on this understanding, they argue that harnesses should be evaluated through how they interact with the underlying model, rather than looking at model scores in isolation. This shift in perspective cannot be ignored for understanding the true capability boundaries of Agent systems.

LLM Agent (large language model agent) refers to a system that uses a large language model as its core reasoning engine and can autonomously plan, call tools, interact with the environment, and complete multi-step tasks. Unlike single-turn question answering, an Agent typically needs to loop through an "observe—think—act" process, so its behavior depends not only on the model's own reasoning and generation capabilities but also heavily on the external execution harness. The term harness is borrowed from software testing, where it originally referred to the shell that wraps the code under test and is responsible for driving execution and capturing results; in the Agent context, it refers to the runtime infrastructure that encapsulates the model, manages context, parses tool calls, and handles error retries and termination conditions. The same model plugged into different harnesses may show vastly different success rates due to differences in prompt templates, tool description formats, and context truncation strategies—this is precisely the fundamental reason why this article expands the unit of evaluation from "model" to "model–harness pairing."

Five Complementary Dimensions: Redefining the Evaluation Framework

To systematically evaluate the interaction between models and harnesses, the researchers developed an evaluation framework with five complementary metrics. These metrics reveal key characteristics of the Agent execution process from different angles.

Valid Completion Rate (VCR)

VCR measures the proportion of times an Agent reliably terminates normally and delivers an evaluable result; it is unrelated to task correctness. The value of this metric lies in distinguishing between "ran to completion but got it wrong" and "didn't even finish." One Agent may give a wrong answer but at least complete the execution flow; another Agent may crash midway due to harness issues and produce no evaluable output at all.

Model-Harness Fit and Cross-Model Fit

Model-Harness Fit measures the degree to which, for a given model, a particular harness supports task achievement through valid completion. Cross-Model Fit is a harness's overall applicability across all evaluated models. These two metrics reveal that harnesses are not one-size-fits-all general-purpose tools—the same harness may perform very differently across different models.

Interaction Residuals

Interaction Residuals capture hidden execution advantages and disadvantages under specific pairings—information that is masked by component-level aggregate data. Simply put, when you look only at a model's total score or a harness's total score, problems specific to certain pairings may be completely invisible.

Residual in statistics and machine learning usually refers to the difference between observed values and model predictions—the part the model fails to explain. Interaction residuals borrow this idea: if an Agent's performance on a task is decomposed into a "model main effect," a "harness main effect," and a "two-way interaction effect," then the interaction residual is the deviation unique to a specific model–harness pairing that remains after removing the first two. Under the two-way ANOVA framework, a significant interaction effect means that the effect of one factor depends on the level of the other—which is exactly the statistical expression of "harness rankings vary by model." The value of Interaction Residuals is that it separates this pairing-specific advantage or disadvantage from aggregated average scores, enabling researchers to identify hidden problems that are smoothed out by "model total scores" or "harness total scores," such as systematic incompatibility between a harness and a specific model's tool-call format.

Token Load

Token Load measures the token demand of Agent execution, including all consumption from failed attempts. This metric is crucial at the economic level—a harness may have a high task completion rate but consume enormous numbers of tokens, making the actual deployment cost far higher than expected.

Token is the basic unit of text that an LLM processes; one token roughly corresponds to half to one word in English and about one to two Chinese characters. Model API billing, context window limits, and inference latency are usually measured in tokens. Token Load matters because an Agent's multi-step execution significantly amplifies consumption: a single task may involve dozens of model calls, each of which must re-send the system prompt, tool definitions, and historical trajectory, and retries after failures accumulate further. Therefore, even if two pairings have the same task success rate, the one with higher Token Load may have a per-task cost several times higher in real deployment. Incorporating Token Load into evaluation essentially places "capability metrics" alongside "economic metrics," avoiding a "paper-optimal" harness that has a beautiful completion rate but unsustainable cost.

Experimental Scale: 111 Settings, 52,168 Trajectories

The scale of the study's experiments is considerable. The researchers evaluated four general-purpose harnesses (OpenCode, Codex CLI, Pi CLI, and DeepAgents) paired with four core models, plus two fixed native model–harness systems, covering six benchmarks and involving capability dimensions such as context management, tool use, long-context planning, multimodal interaction, and artifact handoff.

Including native references and stress conditions, 111 evaluation settings produced a total of 52,168 trajectories. This data volume is sufficient to support in-depth analysis of model–harness interaction patterns.

The experiments yielded several noteworthy findings: harness rankings vary by model and benchmark; task performance and valid completion rate may favor different pairings; and similar task scores may come with vastly different execution profiles. These findings directly challenge the "single harness ranking" approach to evaluation.

Core Finding: No Universal Harness

The experimental data revealed several counterintuitive conclusions.

OpenCode ranks first in cross-model performance, meaning its overall performance across multiple models is the most stable. At the same time, however, DeepSeek^Ad V4 Flash Preview maintained a consistently high average VCR across harnesses on the shared four-benchmark panel. These two "firsts" point to different dimensions—one is a harness's overall adaptability, the other is a model's stability in valid completion.

More critically, harness rankings change depending on the paired model and benchmark. A harness that performs excellently with one model may lose its advantage with another. Pairings with similar task scores may have radically different execution profiles—some may consume fewer tokens, while others may have stronger failure recovery.

These findings support three directional recommendations: evaluate model–harness interactions rather than isolated components; train transferable harness-use capabilities in models; and design harnesses adapted to models and tasks.

Practical Implications: From "Choosing a Model" to "Choosing a Combination"

For teams building Agent systems, this study sends a clear signal: don't just look at model leaderboards, and don't just look at general evaluations of harnesses. What really matters is how your model and your harness perform as a pairing on the tasks you care about.

Specifically, VCR should become an important pre-deployment check—for a pairing that cannot even achieve valid completion, task correctness is out of the question. Token Load needs to be incorporated into the cost model, especially when the Agent requires multiple retries. And Interaction Residuals remind us that component-level aggregate data may mask pairing-specific problems, so validation must be performed for specific combinations.

The researchers also propose a longer-term direction: training transferable harness-use capabilities in models. If models can learn to better leverage the characteristics of different harnesses, the flexibility of harness selection will increase substantially. This may be more valuable than simply pursuing stronger models or better harnesses.

Conclusion

When an Agent fails, who should be blamed? The answer is not either-or. The interaction between the model and the execution harness is the key variable determining Agent performance. The evaluation framework provided by this study—VCR, Model-Harness Fit, Cross-Model Fit, Interaction Residuals, and Token Load—offers tools for systematically understanding this interaction. For Agent developers, this means evaluation strategies need to shift from "choose the best model" to "find the most suitable pairing."

One-line Summary

Agent performance depends on the pairing of model and execution harness, not on a single model's capability.

Paragraph Summary

The article introduces a study on LLM Agent evaluation, pointing out that an Agent's benchmark performance is the combined result of the model and the execution harness and cannot be attributed to the model in isolation. The researchers propose a five-dimensional evaluation framework: Valid Completion Rate (VCR), Model-Harness Fit, Cross-Model Fit, Interaction Residuals, and Token Load, characterizing pairing behavior from the perspectives of reliability, adaptability, hidden bias, and cost. The experiments cover four general-purpose harnesses, four core models, and six benchmarks, totaling 111 settings and 52,168 trajectories. The results show that harness rankings vary with model and benchmark, and there is no universal harness; OpenCode ranks first in cross-model overall performance, while DeepSeek V4 Flash Preview has stable VCR across harnesses. The study recommends evaluating pairings rather than isolated components, training models for transferable harness-use capabilities, and designing harnesses adapted to models and tasks.