[KongchangAI]
· 2 min read· 1,175 words

Same Model, 9 Coding Agent Frameworks: A 17.5x Cost Gap

Same Model, 9 Coding Agent Frameworks: A 17.5x Cost Gap

Same model across different coding harnesses: pass rates differ by 17 points, but per-task cost by 17.5x.

FrontierHarness plugged the same model into 9 coding agent harnesses and ran the same task batch. Pass rates fell within a 17-percentage-point band, but median cost per solved task differed by 17.5x between highest and lowest. The gap stems not from the model but from harness architecture: context management, tool-call loops, task planning, and failure handling. The article recommends testing median cost per solved task, the cost-vs-pass-rate marginal curve, and task-type sensitivity, and treating context and retry strategies as tunable parameters to control long-term costs.

Coding Agent Harness Benchmark source screenshot

Same Model, Wildly Different Costs

FrontierHarness ran an experiment every developer should look at closely: it plugged the same model into 9 different coding agent harnesses, ran the same batch of coding tasks, and measured pass rate and cost. The results are a bit counterintuitive — pass rates across harnesses fell within a 17-percentage-point band, which isn't outrageous; but the median cost per solved task differed by 17.5x between the highest and lowest.

In other words, what you're choosing isn't the model — it's the shell around the model. And that shell determines how much it costs to get things done.

This conclusion has a direct impact on everyday tool selection. Claude Code, Codex, Pi, and OpenCode are often compared side by side, but most evaluations only compare "can it do it right" and rarely put "how much does one correct run cost" on the table. FrontierHarness's data shows that when pass rates are close, cost is the real dividing line.

Why Pass Rates Don't Spread Much

Task Distribution Sets the Ceiling

The same model running different harnesses landed within a 17-percentage-point band. This tells us two things: first, the model's own capability sets the main ceiling — an outer framework can't make the model capable of things it can't do; second, harness differences show up more in "what path it takes to complete the task" than in "whether it can complete the task."

For developers, this means picking tools purely off a pass-rate leaderboard gives you limited information. The mainstream frameworks may be neck and neck on pass rate, and the real differences hide in the process.

Harness refers to the software layer wrapped around the base model, responsible for managing conversation history, scheduling tool calls, organizing task steps, and handling errors and retries. It doesn't change the model weights, but it determines what the model "sees, when it sees it, and how much." A coding harness typically includes prompt templates, tool interfaces (reading/writing files, executing commands), context window management, and loop control logic. So the same model under different harnesses may receive drastically different input sequences and decision paths — which is the direct source of the differences in pass rate and cost.

What 17 Percentage Points Means

A 17-percentage-point spread isn't negligible, but compared with a 17.5x cost gap, it's a completely different order of magnitude. There's a clear nonlinear relationship between cost and pass rate: chasing those last few points of pass rate may require orders of magnitude more tokens and time. Whether that tradeoff is worth it depends on the scenario — for one-off scripts, pushing toward the ceiling isn't worth it; for high-frequency, long-tail production tasks, spending more to buy pass rate may be reasonable.

Where the Cost Gap Comes From

Harness Architecture Choices

Different harnesses diverge on several key design decisions, each of which directly affects cost:

  • Context management strategy: Whether historical conversations are compressed, truncated, or summarized determines how many tokens each request carries. Some frameworks repeatedly stuff full tool-call records into context, and costs naturally climb.
  • Tool call loop: How many times the model can call tools within one task, how many retry rounds after failure, and whether parallel exploration is allowed — more loops mean higher cost.
  • Task planning approach: Whether to have the model produce a plan first and then execute, or think as it goes. The planning phase itself consumes tokens but may reduce wasted tool calls.
  • Failure handling mechanism: On error, whether to retry the whole thing, repair locally, or just give up — different strategies produce completely different cost curves.

None of these designs is optimal in every scenario. The 17.5x gap is essentially the result of these choices stacking up.

Context compaction is the core mechanism harnesses use to control token consumption. Common approaches include: sliding-window truncation of early conversation, summarizing historical tool-call results before re-injecting them, or keeping only fragments relevant to the current subgoal. The tradeoff across strategies lies between information retention and cost: aggressive compression can significantly reduce tokens per request but may lose key constraints, causing the model to repeat mistakes and increasing loop counts; conservative retention makes every call carry large amounts of redundant history, raising unit cost. Understanding your framework's compression thresholds and retention rules is the first step in judging its cost curve.

Open Source vs. Commercial Frameworks

FrontierHarness's test list includes both commercial products (Claude Code, Codex) and open-source frameworks (Pi, OpenCode). The usual assumption is that commercial products are more efficient, but this benchmark doesn't offer a conclusion broken down by category. What can be confirmed: the cost differences don't come from the model but from the framework layer; so as long as an open-source framework is well-architected, it has just as much chance to keep costs very low — provided the user understands its context and loop strategies.

Median cost per solved task is calculated as follows: only tasks judged as passed are counted, and the median cost of a single task is taken. Unlike "average cost per attempt," it spreads the cost of failed attempts across successful tasks, which is closer to the practical measure of "how much it costs to get one thing done." The median rather than the mean is used to avoid a few extremely expensive tasks skewing the overall conclusion. When reading benchmarks like this, pay attention to whether the denominator is "all tasks" or "solved tasks" — the two are not directly comparable.

What to Test Before Choosing

Don't Just Look at Pass Rate

FrontierHarness's conclusion points to a more practical evaluation framework: beyond pass rate, you should measure at least three things.

Median cost per solved task. Averages are easily skewed by extremes; the median better reflects the norm. The 17.5x gap was measured with this metric.

The marginal curve of cost vs. pass rate. If a framework's pass rate is only slightly higher but its cost is many times greater, ask yourself whether you'll actually use that improvement.

Task-type sensitivity. The benchmark uses a uniform task set, and your real task distribution may be completely different. Long-context refactoring, multi-file edits, and small single-file changes place very different stresses on a harness, so applying general conclusions carries risk.

Treat the Harness as a Tunable Parameter

A further idea: since harness cost differences come from context management and loop strategy, those parameters are worth tuning separately on your own task set. Within the same framework, adjusting retry counts or context compression thresholds can noticeably change the cost curve. Choosing a framework is only the first step — configuration is the long-term lever for saving money.

FrontierHarness's experiment is limited in scale, and conclusions from 9 harnesses can't be directly extrapolated to all tool combinations. But it frames the right question: don't ask which coding agent is strongest — first ask which framework gets the job done on your tasks at an acceptable cost. Same model, 17.5x cost difference. That question deserves a serious test.

#coding agent harness#FrontierHarness benchmark#cost per solved task#AI coding agent comparison#context management
Share:

Related articles