KI2026-10-24· By Franz Senn

Study: the agent harness drives token costs up to 3x, the pass rate barely moves

When people pick a coding agent, they usually compare models. A preprint by Yangze Liu and Zhongyi Han (arXiv 2610.04433, submitted on 3 October 2026) starts somewhere else: it holds the model fixed and swaps the harness. The title gives the answer away: "What Does a Harness Buy? Tokens, Mostly". That matters to anyone who runs agents through a paid API or on their own GPU, because there tokens are the bill.

Claude mark — Claude Code is one of the three harnesses compared and, at about 16,600 preamble tokens per step, the largest. Simple Icons has no logo for OpenCode or mini-SWE-agent, so we show only this one. (Quelle: Simple Icons, file claude.svg (CC0))

This post keeps three kinds of statements apart. Verified means we read it in the preprint or in the authors' repository. Inferred is our own conclusion. Vendor-claimed marks claims made by vendors (here Zhipu). Numbers we only know from The Decoder's coverage and did not see in the paper ourselves are called second-hand. It is a preprint without peer review.

How the measurement is set up

Verified: the authors test five models with three harnesses on SWE-bench Verified: Claude Code, OpenCode and mini-SWE-agent. On a pool of 447 tasks, Claude Code and mini-SWE-agent are at most 5 percentage points apart. The Decoder reports less than 2 points for the two models it covers. The README places the authors at Shandong University; the arXiv page itself lists no affiliation.

Costs, on the other hand, spread out. Verified from abstract and README: with the same model and the same tasks, the price per task differs by up to a factor of 3. The README sums it up this way: what the harness decides is the bill. Cached input tokens shift the absolute level but, the abstract says, do not change the ranking of the harnesses.

Where the tokens come from

Every agent step sends the system prompt and the tool definitions to the model again. The longer that preamble, the more each step costs, no matter how hard the task is.

Simplified. The link between preamble and step count is our inference from the findings, the factor 3 comes from the abstract and README. (Quelle: Liu and Han, arXiv 2610.04433)

Second-hand: the values come from The Decoder's coverage of the preprint. We could not see the figure or table in the paper itself. The bar for mini-SWE-agent is an upper bound (under 1,000). (Quelle: Liu and Han, arXiv 2610.04433 (via The Decoder, 24 October 2026))

A large preamble does not automatically mean high cost, though. According to The Decoder, Deepseek-V4-Flash costs about the same with Claude Code as with mini-SWE-agent, because mini-SWE-agent needs many more steps. The small prompt gets eaten up again by the step count. Inferred: what counts is the product of preamble and number of steps.

For GLM-5.3-Flash the gap is wide, according to The Decoder: 1.81 yuan per run with Claude Code, 0.56 yuan with OpenCode. That is one example from the coverage and no average over all models.

How much noise sits in the benchmark

The second part of the preprint concerns how comparable pass rates are. Verified from abstract and README: on the 45 hard tasks, switching the harness flips the same share of tasks as simply rerunning it, 13 percent. Second-hand (The Decoder): that is the median, switching the model flips 22 percent, and repeated runs of the same setup vary by up to 6 points.

On the 45 hard tasks. The 13 percent for rerun and harness switch are in the abstract and README, the 22 percent for the model switch is second-hand. (Quelle: Liu and Han, arXiv 2610.04433)

For small test sets this has consequences, which the authors also work out: 45 hard tasks detect a gap of about 13 points only half the time. The 447 tasks resolve about 5 points. A lead of 2.4 points, which one vendor (Zhipu) claims for itself, is below both thresholds. That number is vendor-claimed and can be neither confirmed nor refuted with this setup.

OpenCode trails by up to 9 points on the 447-task pool. On one model the authors attribute about half of that to an output-token limit: Claude Code prompts the model to keep going, OpenCode ends the step. The rest is unexplained. The Decoder adds a shorter time limit in the runs with local models, an uncontrolled factor.

Putting it in context

A second paper points the same way: arXiv 2607.22585 by Vats and Golev (8 June 2026, under review at a workshop according to the arXiv comment at submission, later status unchecked) reports that the choice of harness produces up to 40-fold differences in tokens per solved task. The setup differs: Goose, OpenCode and OpenHands SDK, two models, 50 Terminal-Bench Pro tasks, a success-rate difference of 0 to 8 points. The factor is larger, the direction is the same. Both papers are preprints with modest task counts (the second uses 50 tasks), so the factors 3 and 40 are good for an order of magnitude and are not constants.

A caveat on reproducibility: the Liu and Han repository contains a script reproduce.sh that runs in under a minute without a GPU or network. The raw traces of the individual runs are missing so far, according to the README. A reader can retrace the cost ratios, but not the per-step preamble values.

We have no measurements of harness tokens on our own stack for this post, so we put no numbers on it.

How to measure your own harness

The findings suggest a procedure anyone can rebuild with their own proxy or provider log. The steps below are our own suggestion.

Measure the preamble: start each agent tool with an empty or trivial prompt and note the input tokens of the first step. That fixed share comes back in every step. Multiplied by the typical number of steps, it gives the floor per task.
Shrink the preamble: switch off unused MCP servers and tools, tighten your own system prompts, then measure again.
A/A before A/B: run the same task list (10 to 20 tasks are enough for a first look) twice with harness A and once with harness B. Log per task: input, output and cached tokens, steps, pass or fail. If no more tasks flip between A and B than between A and A, the difference in pass rate is noise, and only the bill remains as an argument.
Keep a budget in tokens per task and set a step or token limit for agent runs. A harness that asks "continue?" and one that ends the step behave differently under the same limit.