Coding-agent harnesses compared: the same model, 37 points apart

On 9 November 2026 Joel Niklaus's Harness Leaderboard on Hugging Face swapped OpenCode for version 2.2.6, meaning the program around the model, and recomputed the results that OpenCode 1.19.0 had produced. Weighted across all models, OpenCode had sat in eighth place of nine harnesses at 35.5 per cent, with a cost of $3.35 per solved task. After the swap it read 52.0 per cent and $0.45 per solved task, first place, sharing it with four other harnesses whose error bars overlap. The model had not changed in either case.

A harness is the program around the language model. It assembles the prompt, offers tools to the model, executes the calls the model asks for inside a sandbox, feeds the results back, and decides what is still in the context on the next call. The weights stay untouched. The harness determines which tokens the model actually sees, how long a reply may run, which reasoning field goes back upstream, and when history gets summarised. So the comparison measures that shell, while the weights stay identical in every row.

What the public board delivers

The board runs ten harnesses over three open models on tasks from Terminal-Bench 2.1 (89 tasks per the board's configuration) and DeepSWE (113 tasks). The aggregate view I loaded from api/index on 11 November 2026 at 14:44 CET and recomputed by hand holds 32,025 scored evaluations and roughly $4,700 in model spend. The "Solved" column is my own share of solved tasks across both effort levels and both benchmarks; "Cost" is the board's dollars per solved task in the weighted view.

Read as follows. On GLM-5.3-Flash the best harness sits 22.7 points in front of the worst, on DeepSeek-V4.1-Flash 37.4 points, on Flint 16.2 points. On the same GLM model, Pi solves a task for $0.16 while OpenClaw charges $0.84 for the same job, a factor of 5.3. Choosing between two coding agents in front of one in-house model therefore moves the needle more than a routine model change within a generation.

Where the gap comes from

The single largest item is bookkeeping rather than intelligence: does the harness send last reply's reasoning text along again? OpenCode 1.19.0 talked through its built-in OpenAI provider in the Responses format, so the board had to translate those calls across a bridge into Chat Completions. That reasoning provider resends thinking content only together with encrypted_content, and the bridge never produces that field. The board's meters show that for 91 to 94 per cent of OpenCode 1.19.0's call pairs whose reply had reasoned beyond 300 tokens, the prompt grew by less than that reply. For Codex, Pi, Kimi Code, Qwen Code, mini-swe-agent, Hermes and OpenCode 2 the same ratio sits at 0 to 1 per cent. OpenClaw keeps nothing back on its own default for a custom OpenAI-compatible endpoint and lands at 89 to 94 per cent; on the board it runs as shipped.

That the same model scores worse in another shell is one consequence. Money and wall time are the other, and those are less abstract. The Grok Build harness reads only reasoning_content. Against a route that delivers thinking in reasoning and reasoning_details it saw no reasoning, sent nothing back, and its idle timer gave up after 600 seconds of content-free streaming while 7 to 13 megabytes kept arriving. The fx harness did the opposite and forwarded every reasoning_details fragment, roughly 70 bytes per reasoning token: request bodies of 4.4 and 5.6 megabytes, HTTP 413 from the provider, then its own abort message. Their fix is a second route that rewrites incoming thinking fields into reasoning_content; one call fell from 60.8 to 37.5 kilobytes and the connections stopped dying.

This part deserves the honest sentence: the retrofitted resending worked, and the operator wrote at the time that the score gain was not yet distinguishable from the usual spread.

Our own measurement, 11 November 2026

Four harnesses run against our gateway here, which passes requests to a vLLM instance on four cards (vLLM 0.32.0, TP4, measured 11 November 2026). That route answers in the field reasoning_content. For each harness I ran one instruction with exactly one tool call and wrote the request bodies to disk through a proxy of mine. The second request contains the model's first reply, so that is where the thinking has to stand if the model is to receive it again.

Two numbers from the same run belong here. A text of about 250 tokens replayed inside reasoning_content lifts prompt_tokens from 70 to 334; the same text inside content lifts it to 333, so those fields really are serialised into the prompt. Moved into reasoning_details, the same text yields 87, which means that field is ignored on our route. Pi sends reasoning on the board and reasoning_content with us, in each case in the field the text arrived in, so our observation and theirs agree. OMP has a switch for this behaviour in its model files; in version 18.18.7 replay-reasoning-content is rejected as an unknown key at provider and model level, while the rest of the configuration still loads.

How much of this is noise

The board scores the same task three times inside one cell. My recomputation over the ten aggregate cells: repeating a task in the same harness flips the result in 17.6 to 23.6 per cent of cases. The operator's own figures for the board state of 2 November 2026 add: swapping the harness changes the outcome in 29.1 per cent of cases, swapping the model in 33.6 per cent, swapping the effort level in 27.3 per cent. The distance between two neighbouring ranks on the board is usually three to five points and therefore inside the error bars, and the first six places overlap.

Thinking effort is no constant either. On GLM-5.3-Flash every one of the ten harnesses gains when it moves from low to high effort, 5.7 points on average (59.3 versus 53.6 per cent), while cost per solve rises by 20 to 40 per cent. On DeepSeek-V4.1-Flash the same move is worthless: 65.1 against 64.9 per cent on average, with individual harnesses losing up to 5.7 points. The spread a harness swap opens at a fixed model stays wider than the lever of thinking harder.

Counter-check: thinking against added machinery

In November 2026 Surge AI measured four coding-agent setups with Claude Opus 5.5 in Claude Code over 100 of its own software tasks. Vanilla Claude Code at high thinking effort gained 11.3 points, the most effective harness additions gained 2.4 to 3.3 points, and a build with 37 skills and 17 sub-agents cost eight times as much with no clear gain. That measurement comes from a vendor with its own interest, on a different setup (commercial model, different tasks), and it still fits our own observation: what costs money on a real connection are lost fields and needless repetitions. How many agents a single instruction wakes up is the smaller question.

What we took from it

The route we would take on any own connection is short: look first, then change. For us that means the four harness table above is recorded in our notes together with the measurement, and the open question is whether sending reasoning back beats dropping it on our models over twenty multi-step tasks, against a run with reasoning off. Only then would anything be rewritten at the gateway, if at all. Comparing two coding agents with a single pass per task is noise, so run two or three passes and average them. A board like this moves daily, so it is a dated reading rather than a filing number: the values above hold for 11 November 2026, 14:44 CET.

If you want one number to carry away: the largest single improvement the board documents for one harness came from reasoning content and every tool call of a reply actually arriving at the model, and a better model had nothing to do with it. Our posts on measuring tokens and throughput on our own hardware are vLLM prefix caching and vLLM inference; agents and automation in daily operation collect in the AI in production hub. Anyone who wants their own connection measured can do that with us, our IT service for infrastructure and operations.

What these numbers cannot carry

The board is one person's infrastructure, with its own task selection, its own bridge and its own price model built on the list price of the cheapest checked provider per model. The tasks are public (Terminal-Bench 2.1, DeepSWE) and the raw trials sit open in a public bucket of about 848 GB; the scoring scripts are not separately published. Two methodological finds by the operator belong in the balance: FeatureBench left the scoring because its task images held the reference solution and hidden tests that earlier trials had reached, and 21,344 Terminal-Bench trials were re-checked after line numbers in file views had hidden the act of reading the solution. Our side carries the same limit: the four harnesses above are our working versions, the board runs others, and our route is a different router than theirs.

Further Reading

• Harness Leaderboard, live board, state of 11 November 2026, board version 0.1.5
• Harness Leaderboard, machine data, payload of 13:43:58 UTC on 11 November 2026, all percentages recomputed from it
• Space repository with method, changelog and lab notebook, changelog entry "OpenCode 2 replaces OpenCode 1.19.0" of 9 November 2026
• Terminal-Bench 2.1, tasks, commit d49e28f; the count of 89 tasks comes from the board's configuration
• DeepSWE, tasks and grading, 113 tasks
• Surge AI: We Benchmarked Coding Agent Harnesses. Thinking Harder Beat All of Them., November 2026, vendor measurement
• OpenClaw, Pi, OpenCode, mini-swe-agent, Qwen Code, Hermes Agent, MIT or Apache-2.0 licences, state of 11 November 2026