Vendor benchmarks recalculated: where vendor scores sit above independent measurements

The site whatmodel.sh reports a pattern in model cards: vendors publish benchmark scores for their own models that are 6.4 percent higher on average than those from independent testers. Behind it is an open dataset of 611 pairs, each the same model on the same test, vendor score against independent measurement. We reran the calculation on the raw data, read two extreme examples in the original documents, and checked what the dataset offers for our own production model.

A vendor benchmark is a test score that a model's maker measures itself and publishes in a model card or report. It can only be compared with other scores if reasoning level, agent scaffold, task subset, attempts per task and test version are stated alongside. An independent measurement is the same test run by a body with no model of its own in the race, or by another lab.

What the dataset contains

whatmodel.sh is a catalogue of models that run on your own hardware, operated by Merlinsbeard Pty Ltd. The code is MIT-licensed, the numbers fall under CC BY-SA 4.0, logos excluded. Anyone who redistributes parts of the dataset has to do so under the same licence. The snapshot we loaded (commit 46c9a6f of 16 October 2026) holds 22,192 individual results for 1,891 models: 7,687 from the vendor and 14,505 from independent sources such as leaderboards, papers, Epoch AI or another lab's test report. Each result carries a source, a date and usually a note on the setup.

The project is young. The repository was created on 28 September 2026 and on 5 November 2026 had 46 stars, 183 commits and one open pull request, but no open issue. 169 of the 183 commits come from a single account, with a few contributions from others. The curation therefore rests on very few hands, which is why verifiability pair by pair is what counts: each pair points to two documents.

The middle is small, the spread is large

Our script takes, per model and test, the median of the vendor scores against the median of the independent scores and counts only accuracy tests where both sides exist. We exclude tests that report error rates such as WER and CER. The site shows 611 pairs, our filter gives 623. We did not find out why the counts differ by twelve.

Relative gap = (vendor score minus independent score) divided by the independent score; points are on each test's own scale. Both rows analyse the same dataset. Neither is a new measurement of the models.

There is little at the centre and a lot in the spread. In our calculation a quarter of the pairs sit more than 8.1 percent above the independent measurement, 22 percent give the vendor a lead of more than 10 percent, and 9 percent more than 25 percent. In 10 percent of pairs the vendor score is more than 10 percent too low. The mean lives off the outliers, and those come further down.

The gap sits where the setup matters

The test decides how large the gap is. Fixed multiple-choice and completion tests barely move: MMLU 0.0 points median, HumanEval 0.0, WinoGrande -0.1 and ARC Challenge -0.6, measured over 20, 34, 25 and 14 pairs. Where the protocol is part of the result, it looks different: SWE-bench Verified +9.0 points median over 13 pairs with a higher vendor score in 92 percent of them, MBPP +9.0, ARC Easy +4.7, AIME 2024 +3.8, GSM8K +3.3. whatmodel.sh names the agent harness on SWE-bench, sampling and reasoning budget on AIME, and few-shot prompting on GSM8K and MBPP as the causes. Agent tests and math with working steps matter most for a production decision.

The largest pair on the list, gpt-oss-120b on SWE-bench Verified, we read in both documents. In its gpt-oss technical report (arXiv 2508.10925, 8 August 2025), Table 3 gives OpenAI's numbers for gpt-oss-120b: 47.9 percent at low, 52.6 at medium and 62.4 at high reasoning level, a 14.5-point spread inside one document for one model. According to section 5.2.3.1, all SWE-bench runs in the report use a fixed subset of 477 tasks that OpenAI validated on its own infrastructure, with pass@1 as the metric. The four attempts per task mentioned there refer explicitly to o3 and o4-mini; for gpt-oss the text gives no such figure at that point. whatmodel.sh books 62.4 percent as the vendor score.

ServiceNow-AI lists the same model at 31 percent in its Apriel 1.6 card. The card says that tests in the Artificial Analysis Index v3.0 use that index's scores and all others were run internally. The text gives no details on the SWE-bench setup; we find neither reasoning level nor scaffold nor measurement date. That makes 31.4 points between two documents about the same model.

The second example points the same way. The DeepSeek-R1-0528 card lists 57.6 for SWE Verified (Resolved) and notes that the Agentless framework was used. In the Apriel table DeepSeek R1 0528 stands at 29.6 percent. Two protocols, one model, a factor of two.

Against neutral testers the gap shrinks

The independent side of the dataset is mixed: cards from other labs and measurements from neutral bodies. According to whatmodel.sh, the vendor score is higher in 63 percent of 446 pairs against rival labs' release documents and in 56 percent of 236 pairs against neutral leaderboards, Epoch AI and papers that release no model of their own.

The most telling cut is in the same section: where a rival and a neutral body measured the same model on the same test (272 cases), the rival's value was lower 131 times and higher 130 times, median 0.0. Rival labs therefore do not systematically score other models lower than neutral bodies do. What remains is the gap between a lab's own publication and any outside measurement.

The counterexamples are in the same dataset. Among vendor groups with at least ten pairs, BigScience shows the largest negative swing: median -11.9 points, with the vendor score higher in 10 percent of its 20 pairs. At DeepSeek two neighbouring models show both directions. For DeepSeek V4 Flash, the card of 10 September 2026 lists 89.9 percent on GPQA Diamond, while an Epoch AI run of 2 August 2026 gives 91.0 percent. For DeepSeek V4.1 Flash, the card has 90.9 percent against 89.8 percent in an Epoch run of 9 October 2026. The deviation is small in both cases and the sign flips. Epoch AI's data is published, according to its data hub, under a Creative Commons Attribution license, with benchmark questions and answers remaining the property of their creators; rows from third-party sources keep their own licences.

whatmodel.sh names two limits itself: the settings differ (reasoning effort, agent harness, few-shot prompts, subset or test version), and labs choose which tests to publish. This is not a random sample. On selective publication, see The Leaderboard Illusion (Singh et al., submitted 29 April 2025, current version 12 May 2025). That paper studies Chatbot Arena; benchmark scores in model cards are not its subject.

For our own model the comparison is missing entirely

We run Qwen3.8-Flash-Next on four RTX 5090s. In the 14 days up to 20 October 2026 it handled 62,458 requests and 3.23 billion tokens, counted in the spend log of our LiteLLM gateway (our own measurement). The dataset holds eight vendor scores for the model, among them GPQA Diamond 91.7, LiveCodeBench v6 91.9 and Humanity's Last Exam 35.9, plus six independent results. The independent ones come from arena aggregates, a document-parsing test, a clinical text test and a Roblox quiz. None of these tests matches a vendor score. Across 22,192 results there is not a single pair for our production model.

That is the normal case. Of 776 models with at least one vendor score, 212 have no independent result anywhere (our count on the same commit), so almost one in three models with numbers in circulation has only the maker's figure. Independent runs concentrate at the top and the broad middle goes unchecked.

Which number can carry a model choice

Three steps remain from the calculation, and they are less convenient than a table comparison. First, write down the protocol next to every number: reasoning level, scaffold, task subset, attempts per task and test version. Without it the number is a claim without a measurement. Second, look up the same test at a neutral body, such as Epoch AI, a leaderboard with its own runner, or a paper that releases no model. That does not replace your own measurement, because its setup rarely matches the shape of your requests. Third, run 30 to 50 tasks of your own, taken from the traffic that will later go through the lane. This is our working rule; the figure of 30 to 50 comes from experience and is not backed by literature.

Two of our own posts show what that looks like: the comparison of a speed estimate with our measurement on two cards puts estimate and measurement side by side, and the rebuild of our production lane contains the measurements on the four RTX 5090s. If you are planning a model setup of your own, our LLM operation on your own hardware offer is the way in, and the topic AI in production collects more posts like these.

A vendor score on SWE-bench Verified describes the vendor's scaffold; it does not show the model's ceiling. Anyone who runs models locally is as exposed to model cards as someone buying API access, only with their own hardware at stake.

What changed in this post

As of 5 November 2026 we rechecked the numbers against the site and the raw data. We corrected that the 0.7 points is the median gap, not a mean absolute gap (that one is 5.2 points). The gpt-oss protocol is now assigned to the right section of the report. Stars and commits are brought up to date. We removed the split into open and closed weights and a Reddit thread as a source, because we could not verify either.

Further sources

The analysis and its limits are on whatmodel.sh/claims, the raw data and licence in the repository. The gpt-oss protocol is in arXiv 2508.10925, Table 3 and section 5.2.3.1; ServiceNow-AI's comparison figures are in the Apriel 1.6 card at the pinned commit, DeepSeek's own scores in the R1-0528 card. Epoch AI publishes its benchmark data hub under a Creative Commons Attribution license (the page names no version); per the same page, benchmark questions and answers belong to their creators. On selective publication in rankings: Singh et al., The Leaderboard Illusion. The dataset figures are the operator's statements or our analysis of its raw data, not our measurement of the models.