Last week I went through a year of Chinese open weights: the licences that let you run them, the distillation pyramid that produced the small models, the silicon they are shipped to run on. This is the other half of that question, and the harder one. How far behind are the open models, and can anyone measure that honestly?
I have not built, run or audited any of this. I read the leaderboard, its method page and its questions page, and what follows is an assessment of a measurement rather than a measurement of my own.
What the thing is
whatmodel.sh is an open leaderboard built by one developer, who says the reason was simple: he was tired of renting models by the token and wanted to know what he could actually run at home. The claim that caught my eye is the one the site states in its own words: “7 months is the median wait for an open model to match each new closed best — 17 matched, 4 still waiting.”
The data underneath it is the part worth respecting. 22,113 published benchmark results, across 987 benchmarks, from 1,011 sources. For text models alone: 18,755 results covering 1,211 models on 617 benchmarks. Of those text results, 14,436 are independent evaluations — papers whose authors ran the models themselves, plus datasets from Epoch AI, LMArena, EvalPlus and the EleutherAI harness — and 7,677 are scores that labs reported for their own models, labelled as such. The dataset is CC BY-SA 4.0, the scoring code is MIT, and each number links back to where it was published.
Two points on the chart turn the claim into something you can check. o1-preview scored 63.5 in September 2024; DeepSeek R1 scored 63.5 in January 2025, four months later. o1 scored 66.1 in December 2024; Qwen3 235B-A22B Thinking scored 66.1 in July 2025, seven months later. The headline is the median of those waits, and it is visible in the data rather than asserted in a caption.
The method is more interesting than the number
Most leaderboards average percentages, which tells you very little: an easy quiz and a research-level mathematics paper are different rulers, and the mean of those two numbers describes neither. This one uses item response theory instead, the technique that lets exam boards mark wildly different papers on a single scale. In its own notation, result[model, run] ≈ a[test] · score[model] + d[run]: each test contributes a sharpness, each evaluator’s run of that test contributes a level, and a model’s standing is fitted across all of them.
The consequences of that choice are what make it worth reading. A test everybody aces counts for less. A single result only moves the model when its run contains 2 or more models and its test carries 3 or more runs. Choice-style percentages are corrected for guessing first — a quarter of the marks on a four-option question, in effect — and a score pinned at the very top or bottom of a test counts for less, because a ceiling score tells you almost nothing about how much further that model goes. Failed runs stay visible on a model’s page without counting. And thin evidence is handled in the open: with very little data behind it, a model is drawn toward the centre of the scale and given a wider uncertainty band.
And then the mechanism I would point a sceptic at first: results are marked. A star flags a model with fewer than three results behind it; a dagger flags one whose evidence, so far, comes entirely from its own maker. Self-reported numbers are not banned, they are labelled, so you can see for yourself how much of a model’s standing rests on its maker’s own testing.
Why it matters if you own the hardware
If you can only run weights, the lag is your entire decision. It tells you what you give up by choosing the local option — and, just as usefully, what you would be paying an API for.
The practical half of the site is the hardware filter, which works out which downloads fit the memory and the graphics card you actually have, with headroom for the runtime. That is the question a person actually has, and almost no leaderboard answers it. The questions page is also blunt about the trade: “local” does not mean free — you supply the machine and the electricity — and you check the licence for yourself.
The case for it
It is checkable, which almost nothing else in this category is. Every result links to a source, each source’s licence is recorded, and both the dataset and the scoring code are open. The method is a genuine improvement on the genre default. And the caveats are printed where you can read them: the site’s own disclaimer says the figures “can be incomplete, out of date or wrong”, that neither the makers nor the sources have endorsed them, and that a reader should verify before leaning on one. A leaderboard that tells you not to trust it blindly has earned the right to be read.
The case against it
One maintainer, a repository a few weeks old, 31 stars. For something you might lean on, a bus factor of one is a real risk, and it is the first thing I would say to anyone quoting the number.
A single composite score invites false precision. “Best model” is a property of the scale, not of the world, and the uncertainty figure is the honest part while the decimal places are not. Roughly a third of the text data is still lab-reported, and a leaderboard fed by press releases is a press release with arithmetic. And the fitting itself rests on assumptions a statistician can argue with, however carefully it is documented.
The deepest problem is not the method at all. A benchmark gap is not a capability gap. Licence terms, language coverage, tool use, long-context behaviour, and whether the thing runs on the card you own are not in the score. A model can sit 7 months behind on the chart and be useless to you, or ahead on the chart and unaffordable. The number answers one question well and is silent on the rest.
Where this is likely heading
My judgement, not a forecast. Expect more leaderboards like this one, because source-linked aggregation has become a weekend project rather than an institution’s job. Expect the argument to move from who is leading to whose evaluations are independent — the split between 14,436 independent results and 7,677 self-reported ones is where the real controversy lives. And expect the lag to track access to compute more than cleverness in the models, which loops straight back to the retrospective: architecture follows constraint.
What you can do now
Read the method page before the headline, because the headline is only as good as the method. Check the marks on any model you care about: a star means thin evidence, a dagger means nobody but the maker has checked it. Ask whether the tests cover your work at all — a coding harness tells you little about a 20-page contract. And then run the thing on your own machine, because that is the only measurement that belongs to you.
A gap figure like this is worth having for one reason: it is a number you can argue with.
That is more than most of what gets published about AI can offer.
Update 20th Oct 2026: By this time, I have already registered with OpenRouter, Z.ai, Nous Research for my API keys and topup some $ for PAYG credits. Didn’t like the frequent rate limiting and the drop connections, so irritating. Decided to follow up on some community forum advice to go with DeepSeek API & PAYG credits and… BOOM, racked up 5,590 API requests and 1,279,342,135 tokens to-date! Cost-wise? SGD$15 bucks.