Stanford's clinician-built MedHELM is the benchmark buyers should trust

*Saturated math tests say models are ready; the test doctors helped write says admin work is still close to a coin flip.*

Opening read

My read is that this autumn's benchmark story has split in two, and only one half should touch a budget. The headline scores keep climbing. Stanford's MedHELM, built with input from 29 clinicians, finds the best models at their weakest in administration and workflow, which is the part of medicine where the trillion-dollar overhead sits. Buy against the second number, not the first.

Medical AI: the test clinicians helped shape finds the weak spot

The easy read is that the big public benchmarks are running out of road, so the models must be ready for professional work. FrontierMath, a test of research-level math problems, has climbed from 2% at its November 2024 launch to 94% now. Claude Opus 5.5 posts 67.7% on Humanity's Last Exam. Medicine has its own version of that story, and the story keeps getting stronger. In a study cited in late 2024, GPT-4 scored 90% on a diagnostic test against 74% for doctors. This spring GPT-5.4 for clinicians scored 59% on the HealthBench Professional subset, beating both GPT-5.5 and answers physicians wrote themselves Import AI (https://importai.substack.com/p/import-ai-477-in-which-i-tell-the) Nate Jones (https://www.youtube.com/watch?v=PCPVWfMJST0) AI Explained (https://www.youtube.com/watch?v=jz0rNhfAKo8). A sharp hospital or insurer executive could read that as permission to spend. I would not, and the reason comes down to which test you look at.

The number I would weigh instead is Stanford HAI's MedHELM, because it was shaped by the people who do the work, and it is less flattering. Its weakest category is administration and workflow, the paperwork end of medicine. That is the finding to carry into a contract talk. The spread is wide. Across its five categories the strongest models' win rates (the share of head-to-head comparisons the model comes out ahead in) run from 0.53 at the low end up to 0.89, and 29 clinicians gave input on the design. The floor sits at the expensive end. Administrative spending in US healthcare runs above $1 trillion a year. The same write-up recalls that in early 2024 GPT-4 erred on 35% of 983 patient-record questions written by 15 clinicians Stanford HAI (https://hai.stanford.edu/news/measuring-what-medical-ai-can-actually-do).

Insurers add a separate cost line. A Blue Cross Blue Shield Association analysis found that hospitals' AI tools for filing claims added $942 million to healthcare spending over two years TechCrunch AI (https://techcrunch.com/2026/09/26/insurers-claim-ai-is-already-increasing-healthcare-costs/). That figure says nothing about model accuracy or review labor. It says admin AI already moves real money in billing, and in this case the insurers say the money moved the wrong way. My caution comes from the ranking, not from arithmetic on the score. A win rate is a comparison share, not an error rate, so 0.53 does not tell you how often the model gets a claim right against your current process. What it tells you is that paperwork is where the best models are weakest. The one absolute error figure in Stanford's write-up, the dated 35% from early 2024 on a model since replaced, points the same way. Weakest category plus a history of errors means a reviewer stays in the loop. That is a cost line, not a saving.

Two different problems explain why the headline tests mislead here, and they should be kept apart. The first is whether the test itself holds up. FrontierMath needed a second version after an audit found errors in 42% of the original problems, so the climb from 2% to 94% may not compare like with like, since the evidence does not say which version produced the 94% Epoch AI Gradient Updates (https://epochai.substack.com/p/the-epoch-brief-june-12-2026). Reproducibility is improving on this front, and it deserves credit: UK AISI and EvalEval now publish verified results for HealthBench and FrontierMath, among others Hugging Face Blog (https://huggingface.co/blog/evaleval-aisi). That fixes whether a number can be rerun. It does not fix what the number hides. The second problem is what a score summarizes even when the test is sound. On the public HELM leaderboard, the five models ranked best by average score and the five ranked best by their weakest scenario share no members at all across 54 models Hugging Face Daily Papers (https://tldr.takara.ai/p/2607.26191). Averages hide the floor, and the floor is where a workflow breaks.

For a business leader weighing a medical AI contract, the stake is whether the spend line is labor savings or review labor. MedHELM says to assume the second until your own data says otherwise. Here is the check I would run before signing. Pull a batch of real questions from last month's billing or scheduling queue. Have the staff who handle them write the questions in their own words. Run them through the candidate model, then have the same staff judge the model's output against the current process without knowing which is which. Count the wins. Stanford's 0.53 is not the same test, so do not treat it as a pass line. But if your blind wins look close to a coin flip, you are buying a second opinion, not a replacement, and the contract should be priced that way. For an engineer building the pilot, the same exercise tells you which category to put in the dashboard: the weakest one, because that is the number the HELM result says the average will hide.

Agent wrappers: test the wrapper before paying for the model

The unit to measure is the model-and-wrapper pair, not the model alone, and two tests this year show why. Cerebras held the model constant, Claude Opus 4.6, and raced two wrappers (the software around the model that manages files, tools, and steps) on the same whiteboard-app build. Factory Droid finished 11 minutes ahead, roughly 2.4 times the pace of Claude Code, and reached the finish with about half as many tool calls, the steps where the agent reaches for a tool like a file read or a search Cerebras (https://www.cerebras.ai/blog/a-smarter-model-won-t-save-your-bad-harness). With the model fixed, that gap is the wrapper showing. It is the kind of gap a buyer only sees by testing the pair. But the wrapper is only half of what you are buying. An earlier Faros test held the wrapper constant instead and swapped the model: Claude Code on GLM-5.2 slightly beat the same wrapper on Opus 4.8 at about half the cost per task Fireworks AI (https://fireworks.ai/blog/fireworks-nexus). The Cerebras post is a vendor's blog about one app build, so treat it as one run. Even so, an engineer picking an agent stack should race two wrappers on a real ticket with the model fixed, then race two models inside the winning wrapper, before paying for a bigger model.

Open weights: Meta's Mac Studio count is half of Apple's

Meta released Muse Spark Open last week as open weights (weights a company can download and run on its own machines). Apple's researchers have already run it across four Mac Studios at better than 36 tokens a second, roughly a brisk reading speed for the output. An IT leader sizing hardware for it should budget four Mac Studios if a tested speed matters, and keep it off agent work for now. The reason is what has and has not been published. The model carries 816 billion total parameters with 42 billion active per token (the slice of the model that fires for each word piece). Meta says the NVFP4 checkpoint (a compressed 4-bit copy of the weights) fits on two 512GB M5 Ultra Mac Studios or a handful of Nvidia data-center chips. Fitting is all Meta claims. It publishes no speed for the two-box setup. Apple found that even compressed to about 493GB the model still needs more than one 512GB Mac Studio, and four of them is the only configuration with a published speed Latent Space (https://www.latent.space/p/ainews-high-taste-testers-ep-1) Apple Machine Learning Research (https://machinelearning.apple.com/research/mlx-distributed-inference). Two boxes may run fine. No published run shows it. There is a second limit that hardware does not fix. Artificial Analysis scores the model 39 on its Intelligence Index, top among US open-weight models, yet last of the nine strongest open models on its tests of multi-step agent work Latent Space (https://www.latent.space/p/ainews-agents-disprove-polysemanticity). That is the lead's lesson again, an average hiding a floor. The floor here is the agent use case most buyers are sizing for.

Augur calls

• Call: Stanford HAI's MedHELM will publicly show a top model win rate of at least 0.70 in its administration and workflow category by 2027-06-30.
• Criteria: A MedHELM leaderboard page or Stanford HAI publication lists any model at or above 0.70 win rate in the administration and workflow category on or before the due date; absent that, miss.
• Due: 2027-06-30
• Odds: 40%
• Call: Artificial Analysis will list a US-developed open-weight model scoring above 39 on its Intelligence Index by 2027-03-31.
• Criteria: The public Artificial Analysis Intelligence Index shows a US-developed open-weight model above 39 on or before the due date; absent that, miss.
• Due: 2027-03-31
• Odds: 65%

What to watch

• Stanford HAI's MedHELM leaderboard is the place to check whether any model lifts the administration and workflow floor above the 0.53 it reports now. Stanford HAI (https://hai.stanford.edu/news/measuring-what-medical-ai-can-actually-do)
• Artificial Analysis's rankings for multi-step agent work will show whether Muse Spark Open's last-place agent ranking moves in later updates. Latent Space (https://www.latent.space/p/ainews-agents-disprove-polysemanticity)
• OpenAI's new SQLite and software engineering evaluations point to a push into low-resource programming languages. Watch whether it publishes model scores on them, which would show how far behind those languages sit. OpenAI News (https://openai.com/index/low-resource-programming-languages)
• Transformers, Hugging Face's widely used model library, shipped 5.20.2 to stop inputs losing their type during generation for models that take images and text. Teams running such models on 5.20.x should pick it up and watch whether the 5.20 line needs another patch before it settles. Transformers Releases (https://github.com/huggingface/transformers/releases/tag/v5.20.2)

Verification notes

MedHELM's win rates are Stanford's own reporting. The 35% error figure dates to early 2024 on a model since replaced, so treat it as history rather than a current rate Stanford HAI (https://hai.stanford.edu/news/measuring-what-medical-ai-can-actually-do). The GPT-4 90% versus 74% diagnostic figure and the HealthBench Professional result both reach us through video commentary rather than papers, the first from a late-2024 video, so the dating leans on when the commentary ran. The evidence does not say which FrontierMath version produced the 94%. Cerebras's wrapper result is a single vendor-run task. The Faros comparison comes by way of a Fireworks blog post. Meta's two-Mac-Studio fit is a vendor statement with no published speed, while Apple's published run used four machines.

Inspect the evidence behind this dispatch. (https://augurdispatch.ai/evidence/2026-10-20?utm_source=augur&utm_medium=web&utm_campaign=dispatch-2026-10-20)