Disclosure: I run AiMOS Systems and use local AI hardware, so I have a stake in this question. I used Claude, made by Anthropic, to help research and draft this post. Numbers marked “vendor-reported” come from the model makers, not independent testing. Memory-fit figures are my own arithmetic.
On October 19, Stanford HAI published “Measuring What Medical AI Can Actually Do.” It summarizes three years of benchmark work. The headline: today’s best AI agents finish less than half of real clinical and administrative tasks. That finding is useful, and I am not disputing it.
My point is about what a reader should do with the number. A benchmark is a photograph, and this field moves fast enough that the photograph ages in months.
What was measured, and when
PhysicianBench (arXiv 2605.02240, submitted May 4, 2026): 100 long-horizon EHR tasks, 12 models. GPT-5.5 completed 46% on one attempt and 28% across all three runs. The best open model, DeepSeek V4-Pro, completed 19%.
HealthAdminBench (arXiv 2604.09937, submitted April 10, 2026): 135 administrative tasks. Claude Opus 4.6 in computer-use mode completed 36.3%. The best open-weight model, Kimi K2.5, completed 15.6%.
The Stanford article gives no evaluation dates. The papers date the work to spring 2026, so the models were tested five to eight months before the article ran. The article does say “the best agent tested at the time,” and it calls the cross-benchmark comparison “directional.” That is fair, and I credit it. But a hospital CIO reading “less than half” in late October will not read the hedge.
Not all of the models were stale. GPT-5.5 shipped April 23, 2026, so it was current when tested. The older entries are the ones that matter for the open-model story: Kimi K2.5 (January 27, 2026) was the best open model on HealthAdminBench, and DeepSeek V4-Pro was still a preview build when the paper was written.
Newer results exist, but not on the same tests
A third-party HealthAdminBench leaderboard (Steel.dev, updated June 12, 2026, not run by Stanford) lists Claude Opus 4.8 with a browser-use harness at 51.9%, against 36.3% for the paper’s best. The leaderboard flags its Anthropic rows as not directly comparable, so I treat that as a direction, not a measurement.
For open models, I found no published results on either benchmark for anything newer than Kimi K2.6 and DeepSeek V4-Pro. That is the real gap. Everything below about newer open models is inference from other benchmarks, not a claim about clinical performance.
The tested models and their newer replacements
DeepSeek V4-Pro (best open model on PhysicianBench; preview Apr 2026, final Aug 12) is followed by DeepSeek V4.1-Flash (Sep 10; 552B; MIT license).
Kimi K2.5 (best open model on HealthAdminBench; Jan 27) is followed by Kimi K2.6 (Apr 20; 1T total / 32B active; modified MIT) and Kimi K3 (weights Jul 27; 2.8T / 104B; custom license).
Qwen 3.5 and MiniMax M2.7 (also tested; early 2026) are followed by Qwen3.8-27B (Aug 14; 27.8B dense; Apache 2.0) and MiniMax M3 (Jun 1; ~428B / ~23B; license not found).
Not tested by Stanford: GLM-5.2 (Jun 16; ~753B / ~40B; MIT) and Muse Spark Open from Meta (Oct 15; 816B / 42B; Muse Model License).
Sources: model-maker announcements and news coverage; specs are largely vendor-reported. Sizes for V4.1-Flash active parameters were inconsistent across sources, so I omit them.
What the newer models show, on benchmarks other than the Stanford ones (all vendor-reported unless noted):
Qwen3.8-27B: 84.3 on OSWorld-Verified, a computer-use benchmark. The same benchmark scored Claude Opus 4.6 at 72.7 and GPT-5.4 at 75.0 earlier this year. A 28-billion-parameter model sits in that range.
Kimi K3: 57 on the Artificial Analysis index in July, against 56 for Claude Opus 4.8 and 59 for GPT-5.6 Sol on the same snapshot.
DeepSeek V4.1-Flash: Terminal-Bench 2.1 score of 90.6, and 74.2 on DeepSWE against 62.7 for V4-Pro.
Two cautions. First, OSWorld-Verified is saturating. On the harder OSWorld 2.0, the October leaders score about 71-73 while Qwen3.8 Max scores 19.4 and GPT-5.5 scores 13.0, so “matches last year’s frontier” does not mean “solves the task.” Second, none of this proves clinical competence. Medical workflows punish the exact failure Stanford found: writing the order in the note and never placing it. Better scores on coding and computer-use tests are suggestive, not conclusive.
What fits on hardware you can buy
Rule of thumb, my arithmetic: at 4-bit quantization, weights take about half a byte per parameter, plus room for context.
Qwen3.8-27B: ~14 GB at 4-bit. Fits one 24-32 GB GPU, even a 128 GB mini PC.
DeepSeek V4.1-Flash (552B): ~276 GB. Fits 4x RTX Pro 6000 (384 GB) or a 512 GB Mac Studio.
GLM-5.2 (~753B): ~377 GB. Fits a 512 GB Mac Studio; too tight on 384 GB with context.
Muse Spark Open (816B): ~408 GB per Meta, another source says ~530 GB. A 512 GB Mac Studio at the low estimate; the reported demo used two.
Kimi K2.6 (1T): ~595 GB (vendor INT4). Fits a DGX Station (748 GB).
Kimi K3 (2.8T): ~1.56 TB. Fits none of the above.
DeepSeek V4-Pro (~1.6-1.7T): ~893 GB reported. Fits none of the above.
Hardware, with prices as of the sources dated:
RTX Pro 6000 Blackwell: 96 GB GDDR7, 1.8 TB/s, 600 W. $11,500 at B&H in May; $16,000 on Nvidia’s marketplace in August (out of stock).
Mac Studio M5 Ultra: 96/256/512 GB unified memory, 1.2 TB/s. From $5,499; $18,299 for 256 GB/16 TB; the 512 GB model is “well over $20K,” unannounced.
DGX Spark: 128 GB LPDDR5X, 273 GB/s. $4,699 after a $700 increase; a partner box was $3,099 in July.
Ryzen AI Max+ 395 mini PCs: 128 GB shared, 256 GB/s. About $2,500-3,600.
DGX Station GB300: 748 GB combined, 7.1 TB/s, 1.6 kW. Listed from about $95,000.
Speed is the honest constraint. Reported generation rates: roughly 35-60 tokens per second for the 120B gpt-oss model on a DGX Spark or Strix Halo box; a vendor-reported 16-25 tokens per second for the 816B Muse Spark across two M5 Ultra Macs. I found no measured figures for GLM-5.x or V4.1 on this class of hardware. Local frontier-scale inference works today for one clinician or a small team, not for a hospital serving thousands.
My earlier framing of “$100,000 buys four RTX Pro 6000s” needs updating: a four-card build is now roughly $46,000-64,000 in GPUs alone at the prices above, and the memory shortage is pushing prices up, not down.
The slope: three years of evidence
Three years ago, Blackwell did not exist; Nvidia announced it in March 2024, and the H100 was the machine to have. In three more years Blackwell will look dated too: Vera Rubin began production shipments in August 2026, and Nvidia’s cadence is roughly annual. That cuts two ways, and I will state both.
Where the trend is strong:
Open models trail the frontier by months, not years. Epoch AI measured an average lag of about 3 months for the best open-weight models on its capabilities index (October 2025). Artificial Analysis showed the gap between best open and best proprietary narrowing from 13 index points to 6 between April 2025 and April 2026, though the index version may have changed between snapshots.
Models that fit one consumer GPU catch up fast. Epoch found they reach frontier-model capability within 6-12 months (September 2025).
Prices at fixed capability collapse. Epoch estimated the cost of GPT-4-level GPQA performance fell about 40x per year (data through February 2025, with a range of 9x to 900x depending on benchmark).
Agentic scores climbed steeply. OSWorld went from about 5% in 2024 to 66% in 2025 and above 80% for several 2026 models. SWE-bench Verified went from 49% (Claude 3.5 Sonnet, October 2024) to roughly 83-88% in April 2026. Contamination concerns apply to both, and most 2026 figures are vendor-reported.
Where the trend is weaker than the story:
Consumer memory barely grew. The flagship went from 24 GB (RTX 4090) to 32 GB (RTX 5090). Local capability grew because models got more efficient and mixture-of-experts designs run big models from cheaper memory, not because cards got dramatically larger. In 2026 the DRAM shortage pushed street prices up: the RTX 5090 sells near $5,100 against a $1,999 list price. TrendForce-linked reports expect tightness into 2028.
Demand may rise with efficiency. Cheaper tokens have so far meant more tokens. Vercel’s gateway data show open-weight models handling about 71% of tokens but only about 23% of spend in September, with total AI spend up 13x. Efficiency does not automatically shrink data-center demand. This is the Jevons-paradox objection, and it is not refuted.
Hardware obsolescence helps both sides. Depreciation schedules of 5-6 years sit against an 18-24 month architecture cadence, which is a risk for debt-financed builders. For buyers, it is the 2018 pattern: when crypto demand fell, used GPUs flooded the market and a GTX 1080 fell from $1,050 to $709. Whether used H100s and Blackwells follow that path is speculation on my part; today’s used H100 prices are an uncertain $6,000-22,000.
What this means for the Stanford result
The defensible reading is not that the study was wrong. It is that the study measured a point on a curve, and the curve is steep:
In the benchmarks’ own words, the failures are largely harness failures (memory across steps, forgetting to place the order), which is an engineering problem, not a fixed ceiling on model ability.
The best open model scored 19% on PhysicianBench and 15.6% on HealthAdminBench in spring 2026. Within six months, open models gained sharply on the adjacent agentic benchmarks I could find (mostly vendor-reported). Nobody has published the clinical rerun.
If the gap stays near three months and a ~28B-parameter model can match last year’s frontier on computer-use tests, then clinic-scale local deployment, where patient data never leaves the building, becomes plausible well before the end of this decade.
That last point is a projection, not a finding. I would bet on the direction and hold the timing loosely.
What I would ask Stanford and others to do: publish evaluation dates with every result, rerun on a schedule through the MedHELM leaderboard, and include the newest open-weight models, because for hospitals the question “can we run this inside our own walls?” matters as much as “which model is best?”
Sources: Stanford HAI (Oct 19, 2026); arXiv 2604.09937 and 2605.02240; Epoch AI (epoch.ai/data-insights); Artificial Analysis (Apr 30, 2026); Steel.dev HealthAdminBench leaderboard; Macworld, Notebookcheck, Tom’s Hardware, ServeTheHome, PCGH, Letsdatascience, DataNorth, NYU RITS and vendor pages as cited in the research notes.