On 12 June 2026, Nature Medicine published a head-to-head study that should make every team building a “purpose-built” AI model stop and recalculate. General-purpose frontier models — OpenAI’s GPT-5.2, Google’s Gemini 3.1 Pro, Anthropic’s Claude Opus 4.6 — were tested against two specialist clinical AI tools, OpenEvidence and Wolters Kluwer’s UpToDate Expert AI, on medical questions from practising physicians.

The generalists won. Not narrowly, and not on one benchmark — on every benchmark tested. On MedQA, a standard medical-knowledge exam, Gemini scored 97.4% against roughly 89% for the two specialist tools. The models built for everything beat the models built specifically for medicine, on medicine’s home turf.

The question worth asking is the obvious one: what does this mean for frontier models, and does it still make sense to train purpose-built models at this stage? The honest answer is uncomfortable for a lot of AI strategies, and it’s been hiding in plain sight in AI research for decades.

For the rest of us: generalist vs. specialist models

There are two ways to get an AI that’s good at a narrow task like answering medical questions.

The first is to build a specialist: take a model and train or wrap it specifically for the domain — feed it medical literature, fine-tune it on clinical data, curate its sources, design it around how doctors work. This is the intuitive approach, and it’s what OpenEvidence and UpToDate Expert AI are: tools built for clinicians, grounded in verified medical evidence. (MedQA and HealthBench, the tests here, are standard medical-AI benchmarks — exam-style questions and clinician-alignment checks.)

The second is to take a generalist — a frontier model trained on essentially the whole internet to be good at everything — and just point it at the medical question with no special training.

Common sense says the specialist should win. It was built for exactly this. The surprise of the Nature Medicine study — and it’s a surprise that keeps recurring across AI — is that the generalist won anyway. The model that was trained on everything turned out to know medicine better than the models trained on medicine.

What the study actually found

It’s worth being precise, because the result is easy to dismiss as a quiz score and it’s more rigorous than that.

The evaluation ran in three stages. First, 500 MedQA questions testing medical knowledge — Gemini 3.1 Pro led at 97.4%, versus 89.6% for OpenEvidence and 88.4% for UpToDate. Second, 500 HealthBench items measuring how well answers aligned with clinicians, where GPT-5.2 led at around 88%. Third, and most importantly, a “real clinical queries” benchmark built from 100 de-identified questions that physicians had actually asked in a live clinical setting — reviewed blind and at random by 12 US clinicians, producing 1,800 annotations. The frontier models came out ahead in all three.

That third stage matters because it answers the obvious objection. Beating an exam is one thing; the real-query benchmark with blinded clinician review is much closer to the actual job. The generalists led there too.

This is the “bitter lesson,” arriving in medicine

If this pattern feels familiar, it should. In 2019, the AI researcher Richard Sutton wrote a short, now-famous essay called The Bitter Lesson. Its claim, drawn from seventy years of AI research: general methods that scale with computation always, eventually, beat approaches that build in human expertise and domain-specific cleverness.

The history is a graveyard of beaten specialists. Chess programs hand-crafted with grandmaster knowledge lost to Deep Blue’s brute-force search. Go engines built on human strategy were surpassed by AlphaGo — and then AlphaGo Zero discarded human games entirely and got better. Speech recognition built on linguistic rules lost to statistical models trained on raw data. Each time, the field’s instinct was to encode expertise; each time, the general method that scaled won in the end.

The Nature Medicine result is the bitter lesson reaching clinical AI. The specialist tools encoded medical expertise and curated medical sources — exactly the handcrafted-knowledge approach. The frontier models just scaled. And the frontier models won. This is not an anomaly to explain away. It’s the most consistent pattern in the history of the field, showing up on schedule in a new domain.

The fair counterpoint — and where the real moat is

Honesty demands the caveats, because they’re where the actual strategy lives.

OpenEvidence pushed back hard, and some of its objections land. MedQA questions may have leaked into the frontier models’ training data, inflating that 97.4%. HealthBench scoring is partly subjective — stylistic preference can masquerade as clinical quality. And the real-query benchmark was reportedly added only after peer reviewers flagged the original evidence as thin. A benchmark win is not the same as clinical safety, and nobody should read this study as “fire the specialists.”

But notice what the specialists’ defenders fall back on. Not “our model is smarter.” They argue that OpenEvidence and UpToDate offer verified evidence curation, a trusted interface, and accountability — the things that don’t show up in benchmarks. That’s the tell. The defensible value of a specialist tool has migrated away from raw model capability — which the frontier now dominates — toward everything wrapped around the model: the verified sources, the audit trail, the clinician trust, the regulatory clearance, the integration into the workflow.

Which is the same conclusion arriving from a different direction. I argued recently that the frontier model is becoming a commodity — cheap, abundant, interchangeable. This study shows the other face of the same coin: that commodity is also so capable it eats the specialist’s narrow advantage. Both point to the same place. The value isn’t in the model. It’s in the proprietary data, the verified sourcing, and the trust you build on top of it — the same lesson as why most “AI problems” are really data and architecture problems.

What this means

So — does it still make sense to train purpose-built models? For most organisations, the answer is now: rarely, and not the way you think.

Don’t compete with the frontier on raw capability. If your plan is “we’ll fine-tune a smaller model to beat GPT-5 at our vertical task,” the bitter lesson says you’ll likely lose, and the gap will widen with each frontier release. You’d be spending heavily to fall behind on a schedule.

Build on what the frontier doesn’t give you. The frontier model is a commodity input — assume it, and put your effort into the layer benchmarks can’t measure: proprietary or verified data the model wasn’t trained on, an audit trail and sourcing a clinician (or auditor, or regulator) can trust, accountability for outcomes, and deep integration into the real workflow. OpenEvidence and UpToDate will likely survive not because their models are better, but because of exactly those wrappers — if they lean into them.

Reserve true purpose-built training for the genuine exceptions. They exist: domains where the data is private and unlike anything on the public web, where latency or cost demands a small local model, where regulation requires a fully controlled and explainable system. But these are narrower than most roadmaps assume, and shrinking as frontier capability climbs.

The era of “we’ll build our own specialised model and that’ll be our moat” is closing. The generalist just beat the specialist at medicine. Your moat was never going to be the model. It’s the verified data, the trust, and the workflow you wrap around a commodity that everyone — including your competitors — can simply call.

References

  • Nature Medicine (12 June 2026), “General-purpose large language models outperform specialized clinical AI tools on medical benchmarks.” Models: GPT-5.2, Gemini 3.1 Pro, Claude Opus 4.6 vs OpenEvidence and UpToDate Expert AI. MedQA: Gemini 97.4% vs OE 89.6%, UTD 88.4%; HealthBench ~88% (GPT-5.2); real-clinical-queries benchmark, 12 clinicians, 1,800 blinded annotations.
  • OpenEvidence response — objections re MedQA training-data contamination, subjective HealthBench scoring, and the real-query benchmark added after peer review (coverage: Digital Health Wire, Clinical Trial Vanguard, June 2026).
  • Richard S. Sutton, The Bitter Lesson (2019) — general methods that scale with computation beat handcrafted domain knowledge; chess (Deep Blue), Go (AlphaGo / AlphaGo Zero), speech recognition.
  • keller-ai — The Frontier Model Is Becoming a Commodity; Don’t Mistake a Modern IT Architecture for AI.