Brian K. Hudes, MD
The turn from documentation to decision
For two years, the business of clinical AI has mostly been the business of notes — ambient scribes that listen to a visit and draft the documentation. That was the easy chapter, because a draft note has a built-in safety net: the physician reads and signs it. The industry is now pivoting to the harder chapter, and it has a slogan for it. "From documentation to decision." Stop summarizing care, the argument goes, and start improving it — surface risks, close care gaps, turn fragmented data into "actionable intelligence" delivered inside the workflow.
The ambition is sound. Demand outstrips capacity, and a tool that genuinely helps a clinician decide is worth building. But the slogan hides a load-bearing assumption: that the hard part — getting the underlying information right — is already solved, so the remaining work is delivery. It is not solved. And the move from documentation to decision raises the cost of getting it wrong, because a bad summary wastes a minute while a bad decision changes care.
Here is the failure that should organize our thinking. A clinician asks an assistant to summarize a patient before a visit. It returns something clean and readable — and misses that a medication was stopped six months ago, or silently merges two encounters, or presents a resolved diagnosis as if it were current. The output is fluent, plausible, well-formatted, and wrong. Nothing on its surface signals the error. In most domains a convincing answer is a good sign. In medicine it is the exact shape of the most dangerous mistake, because fluency suppresses the scrutiny that would have caught it.
Why the best answer is the wrong thing to measure
The reflex of the field — and of the press that covers it — is to grade clinical AI on its best answer. In 2026, The Lancet published a prospective study of Google's AMIE, a conversational diagnostic AI: roughly a hundred patients interacted with it, then saw a clinician who could view its output, and the researchers tracked the final diagnosis. The number that traveled was "90% accurate."
Read the paper and the number means something narrower. AMIE's differential contained the final diagnosis 90% of the time only at a list depth of seven. Its single most likely diagnosis — the one a hurried clinician would actually anchor on — was correct in 56% of cases. It reached 75% by the third candidate. A differential that is right at depth seven is useful; a first guess that is right barely more than half the time is a trainee you keep in the room, not one you send to speak to patients alone.
The safety finding deserves the same scrutiny. The study reported zero safety-stop interventions across all 98 completed AI interactions. That does not mean AMIE was unmonitored: physician safety supervisors watched every conversation in real time under prespecified criteria. They did not need to stop a consultation, but they noted one hallucination and added clinical information in five interactions. The study therefore shows feasibility inside an explicit human-oversight scaffold; it does not establish the safety of unsupervised use.
That is not a criticism of the study — it is arguably the study's most honest contribution. But it means "zero safety stops" should be read together with "continuous physician monitoring." The finding tells us about the model and the scaffold around it, not the model alone.
Safety is borrowed, not built — until you prove otherwise
Utah's prescription-renewal pilot makes the same point in the field rather than the lab. Physicians agreed with the AI's renewal recommendations 91% of the time; when the AI escalated a case instead of recommending renewal, physicians judged 31% of those escalations "too cautious." Utah's own report described that overcaution as appropriate for a new system. That is the correct direction for a tool that proposes rather than decides: it should err toward escalation, never past it.
But the reason the Utah pilot is defensible is structural, not technical. The state's current public page says the Doctronic pilot remains in Phase 1, where a licensed medical practitioner authorizes every request. The Utah Medical Licensing Board nevertheless recommended immediate suspension of the program in April 2026 after learning it had launched without the board's prior involvement; the Office of AI Policy declined to suspend it. The lesson is not that the AI is inherently trustworthy. It is that the current safety case depends heavily on the human-review step. Remove the reviewer and you have changed the intervention, not merely streamlined it.
This is the pattern worth naming: in credible demonstrations of clinical AI, much of the safety being reported belongs to the oversight scaffold as well as to the model. That is fine — good, even — as long as everyone is honest about what the scaffold contributes, and as long as the scaffold is actually funded and staffed when a pilot becomes a product. The danger is the quiet transition where the demo's human supervisors are removed to capture the efficiency gain, and the safety case is assumed to travel with the model.
"The AI" is never one thing
Part of why evaluation is hard is that we talk about "the AI" as if it were a single object with a single accuracy. It never is. Consider the ambient scribe, the most-deployed clinical AI there is. In their 2026 NEJM AI article, Privacy Considerations of Artificial Intelligence Scribes, Sara Gerke and David Simon describe how one encounter can generate distinct data artifacts — audio, a transcript, an AI-generated draft note, and the validated or final clinical note — with vendor practices differing in how long those artifacts are retained and whether they are used for AI training. Only the final clinical note has the clinician's judgment attached to it in the form in which it enters the record.
The point generalizes. Every clinical AI is a pipeline — retrieval, interpretation, synthesis, presentation — and each stage has its own failure mode. A summary can be wrong because it retrieved the wrong record, because it misread a correct one, because it reconciled a contradiction badly, or because it presented a correct fact without the context that makes it safe. "How accurate is it?" is not answerable until you say which stage, doing which task, on which data.
The five questions that actually matter
So if best-case accuracy is the wrong test, here is the right one. Before a clinical AI influences care, I want to know what it does with a hard case, not an easy one:
Traceability. Can every clinical claim in the output be traced to a specific source in the record? An assertion you cannot check is an assertion you cannot trust, however well it reads.
Temporal context. Does the system distinguish current from historical — an active drug from a discontinued one, an open problem from a resolved one, today's value from last year's? A timeline error can turn a true fact into a dangerous instruction.
Contradiction handling. When two records disagree, does it show the clinician both values and flag the conflict, or does it silently choose one and hand back a tidy, confident answer? The tidy answer is the one that erodes trust the moment a clinician discovers they still had to verify it by hand.
Abstention. When the record does not contain the answer, does the system flag the gap, or fill it? This is the test I would run first, because it is the hardest to fake. A model optimized to always return a fluent answer will always return one. A model willing to say "I don't have enough here" has been built around the cost of being wrong.
Human review. Can the clinician see, question, and correct the output before it acts — with enough transparency to do so meaningfully, not a buried "accept" button? This is the scaffold from the AMIE and Utah examples, made explicit and designed in rather than assumed.
These are not prompt tweaks. They are architecture: retrieval and source attribution for traceability, explicit temporal modeling, a reconciliation layer that preserves conflicts instead of hiding them, an abstention policy with teeth, and a review surface built for scrutiny. The right combination depends on the task and the cost of an error — a medication reconciliation and a reading-level rewrite do not need the same safeguards — which is exactly why "we use AI" is never a sufficient description of what a system is or how it should be judged.
What this asks of us
In September, six of the largest U.S. physician organizations warned against the framing that AI is inherently better-informed than physicians, or that doctors should not make decisions without consulting it. The statement was read as a defense of professional turf. I read it as a design specification. Keeping the physician central is not a sentiment to affirm; it is a set of requirements to build — traceability, temporal context, contradiction handling, abstention, and genuine review — without which "the physician is still in charge" is a claim the architecture cannot actually support.
The move from documentation to decision will happen; it should. But it should happen on systems designed around the consequences of being wrong, not the quality of their best answer. The useful question is never how impressive the demo is. It is whether the system, handed a contradictory, incomplete, out-of-date record, will tell you so — or hand you a clean summary that looks exactly like the truth. A clinical AI earns its place not by being brilliant on its best day, but by being honest on its worst.
(The governance half of this argument — who is accountable, and who monitors the system after it ships — I took up separately in "The accountability layer: what buying an EHR does not settle.")
Brian K. Hudes, MD, is a practicing gastroenterologist and writes The Renaissance Physician. Disclosure: he is the founder and CEO of AiMOS Systems Corporation and has a commercial interest in how clinical AI is built and evaluated.