Will healthcare AI agents escalate when information changes?

The answer: clean accuracy did not predict safe hand-off behaviour

Not reliably in this experiment. Five open-weight language-model configurations often selected the expected action when the synthetic workflow was complete, yet several continued answering, abstained or merely asked a question after information changed in ways the study policy designated for immediate escalation. The strongest tested model narrowed the gap but did not remove it: its decision accuracy was 0.97 on clean episodes and 0.78 when sources conflicted.

The finding matters because clinical work is not a static question-answer task. A portal draft can become unsafe when a red-flag symptom arrives after the request; a medication instruction can become ambiguous when a more recent result conflicts with the old summary. A model that can state relevant medical facts may still choose the wrong operational action when the state of the workflow changes.

This is an unreviewed preprint submitted on 15 October by researchers at Washington University in St Louis and Southern Methodist University. It is not a clinical trial or a test in a hospital. All cases were synthetic, all target actions reflected a study-authored policy and no patients or clinicians were enrolled. The paper does not present a funding or competing-interest statement, so those disclosures cannot be assessed from the manuscript.[1]

How the benchmark was constructed

The authors wrote 40 semantic seeds across four task classes: documentation, evidence retrieval, medication-related work and patient messaging. Each seed had two surface forms and five controlled variants—clean, missing, stale, conflict and late-risk—creating 400 episodes. The permitted actions were ANSWER, ASK CLARIFICATION, ABSTAIN and ESCALATE. The variants were assembled deterministically rather than generated by another model, allowing paired comparisons in which the manipulated workflow state was known.

Two seeds per task class supplied 80 development episodes used to tune a post-processing controller. The remaining 320 episodes formed the test set, including 100 whose study policy required escalation. Because two phrasings and five variants shared one underlying seed, the researchers treated 32 held-out seeds as the independent units for inference. They used 5,000 seed-clustered bootstrap resamples and paired Wilcoxon tests, a much more defensible denominator than treating all 320 outputs as unrelated observations.

The five configurations were Qwen3-1.7B, Qwen3-4B, Llama-3.2-3B, Gemma-3-4B and a quantised Qwen3.5-9B. Every episode was run with a neutral prompt and with an uncertainty checklist that explicitly told the model to check for missing, stale, conflicting and late-arriving information. That produced 4,000 logged generations. The two model-serving runs used different Ollama versions and the largest model was added separately, which the authors recorded in distinct manifests.

Reference actions were not clinician consensus. One study author supplied the labels, sometimes permitting more than one acceptable action. Accuracy therefore means agreement with this operational policy, not verified clinical correctness. The authors say the episode data, raw outputs, manifests, tuned parameters and evaluation code are packaged for audit, but independent reproduction and clinician re-annotation remain necessary.[1]

What the models and controller did

Under the uncertainty prompt, Qwen3-1.7B escalated only 3% of the 100 escalation-required test episodes and missed 97%; Llama-3.2-3B missed 90%. Qwen3-4B missed 46%, Gemma-3-4B missed 48% and Qwen3.5-9B missed 32%. The errors were not always ignorance. Free-text reasoning sometimes named the red flag while the selected action remained too weak—for example asking a follow-up instead of escalating.

Stated confidence was a poor standalone control. Expected calibration error ranged from 0.10 to 0.54, while the area under the curve for using one minus confidence to predict a failure ranged from 0.48 to 0.77. A confidence-only policy tuned on development episodes did not improve utility. The checklist itself was model-dependent: it improved escalation sensitivity for two Qwen models but reduced Gemma-3-4B sensitivity from 0.64 to 0.52 and shifted some responses from escalation toward clarification.

The authors then combined low stated confidence with three regular-expression detectors for conflict, missing information and high-risk cues. Under a pre-declared cost that made a missed escalation five times as costly as an unnecessary one, this controller raised sensitivity to between 0.76 and 0.88 and reduced the missed-escalation rate to 0.12–0.24. Specificity fell to 0.83–0.93, implying an unnecessary-escalation rate of roughly 7%–17% under the benchmark labels.

That apparent mitigation failed its most revealing test. For each detector, ten familiar trigger phrases were paired with ten meaning-preserving paraphrases it had not seen. Recall fell from 1.00 to 0.00 for all three detector types—for example, a rule recognising 'shortness of breath' did not recognise 'dyspnea'. The controller result is therefore an in-benchmark engineering demonstration, not a deployable safety guarantee.[1]

What clinicians and service owners should do with the result

Hospitals should test the action boundary, not only answer quality. A deployment evaluation needs paired workflows where records arrive out of order, summaries are superseded, patients minimise symptoms and late information changes urgency. Reviewers should record whether the system answers, asks, abstains or hands off, then price both missed and unnecessary escalation with the clinical team that will absorb the consequences.

A prompt that says 'be cautious' is not evidence of safer behaviour. Nor is self-reported confidence a sufficient stop signal. Operational controls should use structured provenance, recency, contradiction and risk information where possible, with escalation logic tested on paraphrases, abbreviations, local terminology and different institutions. Staff also need a reliable human queue; routing everything to clinicians may reduce one risk while creating delay and alert burden elsewhere.

For patients, the study does not show that current clinical chatbots routinely miss emergencies. It shows a specific evaluation gap: high performance on clean synthetic tasks can coexist with weak hand-off decisions under controlled changes. Providers should avoid presenting clean-answer benchmarks as proof that a system can manage evolving patient information without supervision.[1]

Evidence limits and what would change the assessment

The 400 episodes derive from only 40 hand-authored seeds, with 32 independent seeds in the test split. Surface forms are near-duplicates, detector writers knew the seed pool and stylistic regularities may make the in-benchmark controller look stronger. No proprietary frontier model, clinically adapted model, live electronic-record workflow or patient outcome was studied. Over-escalation workload was counted but not observed in practice.

Confidence would rise with clinician-authored cases, independent dual annotation and adjudication, preregistered cost choices, larger multi-institution test sets and held-out linguistic styles. A serious deployment study should compare prompt-only, rule-based and learned controls, measure queue burden and time to human review, and report failure rates for each clinical risk class rather than one pooled score.

The assessment would strengthen if different teams reproduced the clean-to-perturbed gap and a controller generalised across institutions, terminology and paraphrases without overwhelming staff. It would weaken if clinician adjudication substantially changed the target actions or if the failures disappeared on independently written workflows. For now, the solid conclusion is about evaluation practice: healthcare agents need state-change and escalation tests, while the proposed lexical controller remains demonstrably brittle.[1]