Medical AI can notice a new guideline—and still give the old answer
A 1,709-revision preprint found that models often recognised changed clinical guidance without carrying every consequence into their recommendation. It is a benchmark result, not evidence of patient harm.
The short version
- What happened
- Researchers built ChronoEvo from 1,709 recommendation-level changes across 275 guideline lineages and tested 14 models with old, updated or no guideline evidence.
- Who it affects
- A clinician may see that an assistant has cited the newest guidance and reasonably assume the recommendation reflects it. This study shows why the final patient-specific advice still needs a line-by-line clinical check.
- What remains unproven
- The work is a preprint and has not been peer reviewed.
Keep in mind
The work is a preprint and has not been peer reviewed.
Most large-scale labels came from model evaluators; two clinicians adjudicated a 300-response subset rather than all 1,709 revisions.
The benchmark tests generated recommendations against guideline changes, not real clinical decisions, workflow use or patient outcomes.
Guideline sources and revision categories may not represent every specialty, country or form of evidence update.
Source analysis · 1 linked source. Primary analysed source date: 22 October 2026.
Read the full report · 3 min The evidence, methods, implications and limitations
Seeing the update was not the same as using it
Clinical guidance changes in ways that matter to an individual patient: a treatment can be withdrawn, a threshold moved or eligibility narrowed. University of Minnesota researchers paired old and revised recommendations across 1,709 changes, then asked models to respond with the revised evidence available. The primary model recognised 90.8% of changes but fully incorporated only 58.5%; two clinicians reviewing 300 responses found a 40.3-point recognition-to-incorporation gap.[1]
A stronger reminder fixed only part of the problem
An instruction to prioritise new evidence raised clinical correctness from 80.2% to 82.6% and reduced retention of outdated content from 5.9% to 4.3%. Full incorporation barely changed, from 58.5% to 59.3%, and that difference was not statistically detectable. Across all 14 tested models, recognition stayed ahead of full incorporation; even the strongest model retained a 20.1-point gap.[1]
The distinction that matters
Where the update can be lost
The benchmark separates three steps that are often collapsed into one accuracy score.
Recognise
Identify that guidance changed
90.8% in the primary test
Recommend
Produce a clinically correct answer
80.2% in the primary test
Incorporate
Carry every material consequence into the answer
58.5% in the primary test
The safe lesson is about verification, not diagnosis
Clinicians and health services should not treat retrieval of a current guideline as proof that an AI recommendation follows it faithfully. A useful check must compare the patient-specific output with every material change in the source. Patients should not use this preprint to choose or stop treatment: the benchmark generated recommendations from guideline text and did not test care pathways, outcomes or a regulated clinical product.[1]