The same cancer case changed medical AI staging accuracy by over 50% when its format changed
Across 3,035 physician-validated perturbations, three medical language models gave markedly different clinical answers when equivalent cases were reformatted. The peer-reviewed study supports model-specific safety testing, not a claim about patient outcomes.
The short version
- What happened
- Researchers tested GPT-5, MedGemma-27B-Text-IT and OpenBioLLM-Llama3-70B with 3,035 systematic changes to physician-validated oncology and hospital cases.
- Who it affects
- Clinicians could receive different recommendations from equivalent records, making local format testing and visible uncertainty important before use in care.
- What remains unproven
- No prospective clinical workflow, clinician comparison or patient outcome was measured.
Keep in mind
No prospective clinical workflow, clinician comparison or patient outcome was measured.
Only three models and selected oncology and hospital-case formats were tested; results may not transfer to other specialties, languages or model versions.
The mechanistic analysis covered the two open models; it should not be read as an explanation of GPT-5's internal representations.
The work was funded by the US National Institutes of Health and the Pelotonia Institute for Immuno-Oncology. The authors declared no competing interests.
Source analysis · 1 linked source. Primary analysed source date: 31 October 2026.
Read the full report · 3 min The evidence, methods, implications and limitations
Equivalent cases did not produce equivalent reasoning
A medical language model's answer can depend heavily on how a case is written, even when the clinical facts stay the same. Ohio State researchers evaluated GPT-5, MedGemma-27B-Text-IT and OpenBioLLM-Llama3-70B using 3,035 physician-validated perturbations of oncology SOAP notes, short vignettes and MIMIC hospital cases. They changed presentation rather than the underlying diagnosis question.
The paper reports staging-accuracy swings of more than 50 percentage points from prompt format alone. Models also sometimes gave a definite stage when the supplied case was clinically insufficient. For a clinician, that matters because copied notes, terse summaries and structured records can express the same evidence differently; a benchmark average can hide that sensitivity.[1]
Safety fixes may need to match the model
To examine why, the team trained sparse autoencoders on one billion tokens from 50,000 MIMIC-IV notes and analysed the two open models' internal representations. MedGemma showed a hierarchy in which stronger features tracked lexical identity and weaker ones carried contextual meaning, while OpenBioLLM distributed information more evenly. Retrieval and reasoning interventions therefore behaved differently across architectures.
This does not establish how often a deployed system would harm a patient. The cases were controlled tests, the models were a small selection, and the analysis did not compare clinicians using the tools with clinicians working alone. The practical result is a validation requirement: hospitals should test their exact model, prompt, record format and failure-handling process before relying on a general medical benchmark.[1]
The distinction that matters
Why formatting can become a clinical safety problem
The study kept the clinical substance stable while varying how the case reached the model.
Case format
Rewrite the same evidence as notes or vignettes
Staging accuracy moved by more than 50 percentage points
Missing evidence
Present a clinically insufficient scenario
Some outputs still supplied a definitive stage
Model structure
Inspect two open models' internal features
The same intervention did not transfer safely across architectures