Ten LLMs faced complex colorectal-cancer decisions; none established a stable clinical lead
Across 1,080 primary responses to synthetic cases, full concordance ranged from 49.1% to 63.0%. Stricter scoring, counterfactual changes and confident wrong answers exposed why a fluent plan is not clinical validation.
The short version
- What happened
- Researchers tested ten LLMs three times on 36 primary synthetic scenarios, producing 1,080 responses. Fully concordant plans ranged from 49.1% to 63.0%, with no statistically robust model effect.
- Who it affects
- For patients and oncology teams, an apparently polished treatment sequence can hide an omitted condition or a failure to react when one decisive fact changes. The study supports supervised testing of decision support, not direct use of a general-purpose model for treatment choices.
- What remains unproven
- Synthetic vignettes
Keep in mind
Synthetic vignettes
36 independent primary scenarios
Ten prompt-development cases remained in the benchmark
One zero-shot prompt
No retrieval or multimodal record
Models tested at one September 2026 time point
Consensus-derived reference retains judgment
No clinician-plus-LLM arm
No patient outcomes
Source analysis · 1 linked source. Primary analysed source date: 28 October 2026.
Read the full report · 3 min The evidence, methods, implications and limitations
A benchmark built around linked treatment decisions
Colorectal cancer that has spread to the peritoneum forces teams to connect several difficult choices: referral, systemic therapy, restaging, surgery, intraperitoneal treatment and surveillance. The researchers translated a 2026 expert-consensus pathway into 48 synthetic vignettes: 36 primary scenarios and 12 counterfactual cases arranged as six pairs where one decisive clinical feature changed.
Ten LLMs answered every vignette three times using one frozen zero-shot prompt. Two blinded colorectal surgeons validated the reference classifications. Across 1,080 primary responses, 56.4% were fully concordant under the structured scoring rule, 10.8% partial and 32.8% discordant. Model-level full concordance ran from 49.1% to 63.0%, but the statistical tests did not support a reliable ranking.[1]
Small wording changes were not the main problem
The result weakened when the evaluation asked whether the whole recommendation was semantically complete: full concordance fell to 45.5%. Counterfactual tests were also unstable. No model responded in the expected direction for more than three of six paired changes in a majority of runs, even when the altered fact should have changed management.
Confidence did not reliably protect the reader. Twenty-eight per cent of semantically discordant answers expressed certainty of at least 90. The 21 prespecified clinically consequential flags came from only five independent scenarios, so they are warning patterns rather than 21 observed harms. No real patient record, multidisciplinary team, clinician-plus-model workflow or patient outcome was tested.[1]
The distinction that matters
Why one benchmark score was not enough
The study checked agreement, response to a changed clinical fact and the relationship between confidence and correctness.
Concordance
Compare plans with an expert-consensus pathway
49.1%–63.0% full by model
Counterfactual
Change one decision-critical feature
No stable majority success beyond 3 of 6 pairs
Certainty
Compare stated confidence with correctness
28% of discordant answers still claimed at least 90 certainty