Do AI models favour one cancer guideline?
Four models produced 6,000 choices across 15 conflicts between US immunotherapy guidelines and usually selected NCCN. But 10 conflicts involved one lung-cancer setting, and removing them weakened or reversed the pattern for most models.
The short version
- What happened
- Researchers extracted 15 disagreements from ASCO and NCCN immune-checkpoint-inhibitor guidelines dated October 2023 to October 2024.
- Who it affects
- A consistent hidden preference could steer clinicians or patients toward one recommendation without explaining why guidelines differ.
- What remains unproven
- The clinical denominator is 15 guideline disagreements; 10 concern non-small-cell lung cancer.
Keep in mind
The clinical denominator is 15 guideline disagreements; 10 concern non-small-cell lung cancer.
Repeated outputs increase precision about model variability but do not create new clinical cases.
The two compared guideline families are US based, limiting geographic transfer.
Different web-interface temperatures and inaccessible training data limit causal interpretation and model comparison.
No clinician decision, patient outcome, adverse event or prospective deployment was measured.
Source analysis · 2 linked sources. Primary analysed source date: 8 October 2026.
Read the full report · 7 min The evidence, methods, implications and limitations
The direct answer: the models showed a preference, but the small case mix shaped it
All four tested language models more often chose National Comprehensive Cancer Network recommendations when asked to select between NCCN and American Society of Clinical Oncology guidance that differed on immune-checkpoint-inhibitor treatment. Gemini 1.5 Pro selected NCCN in every run; Claude 3.7 Sonnet, DeepSeek-V3 and GPT-4o selected it in roughly seven out of 10. Reversing the order of the two guideline options did not remove the direction of the result, making a simple first-choice position effect less likely.
The experiment is not evidence that NCCN is better, that the models made the clinically correct decision or that one can safely resolve a guideline conflict. It measured preference across only 15 disagreements. Ten concerned non-small-cell lung cancer, and the sensitivity analysis changed the picture substantially when those cases were removed. The practical finding is that a general-purpose model may add a repeatable, model-specific tilt to an already uncertain clinical question unless its sources and reasoning are made explicit.[1][2]
The study generated 6,000 choices from 15 guideline disagreements
The researchers searched ASCO and NCCN immune-checkpoint-inhibitor guidelines published or updated between October 2023 and October 2024. Two investigators independently extracted discordant recommendations for comparable tumour types, patient characteristics and treatment contexts, then resolved disagreements by consensus or senior adjudication. The 15 final cases were classified by cancer type, line of therapy, clinical stage, monotherapy or combination immunotherapy, concurrent chemotherapy and checkpoint-inhibitor class.
Four model versions were accessed through their official web interfaces in April 2025: GPT-4o, Claude 3.7 Sonnet, Gemini 1.5 Pro and DeepSeek-V3. Researchers manually entered a standard prompt that included each recommendation, evidence level, recommendation grade and treatment context. Each model answered each of the 15 cases 100 times in separate conversations without prior context. That produces 1,500 choices per model and 6,000 overall, although repeated generations do not create 6,000 independent clinical scenarios.[1]
The headline preference ranged from 71.3% to 100%
Gemini selected NCCN in 100% of runs. Claude selected it in 73.7%, DeepSeek in 72.0% and GPT-4o in 71.3%. The paper reports uncertainty intervals clustered around the 15 questions, rather than treating all repeated outputs as if they were unrelated cases. Claude's reported 95% interval was 62.0% to 85.3%, DeepSeek's 61.3% to 84.5%, and GPT-4o's 53.3% to 88.4%. Those broad intervals are an important reminder that the clinical-case denominator is 15, not 6,000.
A control reversed the presentation order of the options and the models continued to favour NCCN, according to the paper. That addresses one familiar source of model bias, but not others. Differences in wording, evidence summaries, publication prominence or training-corpus availability could influence a model without reflecting a considered clinical judgment. The study did not inspect proprietary training data or establish why the preference arose, so explanations about corpus distribution remain hypotheses.[1]
Removing 10 lung-cancer cases altered three of the four conclusions
Ten of the 15 discordant recommendations concerned non-small-cell lung cancer. When the authors repeated the analysis on the remaining five questions, GPT-4o selected NCCN in 39.4% of trials, reversing the direction of its full-set preference. Claude's NCCN share was 56.6% and was not statistically significant. DeepSeek retained a significant but weaker preference at 70.0%. Gemini remained at 100%. This is the study's most consequential qualification because it shows that one disease area drove much of the pooled pattern.
Five remaining cases are too few for broad claims across oncology. The result also cannot be assumed to extend to European, Asian or national guidelines written for different health systems, drug approvals and resource constraints. A useful replication would pre-register a larger, balanced set of disagreements across tumour types and guideline producers, freeze model versions and prompts, and report each clinical question separately before pooling. Otherwise, repeated sampling can make a narrow case set appear more general than it is.[1]
Three oncology experts scored the answers, but this was not a care study
Three oncology specialists who were not involved in generating the questions independently rated responses for accuracy, comprehensiveness, readability and potential harm on five-point scales, resolving initial rating differences by consensus. Mean accuracy scores ranged from 3.51 for Gemini to 3.89 for DeepSeek, and readability from 3.91 to 4.29. Those descriptive scores suggest the outputs often appeared serviceable to experts, but they do not show that clinicians or patients would make better decisions with them.
The study did not randomise doctors to use or avoid an assistant, compare treatment plans with a multidisciplinary tumour board, or follow any patient outcome. It also did not test whether a model appropriately abstained when guidelines reflected different values or evidence thresholds. Potential harm was a reviewer rating of text, not an observed adverse event. Before clinical use, a system would need current source retrieval, provenance, calibrated uncertainty, escalation rules and prospective workflow testing under governance appropriate to a medical device or decision-support tool.[1]
Model settings and time make this a snapshot, not a permanent ranking
The models were tested through consumer web interfaces rather than a reproducible API, and the authors manually entered prompts. Default generation settings differed: GPT-4o and DeepSeek used temperature 1.0, Claude 0.7 and Gemini 0.2. No system prompt was used and no safety-filter refusal occurred. Those choices reflect the available interfaces but complicate direct comparison, because lower temperature can reduce variation and may help explain Gemini's rigid 100% pattern.
The underlying guidelines ended in October 2024, model access occurred in April 2025 and publication followed in October 2026. Both clinical recommendations and model behaviour can change. The next assessment should repeat the protocol with current and archived versions, equalise controllable settings, publish complete prompts and raw outputs, and include retrieval from the latest authoritative guideline text. For patients and clinicians, the safe lesson is not to select a preferred chatbot; it is to verify the source, date and rationale whenever a model turns legitimate guideline disagreement into one confident answer.[1]