Can a strong cancer-care answer survive the whole patient pathway? A new benchmark found strict success peaked at 23%
A 5 November follow-on benchmark combined guidelines, pathology, MRI, longitudinal records and 100 patient episodes. One model answered 89.9% of care decisions correctly yet achieved 11% strict patient-level success once image interpretation was included. The study remains retrospective and unreviewed.
The short version
- What happened
- The original 26 October preprint used 941 phase-specific units from 405 patients at two Guangzhou cancer centres and found that diagnosis was consistently stronger than staging and follow-up across 30 models.
- Who it affects
- Patients need decisions grounded in the correct images and longitudinal record, not merely a plausible recommendation that happens to match a guideline.
- What remains unproven
- Both sources are unreviewed preprints; conference acceptance of NPCBench does not replace full peer review of the arXiv version.
Keep in mind
Both sources are unreviewed preprints; conference acceptance of NPCBench does not replace full peer review of the arXiv version.
The original benchmark used two Guangzhou centres, while NPCBench is primarily single-centre and focused on one cancer pathway.
NPCBench uses assigned pathology in 75 of 100 CarePath episodes and only 25 fully matched episodes for evidence-replacement analysis.
Strict success is affected by class balance and the number of decisions in an episode.
Open-ended consultation scoring depends on an automated judge whose expanded-audit sensitivity was 51.8%.
No prospective clinician-use comparison, treatment outcome, toxicity outcome or patient benefit was measured.
The accessible NPCBench manuscript does not state funding or competing interests; the October paper includes disclosed company affiliations.
Multi-source analysis · 2 linked sources. Primary analysed source date: 26 October 2026. Latest linked supporting source: 5 November 2026.
Read the full report · 8 min The evidence, methods, implications and limitations
What changed: the new study links correct answers to complete patient episodes
The original October benchmark showed that the same language model could look capable at diagnosis and much weaker at staging, treatment toxicity or follow-up. A new, separately submitted preprint now asks a harder question: when pathology, MRI interpretation and care decisions are connected for the same person, does a model remain correct across the pathway? NPCBench combines six progressive task levels with open-ended consultation rubrics. It includes 4,777 component and consultation items and a de-identified 100-patient CarePath cohort containing 417 scored diagnostic and care tasks.
The central finding is that high local scores can coexist with low patient-level consistency. Gemini-3.1 produced the highest decision-level care accuracy, answering 195 of 217 care decisions correctly, or 89.9%. Yet it achieved strict success for only 11 of 100 patient episodes once every scored care decision, pathology assessment and MRI T-stage label had to be correct. Claude-4.6 had lower care accuracy at 80.6% but the highest strict patient-level success at 23%, with an exact 95% confidence interval of 15.2% to 32.5%. Neither result is a clinical outcome or deployment validation.[1][2]
How the two benchmarks differ without becoming duplicate stories
The 26 October study reconstructed six decision points—work-up, diagnosis, staging, treatment, toxicity and follow-up—from 405 people treated at two Guangzhou cancer centres. It created 941 patient-phase units and 17,688 patient-specific criteria, then compared 30 models. Diagnosis was the strongest phase for every model, while staging produced the greatest explicit decision-error rate and follow-up often lost required actions. The full benchmark relied mainly on an automated judge, with three physicians auditing a selected subset of ratings.
NPCBench is a follow-on dataset rather than a rerun of that headline. It builds a clinical-apprenticeship ladder: 250 guideline-memory questions, 185 atomic application vignettes, pathology and MRI tasks, 165 planning questions covering 84 patient groups, 305 longitudinal questions across 100 trajectories, a 100-patient linked CarePath cohort and 103 open-ended consultations. Thirty medical and general models completed the text tasks; 21 multimodal models also completed image-grounded levels. The paper says the dataset was accepted to the NeurIPS 2026 Evaluations and Datasets Track, but the arXiv manuscript itself remains a preprint.[1][2]
The distinction that matters
Local accuracy versus strict patient-level success
NPCBench counts a patient as strictly successful only when all scored care decisions, pathology and MRI T-stage answers are correct.
CarePath
100 patients
417 diagnostic and care tasks
Gemini-3.1
89.9% care accuracy
11% strict success
Claude-4.6
80.6% care accuracy
23% strict success
Not measured
Clinical deployment
No patient benefit or harm outcome
What the 100-patient denominator contains
CarePath contains 100 pathology assessments, 100 MRI T-stage assessments and 217 selected care decisions. Strict success is demanding by design: every scored element for a patient must be correct. That endpoint reveals errors hidden by an average across questions, but it also becomes harder as an episode contains more decisions. The authors report results separately by decision count and acknowledge sensitivity to episode length and class balance. In 75 episodes, pathology specimens were assigned rather than patient-matched; only 25 episodes had matched pathology for controlled evidence-replacement analyses.
The multimodal evidence tests were broader than the linked cohort. Pathology included 69 overview cases, 39 positive and 30 negative. MRI testing included 200 sequence-recognition questions, 400 T-stage cases and 400 N-stage cases. A 100-patient anatomical panel produced 2,700 structure cards covering 18 base structures. Complete invasion correctness reached at most 5% on that panel, suggesting that a correct stage label often lacked fully correct anatomical support. A constant T3 prediction could score 58% in CarePath because of class imbalance, above the best model’s raw T-stage accuracy there, while balanced accuracy told a less flattering story.[2]
Why accurate care recommendations did not guarantee evidence-grounded care
For Gemini-3.1, 72 of 100 patients had all care decisions correct despite at least one incorrect pathology or MRI T-stage label. Claude-4.6 showed the same mismatch in 41 patients. Conversely, Hulu-Med-32B had 38 patients whose two diagnostic labels were correct but at least one care decision was wrong. Those patterns show why decision accuracy and evidence accuracy should not be treated as interchangeable. A model can arrive at a guideline-consistent action for the wrong apparent reason, or correctly read evidence yet fail to carry it into the next decision.
The authors also replaced predicted diagnostic evidence with reference pathology, T-stage and invasion findings for 25 patients, holding other context fixed. Across 14 local models and 146 care questions, patient-averaged care accuracy changed from minus 3.2 to plus 4.1 percentage points; 12 of 14 intervals included zero. Better supplied evidence therefore did not consistently improve the final care answer under this setup. That does not prove models ignore evidence in every clinical task. It shows that benchmark performance can be weakly coupled across components, so product testing must trace how evidence changes a recommendation rather than merely scoring the final sentence.[2]
Human review improves the benchmark, while automated judging remains a limitation
NPC oncology specialists reviewed clinical validity, reference answers, supporting evidence and transitions between decisions. Multidisciplinary reviewers in oncology, pathology and radiology assessed whether the evidence was sufficient and whether imaging labels and care references were defensible. Retrospective clinical-data use was approved under IRB protocol B2026-465-01, direct identifiers were removed and dates were expressed relative to the clinical course except where a guideline edition required a date. These safeguards make the benchmark more credible, but they do not turn it into a prospective care study.
Open-ended consultations were scored by automated judges calibrated against physicians. In an expanded audit of 1,013 criterion labels from 54 responses across nine models and six cases, the primary judge agreed with physicians on 70.5% of criteria. Sensitivity was 51.8% and specificity 85.8%, meaning it was substantially better at recognizing negative than positive labels under that coding. The authors acknowledge dependence on the automated judge and a six-case expanded audit. The original October benchmark had a different selected physician audit, so neither paper supports treating judge scores as ground truth without qualification.[1][2]
What this means for clinicians—and what would change the assessment
For oncology teams, the combined lesson is not that one vendor model wins. It is that a model should be tested at the point where its output affects care: evidence extraction, staging, planning, treatment changes, toxicity management and follow-up. A single aggregate score can hide a stage-specific failure, while a correct final action can conceal faulty image interpretation. Interfaces should expose the supporting evidence, require clinician confirmation and preserve the longitudinal record so contradictions can be reviewed. These systems should not diagnose, recommend treatment or triage patients without physician supervision.
The evidence would become more persuasive with external, multicentre cohorts; fully patient-matched images; prespecified scoring; larger physician audits; prospective clinician-in-the-loop testing; and measured effects on time, error detection, decision quality and patient outcomes. The current studies are retrospective and disease-specific, and NPCBench is primarily single-centre. Nasopharyngeal cancer also has distinctive epidemiology and care pathways, limiting transfer to other cancers. The update strengthens the case for pathway-level evaluation. It does not show that any model improves survival, reduces toxicity or can safely manage a real patient.[1][2]