A pathology AI answered almost as well without the slide—was it reading the image?
In an unreviewed controlled evaluation, SlideChat scored 74.82% with a pathology slide and 74.41% without it on 1,227 multiple-choice items. The result exposes benchmark shortcuts, not clinical performance.
The short version
- What happened
- SlideChat answered 74.82% of 1,227 multiple-choice pathology questions correctly with the slide and 74.41% after the slide was removed; the difference was statistically indistinguishable from zero.
- Who it affects
- Pathologists and clinical teams should ask whether an answer changes when the image changes, not accept a benchmark score as proof that the model inspected tissue.
- What remains unproven
- The manuscript is an unreviewed preprint using retrospective public data and released checkpoints.
Keep in mind
The manuscript is an unreviewed preprint using retrospective public data and released checkpoints.
The slide-grounding probe covered 1,227 multiple-choice items and two assistants; free-text reporting and other systems may behave differently.
Cross-model feature comparisons are confounded by architecture, data and aggregation differences, and no prospective clinician study was performed.
Source analysis · 1 linked source. Primary analysed source date: 19 October 2026.
Read the full report · 3 min The evidence, methods, implications and limitations
The benchmark answer barely depended on the slide
An unreviewed evaluation found that one pathology vision-language assistant could keep nearly all of its multiple-choice accuracy after researchers removed the pathology slide. On 1,227 re-encoded SlideBench-VQA items, SlideChat scored 74.82% with the slide and 74.41% without it. A mismatched slide also produced no significant gap. Its underlying text model scored 43.52%, close to the benchmark's 40.7% most-frequent-letter baseline, suggesting that instruction-tuning taught answer-pattern shortcuts that did not require visual evidence.
The researchers ran four controlled probes using public checkpoints and data while varying one input at a time. They also shifted answer positions, substituted 'none of the above' and paraphrased questions. Paraphrasing did not materially change average accuracy, but 8–23% of individual answers flipped. That distinction matters: a stable headline score can conceal instability on the same case.[1]
Language added some signal—but not a clinical validation
Language was not useless. Across 1,633 TCGA patients—394 progression-free and 345 overall-survival events—adding a frozen embedding of stage-redacted report text to a slide embedding increased pooled C-index by 0.019 and 0.016 respectively. In lung and renal cancer subtyping, however, zero-shot prompting was statistically indistinguishable from a supervised linear probe when features and pooling were held fixed.
These experiments test benchmark construction and representation quality, not care. The data are retrospective and largely TCGA-derived; the grounding test uses multiple-choice questions, only two slide-level assistants were probed, and free-text behaviour may differ. Several contributors disclosed employment and possible stock holdings in Merck; the accessible manuscript does not state a separate funding source. Prospective multisite studies with pathologist readers, calibrated abstention and tissue-linked citations would be needed before the systems could support clinical decisions.[1]
The distinction that matters
A simple test of whether the model used the slide
The researchers kept the question and answer choices fixed while changing the visual evidence.
Correct slide
Give SlideChat the intended pathology image
74.82% correct
No slide
Remove the image but keep the question
74.41% correct
Interpretation
Compare the two conditions
No measurable grounding gap