Why did a pneumonia AI fall from 98% accuracy to 60% on outside data?
A peer-reviewed chest X-ray model scored 98.12% on its internal test set, then 60.42% and 74.30% on two external datasets. The gap is a warning about domain shift—not evidence of a deployable diagnostic tool.
The short version
- What happened
- The model was trained and tested on a balanced public set of 4,800 images; the reported 960-image internal test accuracy was 98.12% (95% confidence interval 97.42%–98.64%).
- Who it affects
- For patients and clinicians, an impressive benchmark score is not enough: a chest X-ray model must work on the scanners, populations and disease mix where care is actually delivered, without hiding confident errors behind an average.
- What remains unproven
- Retrospective public-image benchmark with no clinical workflow or patient outcome test
Keep in mind
Retrospective public-image benchmark with no clinical workflow or patient outcome test
No patient identifiers or demographic metadata for patient-level splitting or subgroup analysis
230 within-class near-duplicate pairs remained in the 4,800-image development set
External repositories combine images from earlier sources and include previously augmented material
Internal inconsistencies in the reported test-set counts and transformer-block description require clarification
Source analysis · 3 linked sources. Primary analysed source date: 26 October 2026. Earliest linked supporting source: 11 June 2020.
Read the full report · 4 min The evidence, methods, implications and limitations
The strong internal score did not travel
A peer-reviewed study of automated chest X-ray classification produced two very different pictures of performance. Researchers trained a self-supervised Vision Transformer to classify COVID-19, pneumonia and normal images. On the paper's balanced 4,800-image development set, 3,072 images were assigned to training, 768 to validation and 960 to a held-out test. The reported internal test accuracy was 98.12%, with a 95% confidence interval from 97.42% to 98.64%; five-fold cross-validation averaged 98.27%.
The researchers also checked for obvious image redundancy. They reported no exact duplicates but found 230 near-duplicate pairs, all within the same class. That is useful scrutiny, yet the images carried no patient-level identifiers or demographic metadata. The split was therefore described at image level, and the paper does not establish that every repeated view or image from one patient stayed in only one partition. Near-duplicate images can still make an internal benchmark easier even when they never cross diagnostic classes.[1]
Two outside datasets exposed domain shift
When the same model was applied to two independently assembled public datasets, accuracy fell to 60.42% and 74.30%. The corresponding areas under the ROC curve were 69.13% and 95.49%, showing that ranking ability and final classification accuracy can diverge. The first repository contains 4,575 images balanced across the three classes. The second contains 603 images: 221 COVID-19, 234 normal and 148 pneumonia images. These are image denominators, not unique-patient counts.
The external sets are not pristine multicentre clinical cohorts. Both combine images from earlier repositories, and the larger set says its COVID-19 class includes 912 previously augmented images. Differences in acquisition, compression, labels, class balance and source population can all alter performance. That variability is precisely why external validation matters: a model can learn source-specific cues that look predictive inside one collection but do not hold when images come from somewhere else.[1][2][3]
The distinction that matters
The same model, three image collections
Internal testing suggested near-perfect classification; outside data changed the result substantially.
Development set
4,800 public images; reported 960-image held-out test
98.12% accuracy
External set 1
4,575 images assembled from several repositories
60.42% accuracy
External set 2
603 images across three unequal classes
74.30% accuracy
Clinicians should read the gap, not the 98% headline
This was a retrospective computing experiment, not a clinical trial. No radiologist comparison, diagnostic workflow, treatment decision or patient outcome was measured. The task also grouped COVID-19, other pneumonia and normal images into three labels rather than testing the full range of conditions encountered in practice. The authors therefore describe the model as research-stage and call for geographically diverse, prospective multicentre evaluation.
The publication record needs clarification too. Its methods specify a 960-image test set, while the reported confusion-matrix counts total 1,920 images; one architecture description says four transformer blocks and another says eight. Those inconsistencies do not erase the external-performance gap, but they reduce confidence in the precise internal comparison. A clinically useful assessment would require a frozen model, patient-level sampling, verified labels, prespecified thresholds, calibration, failure reporting and comparison with ordinary radiology practice at each intended site.[1][2][3]