When reliable clinical AI is wrong, who catches the error?
In a 14-reader experiment, radiology residents accepted 52% of the reliable model's incorrect suggestions. Three neonatologists overrode 91.7% of deliberately flawed advice, but that specialist result is too small to generalise.
The short version
- What happened
- Fourteen clinicians interpreted 125 neonatal radiographs in a crossover experiment, producing 1,750 decisions across unaided, reliable-AI and error-injected-AI conditions.
- Who it affects
- In neonatal emergencies, a confidently missed finding or false alarm can delay surgery or trigger unnecessary escalation. Trainees need protected opportunities to practise independent verification before AI enters routine workflow.
- What remains unproven
- Retrospective web-based experiment with 14 readers and only three neonatologists.
Keep in mind
Retrospective web-based experiment with 14 readers and only three neonatologists.
The 125-case external set targeted one neonatal radiographic emergency; results may not transfer to other tasks or clinical environments.
Deliberate model errors and voluntary saliency-map use do not reproduce the full distribution of failures or attention in live care.
Source analysis · 1 linked source. Primary analysed source date: 20 October 2026.
Read the full report · 3 min The evidence, methods, implications and limitations
Wrong advice exposed a trainee vulnerability
Researchers recruited six pediatric radiologists, three neonatologists and five radiology residents. Each interpreted cases in counterbalanced sessions separated by six weeks. The external set contained 125 radiographs from 11 hospitals, including 40 positive cases; 41 cases were unaided, 40 used the reliable model and 44 used a deliberately error-injected model. Readers were not told that two reliability levels existed.
The reliable model's external AUC was 0.948, although it was 0.861 on the 40 cases assigned to that study arm. It did not significantly improve any group's accuracy over unaided reading. When it was wrong, residents accepted 52% of its suggestions, compared with 20% for pediatric radiologists. That difference remained significant after the authors' multiple-comparison correction.[1]
Expert vigilance is not a safety guarantee
Against deliberately flawed advice, neonatologists overrode 66 of 72 errors, or 91.7% (95% confidence interval 83.0%–96.0%). Pediatric radiologists overrode 85.4% and residents 81.7%. Disagreeing with AI also took longer across every group. The study therefore shows why hospitals should test the human-plus-system workflow under plausible failure, not only report a model's standalone AUC.
But the experiment was retrospective and web-based, used one narrow emergency finding and included only three neonatologists. Its error-injected model was engineered to fail in specific ways, while saliency-map use was voluntary and confounded by case difficulty. No patient outcome or live clinical workflow was tested. Larger prospective studies should vary failure type and frequency, measure overrides and delays, and test whether training reduces automation bias without creating blanket distrust.[1]
The distinction that matters
The safety test the study adds
Standalone model accuracy leaves the most important failure question unanswered.
Model test
Measure AI accuracy alone
Finds technical errors
Team test
Give wrong advice to clinicians
Measures acceptance and override
Clinical test
Run prospectively in care
Still needed for patient benefit and harm