A mammography AI wrote fluent reports—did it get the clinical judgement right?
A preregistered evaluation found that synthetic pretraining did not cause the model's apparent gain. On 49 leak-free test images, strong language-overlap scores coexisted with BI-RADS agreement no higher than κ=0.14.
The short version
- What happened
- Researchers generated 400 synthetic reports, retained 390, pretrained MedGemma-4B and then fine-tuned it on 407 real reports.
- Who it affects
- Radiologists and patients need generated reports to preserve clinically important findings, not merely reproduce plausible phrasing. A system that omits abnormalities could make a smooth-looking workflow less safe rather than more useful.
- What remains unproven
- The leak-free test subset contained only 49 images and the split was image-level rather than patient-level.
Keep in mind
The leak-free test subset contained only 49 images and the split was image-level rather than patient-level.
The study did not include prospective deployment, patient outcomes or a blinded expert reader study.
Results concern one model family and selected public datasets; they do not establish performance across hospitals or populations.
Language-overlap metrics and BI-RADS agreement assess different properties and should not be treated as interchangeable.
Source analysis · 1 linked source. Primary analysed source date: 23 October 2026.
Read the full report · 3 min The evidence, methods, implications and limitations
The first result looked better than the model's clinical performance
The team converted structured mammography annotations into 400 synthetic reports, retained 390, pretrained MedGemma-4B and then fine-tuned it on 407 real reports. On a leak-free test subset of 49 images, the system reached ROUGE-L 0.656, METEOR 0.666, CIDEr 0.649 and BERTScore 0.917. Those metrics reward similarity to reference wording; they do not by themselves establish that a report gives the right clinical category or recommendation.[1]
A controlled rerun removed the synthetic-data advantage
The apparent gain came from comparing adapters with unequal capacity. In a preregistered analysis covering 60 runs—three corpus sizes, three adapter capacities and five initialisations—no matched configuration favoured the two-stage synthetic-to-real approach. Capacity alone reproduced the gap. Within one comparison, ROUGE-L improved by 0.058 while BI-RADS agreement and sensitivity fell. Agreement never rose above κ=0.14.[1]
The distinction that matters
Why the headline language scores were not enough
The paper tested three different links in the evidence chain.
Synthetic reports
Pretrain on converted annotations
390 reports retained
Matched experiment
Hold adapter capacity constant
No synthetic-pretraining advantage
Clinical check
Compare BI-RADS judgement
Agreement never exceeded κ=0.14
Fluency cannot substitute for radiology validation
The useful lesson is negative: a report generator can sound closer to a reference while becoming less clinically accurate. The study also found omission, rather than invented findings, as the main error on normal studies. But this was a small, image-level evaluation using public datasets, not a patient-level clinical trial or an expert reader study. Before generated reports could support care, evidence would need patient-separated validation, radiologist review, robust abnormal-case performance and prospective testing in real workflows.[1]