A local model simplified radiology reports—but 45 contained clinically relevant hallucinations
Gemma-3 made Dutch reports easier to read without sending text to a cloud service. Expert reviewers still found potentially consequential invented content in one in eight reports.
The short version
- What happened
- Researchers randomly sampled 360 Dutch primary-care reports, 60 from each of six X-ray and ultrasound examination types, from a database of 5,534.
- Who it affects
- Plain language could help patients understand a radiology conclusion, but an invented or altered detail may mislead them. A clinician must remain responsible for checking every patient-facing rewrite.
- What remains unproven
- Retrospective proof of concept
Keep in mind
Retrospective proof of concept
One institution
One model and prompt
Dutch reports only
Mostly one expert reviewer
No patient comprehension test
No prospective workflow
Reproducibility not tested
Source analysis · 1 linked source. Primary analysed source date: 29 October 2026.
Read the full report · 4 min The evidence, methods, implications and limitations
Readable conclusions performed better than full reports
A locally hosted language model produced simpler Dutch radiology reports without transferring report text to an external cloud, but expert reviewers found potentially clinically relevant hallucinations in 45 of 360 reports. The retrospective proof-of-concept study used Gemma-3 12B on a random sample drawn from 5,534 primary-care reports at one Dutch institution. The sample contained 60 reports for each of six common examinations: chest, knee, lumbar spine and shoulder X-rays, plus shoulder and abdominal ultrasound.
Researchers evaluated 359 rewritten main sections and 350 rewritten conclusions. Seventy-one per cent of main sections and 96% of conclusions met the prespecified expert threshold for accuracy and completeness, both scored at least four on five-point scales. Yet only 48% of combined main-and-conclusion rewrites also passed the readability threshold. Conclusions were the stronger candidate: their median accuracy score was five, compared with four for the main report.[1]
Easier words came with invented details
Readability improved sharply on the Dutch Flesch–Douma scale, with an average increase of 54.2 points, although rewritten reports also became a median 65 words longer. A readability formula counts features such as word and sentence difficulty; it does not show that a patient understood the result, felt less anxious or made a better decision. No patients evaluated the output.
Reviewers documented 69 hallucinations across 58 of 359 main-section rewrites. They judged 54 events in 45 reports—12.5% of the sample—as potentially clinically relevant. The paper's examples include adding an incorrect explanation or changing the meaning of a finding. A patient-friendly version can therefore sound clearer while becoming less faithful to the clinical report. Local hosting limits one privacy risk, but does not solve the accuracy problem.[1]
The distinction that matters
What the local model achieved—and risked
Quality, readability and fidelity were separate tests.
Quality
71% of main sections; 96% of conclusions passed
Conclusions performed better
Readability
48% passed quality plus readability together
Easier language was not universal
Safety
45 reports had clinically relevant hallucinations
Clinician review remains necessary
The evidence is promising, not deployment-ready
Most reports were rated by one expert. Four reviewers independently scored a 36-report subset, and agreement was mixed: moderate for conclusion accuracy, lower for main-report accuracy and poor for completeness. The experiment used one model, one prompt, one institution and one language. It did not formally test whether repeated runs would reproduce the same rewrite, and it did not place the system in clinical care.
For patients, a reliable plain-language conclusion could make a portal result less opaque and give a better starting point for questions. For radiologists and GPs, unchecked invented details could create confusion, extra calls or unsafe reassurance. The study did not measure either effect, so reduced workload and better understanding remain hypotheses rather than outcomes. Confidence would change with patient comprehension testing, independent multi-reader evaluation, repeated-output testing and a prospective workflow that requires clinician approval and records corrections. External studies should include abnormal and high-stakes findings, other languages and comparisons with human-written summaries. They should also report how often clinicians reject or substantially rewrite the model's draft. The present result supports cautious development—especially for conclusion sections—not automatic publication to patients.[1]