Can an AI medical council give a safer second opinion?
A 300-case preprint found structured multi-stage reports scored better on management planning, but an automated judge, unusual published cases and unequal compute prevent any claim of clinical benefit.
The short version
- What happened
- Eight model pipelines generated reports for 300 resolved NEJM cases: 110 for physician-based judge calibration and 190 for held-out evaluation.
- Who it affects
- Clinicians may benefit from a report that separates evidence, uncertainty and management, but they still need an independent way to verify every consequential recommendation.
- What remains unproven
- This is an unreviewed benchmark of published cases scored mainly by an automated judge. It tested neither clinician decisions nor patient outcomes and did not match compute between the compared workflows.
Keep in mind
This is an unreviewed benchmark of published cases scored mainly by an automated judge. It tested neither clinician decisions nor patient outcomes and did not match compute between the compared workflows.
Source analysis · 1 linked source. Primary analysed source date: 15 October 2026.
Read the full report · 4 min The evidence, methods, implications and limitations
The benchmark found better reports, not better care
A structured, multi-stage AI 'council' produced more highly rated clinical second-opinion reports than a single role-specific response in a new 300-case benchmark. Its clearest advantage was management planning. That finding could help researchers evaluate decision-support systems more realistically than asking only whether a model names the diagnosis. It does not show that an AI council makes clinicians more accurate or patients safer: no clinician used the reports in care, and no patient outcome was measured.
The unreviewed preprint, submitted on 15 October, used diagnostically resolved case records published in the New England Journal of Medicine between 2015 and 2023. Researchers assigned prompted clinical roles, generated independent opinions, then had the same model synthesise them through up to five review rounds. Eight model pipelines were tested under free-form and structured instructions. The correct-answer field was withheld from generation.[1]
How the 300 cases were divided and scored
A fixed 110-case set supplied physician reference ratings and calibration data for an automated LLM judge; the remaining 190 DOI-distinct cases formed the held-out evaluation. Six physicians rated outputs blind to model and format on a 0–3 rubric covering grounding, prioritisation, use of evidence, reasoning, differential diagnosis, final conclusion, management and safety. The calibrated judge then scored the large evaluation set: 6,080 response components and 54,720 criterion outcomes.
Across the held-out cases, structured final reports had higher overall mean scores in seven of eight pipelines. The pooled management score rose from 2.436 for free-form reports to 2.740 for structured reports. Final structured synthesis also scored above the pre-council primary-role output across five shared criteria. Model identity separated quality more strongly than format, and the strongest pipeline in the paper also consumed substantial resources: its mean tokens per case rose from about 65,200 free-form to 132,100 structured.[1]
The distinction that matters
What the benchmark compared
Each route began with the same published case, but the generation process differed.
Single role
One prompted specialty writes an opinion
Role-specific report
AI council
Several prompted roles are synthesised and reviewed
Final multidisciplinary report
Why the council effect cannot be isolated
The council condition combined extra sampling, role prompts, aggregation, critique, revision and more context. There was no compute-matched control, so the study cannot say discussion itself caused the gain. Prompted specialties were simulated viewpoints rather than independently validated specialists and could repeat the same error. The cases also overrepresent rare, difficult diagnoses; titles and DOIs were supplied during synthesis, and public cases may have appeared in model training data.
The automated judge is another boundary. Physician calibration improved agreement on some criteria, but validation error remained high for the composite final-conclusion score. The paper reports no funding or competing-interests statement in the accessible version, while authors list academic, clinical, institute and commercial affiliations. A stronger next step would compare equal-compute pipelines prospectively, then test whether blinded clinicians using the reports make safer decisions. For now, the study supports a richer benchmark—not clinical deployment or patient self-diagnosis.[1]