Can a medical AI support a clinical board without becoming the decision-maker?
A peer-reviewed German study put OpenEvidence into discussions of 100 real inflammatory-disease cases. Specialists found the answers useful, but complete agreement with the board was limited, prompt tuning was inconclusive and outputs changed over time. The study tested workflow feasibility—not patient benefit or safety.
The short version
- What happened
- Twenty-one specialists assessed OpenEvidence during discussion of 100 consecutive real cases, generating 759 clinician-case ratings; the design tested whether the tool could fit into a board workflow, not whether it improved patient outcomes.
- Who it affects
- Patients should not treat a model-supported discussion as evidence that care is safer or more accurate; this study did not measure outcomes or harm.
- What remains unproven
- Single-centre observational study of 100 inflammatory-disease cases and 21 specialists; there was no randomised control group or external validation.
Keep in mind
Single-centre observational study of 100 inflammatory-disease cases and 21 specialists; there was no randomised control group or external validation.
The final board recommendation was confirmed after clinicians saw the model output, so it was not a fully independent reference standard.
Two physician authors rated concordance; one had helped prepare 10 prompts, and blinding to the evaluation round was not possible.
The proprietary platform did not disclose a model or product version for the first two rounds, limiting reproducibility and attribution of changes over time.
Patient outcomes, clinical harm, hallucinations, citation accuracy and economic effects were not assessed.
Source analysis · 1 linked source. Primary analysed source date: 2 October 2026.
Read the full report · 9 min The evidence, methods, implications and limitations
The study tested a real workflow, not an exam benchmark
Researchers at University Hospital Erlangen and Friedrich-Alexander-Universität Erlangen-Nürnberg prospectively observed 100 consecutive cases discussed by an interdisciplinary board between July and November 2025. The cases involved chronic inflammatory disease and commonly crossed specialty boundaries. Dermatology appeared in 82% of cases, gastroenterology in 52% and rheumatology in 36%, with smaller contributions from haematology-oncology, neurology, paediatrics and nephrology. On average, 2.3 specialties were directly involved in a case.
That makes the paper more informative than a question-answer benchmark built from examination items. These were real clinical problems seen by a functioning board, and 21 specialists took part. Yet it remains a prospective, observational feasibility study at one German centre. It did not randomise cases to AI-supported and unsupported care, compare decisions against later diagnoses, or measure treatment response, complications, time to care, costs or patient experience. The strongest justified conclusion is therefore about whether an AI evidence tool could be inserted into this particular meeting process.[1]
Human discussion came first, but the final reference was not independent of the AI
For each case, clinicians first discussed the question without the model and entered a preliminary recommendation in the hospital documentation system. A prepared, de-identified case prompt was then submitted to the commercial OpenEvidence platform. The board saw the answer, could add or amend points, and only then confirmed its final recommendation. No personal patient data was entered into the service, according to the paper.
This sequence protects an important human-first step, but it complicates interpretation of agreement. The final board recommendation used as the clinical reference could itself incorporate something raised by the model. Agreement with that final decision is therefore not the same as an independent test against an untouched expert answer, and still less a test against patient truth. The authors' design is useful for studying integration; it cannot establish that the model matched an independent gold standard or caused a better decision.[1]
Specialists liked the answers more than the agreement scores imply
The board members completed 759 clinician-case evaluations using eight five-point rating items and a free-text field. The highest mean score was 4.06 for providing clinically relevant information. Recommendation for future use averaged 3.95. Time saved was the lowest-rated item at 3.50 and also varied most. All eight medians were four, suggesting that specialists generally regarded the tool as useful and workable during discussion.
Perceptions were not uniform. Nephrologists gave the most conservative overall ratings, while some other specialties were more positive; the paper warns that only two to six raters represented each specialty and case exposure was uneven, so those differences should not be treated as comparative evidence. Only seven of the 759 evaluations included a free-text comment. Positive scores show acceptability among participating clinicians, but repeated ratings from the same small group are not equivalent to 759 independent users or 759 independent cases.[1]
Full agreement was a minority-to-bare-majority result
Two physician authors independently reviewed outputs for 97 evaluable cases and classified agreement with the final board recommendation as none, partial or complete. One rater judged complete agreement in 40.2% of cases and the other in 50.5%. They judged no agreement in 21.6% and 24.7%, respectively. The raters assigned the same category in 62 of 97 cases; within that consensus subset, 51.6% were complete, 22.6% partial and 25.8% no agreement.
Those figures resist both optimistic and dismissive readings. Partial or complete agreement in a majority suggests the system often surfaced material compatible with expert thinking. But a tool that is not fully aligned in roughly half the cases, and is judged not aligned in around a quarter, cannot safely replace the specialists capable of spotting what is missing. Inter-rater agreement was only moderate by the paper's reported statistics, which also shows that even physicians on the same board did not always agree about how closely an answer matched the recommendation.[1]
Prompt optimisation was not a proven fix
The researchers later reran all cases with a structured prompt asking the model to assess whether information was sufficient, request missing details and then give a justified recommendation. The model asked follow-up questions in every case, but the researchers supplied no additional information before it generated the recommendation. Complete agreement increased numerically to 50.5% and 51.5% for the two raters; in the consensus subset it reached 56.5%.
Across the paired comparison, 29 cases improved, 48 were unchanged and 20 worsened. The mean score rose from 1.222 to 1.361 on the study's zero-to-two scale, but the reported p-value was 0.0642 and the 95% confidence interval for the effect crossed zero. That does not prove the structured prompt had no value; it means this study did not establish a reliable improvement at its stated threshold. Prompt templates may help organise questions, but they are not a substitute for missing clinical information, external validation or outcome testing.[1]
The same cases did not produce stable evidence summaries
When the researchers resubmitted the cases months later, the outputs changed substantially. For the original prompts, mean response length rose by 91.1%, the number of cited references by 93.0%, and only 20.4% of the originally cited references were retained. For the optimised prompts, responses grew by 33.5%, citation counts by 22.1%, and 23.6% of references were retained. The platform is proprietary and continuously updated; the provider did not disclose a model or product version for the first two rounds, so the study cannot separate model updates from other sources of variation.
A same-day comparison was more reassuring but still not identical: changing the prompt left the core clinical recommendation unchanged in 87 of 100 cases. The practical lesson is not that later answers were necessarily worse. Longer responses and newer sources could reflect improvement. It is that a clinician cannot assume yesterday's answer, evidence set or rationale will be reproducible today. Governance needs dated outputs, recorded prompts, accessible citations, version information where available and a clear responsibility for re-checking the underlying evidence.[1]
What this means for patients and clinical teams
For clinicians, an evidence-synthesis model may be most useful as a second-pass literature assistant after experts have framed the case and identified missing facts. It could surface therapeutic options or papers during a meeting, especially when knowledge spans several disciplines. The workflow should preserve the preliminary human recommendation, require a qualified reviewer to open the cited evidence, and document why any model suggestion changed—or did not change—the final plan.
For patients, the paper offers no evidence of better outcomes, fewer errors or safer care. It did not measure hallucinations or clinical harm. People should not infer that use of the model improved their diagnosis or treatment simply because clinicians rated some answers as relevant. Deployment would also require local rules for de-identification, data processing, audit trails, conflicts, consent or notice where applicable, and a route to challenge a decision. The study's ethics approval and avoidance of personal data are useful design choices, not a complete governance framework for routine use.[1]
What evidence would change the assessment
Confidence would increase with preregistered, multicentre trials that keep the comparator independent: teams could be randomised to standard review or model-assisted review, with predefined measures of decision quality, time, resource use, errors, equity and patient outcomes. Cases should span institutions, languages, specialties and levels of complexity. Independent adjudicators should be blinded to the system used, and subgroup results should show where the tool helps, where it adds noise and whether benefits persist after clinicians become familiar with it.
Future evaluations also need fixed model identifiers or archived outputs, reproducibility checks, verified citation accuracy, explicit hallucination and harm review, and disclosure of commercial relationships. This study reported German Research Foundation funding for two authors, open-access support through Projekt DEAL and no competing interests. Its result is valuable precisely because it is bounded: OpenEvidence could be fitted into one expert board and was often perceived as useful, but the evidence does not support autonomous use, a safety claim or a promise of improved care.[1]