Microsoft's diagnostic agent beat doctors on a benchmark—what has it proved?
A peer-reviewed Science paper reports higher accuracy and lower estimated testing costs on 304 difficult case reports and 325 emergency-department records. It is still a retrospective simulation, not evidence that the system improves care for real patients.
The short version
- What happened
- The benchmark turns 304 New England Journal of Medicine case reports and 325 de-identified emergency-department admissions into step-by-step diagnostic encounters.
- Who it affects
- A reliable sequential assistant could help clinicians consider alternatives and avoid low-value tests. A system that is wrong, overconfident or poorly integrated could instead delay treatment, order needless procedures or shift extra checking work onto already stretched teams.
- What remains unproven
- This was a retrospective benchmark study, not a prospective trial or evidence of improved patient outcomes.
Keep in mind
This was a retrospective benchmark study, not a prospective trial or evidence of improved patient outcomes.
The NEJM set deliberately contains diagnostically difficult published cases and does not represent the frequency of conditions in routine practice.
The gatekeeper can synthesise results absent from the original case, while another language model judges the answer; both components can introduce error or favour model-native reasoning.
The physician and AI conditions differ from team-based clinical care, where specialists, records, repeated observation and local workflows influence decisions.
The research is Microsoft-led and supports a Microsoft system. The accessible Science page was blocked, so final funding and conflict declarations could not be independently read in this review.
Multi-source analysis · 4 linked sources. Primary analysed source date: 22 October 2026. Earliest linked supporting source: 27 June 2025.
Read the full report · 4 min The evidence, methods, implications and limitations
The strongest result is performance inside a harder benchmark
A Microsoft-led study published in Science on 22 October reports that its MAI Diagnostic Orchestrator, or MAI-DxO, reached 83% diagnostic accuracy on complex New England Journal of Medicine cases. Generalist physicians scored 36% when allowed textbooks and internet references and 22% without them. On a second set drawn from de-identified US emergency-department records, Microsoft reports 80% for the system and 47% for physicians. Those are large benchmark differences, but they do not show that an AI safely outperforms clinical teams at the bedside.[1][2][3]
The system asks questions and orders tests before answering
SDBench converts 304 difficult case reports and 325 emergency-department admissions into interactive encounters. The diagnostic agent starts with a short summary, then requests history, examination findings or tests before committing to an answer. MAI-DxO makes one model play five roles: maintaining a differential diagnosis, choosing tests, challenging early assumptions, checking cost and enforcing consistency. The earlier public manuscript compared 21 US and UK generalist physicians, with a median 12 years of experience; the peer-reviewed release adds the reference-enabled and emergency-care comparisons.[2][3][4]
The distinction that matters
How the benchmark differs from an exam—and from real care
The evaluation adds sequential decisions and estimated costs, while still operating on retrospective records through model-mediated components.
Start
Show a short case summary
Diagnosis is initially hidden
Investigate
Ask questions or order tests
Gatekeeper reveals or synthesises findings
Commit
Submit one diagnosis
Model judge scores accuracy; costs are estimated
Estimated cost improved, but the simulation supplies part of the world
Microsoft says a cost-focused configuration retained 81% accuracy on the difficult cases at less than half the estimated testing cost of comparison systems. Costs came from US procedural-price mappings, not hospital bills. More importantly, an AI gatekeeper reveals information from the source case and generates realistic, case-consistent results when a requested test was not recorded. Another model judges whether the final diagnosis matches the reference answer. Physician review found strong agreement with that judge in the earlier manuscript, but model-generated findings and model scoring still make this a simulation rather than a prospective clinical trial.[2][3][4]
The next study must test patients, workflow and harm
No patient was diagnosed or treated by MAI-DxO in this evaluation. Microsoft says it plans a prospective study with Beth Israel Deaconess Medical Center in which patients will interact with the system and independent physicians will assess performance. That work needs prespecified safety outcomes, subgroup analyses, missed and delayed diagnoses, unnecessary testing, clinician workload and the consequences of disagreements—not accuracy alone. Until those results exist, the paper supports a better way to test sequential reasoning and a promising research system, not a clinical product or a replacement for doctors.[1][2][3]