A dental guideline improved one chatbot’s answers—but the test graded against the same guideline
Claude gained 0.39 points across 18 fictional jaw-risk scenarios after receiving the complete guideline. No patient, diagnosis or treatment outcome was tested.
The short version
- What happened
- Researchers presented ChatGPT, Claude and DeepSeek with 18 fictional endodontic scenarios involving medication-related osteonecrosis of the jaw, before and after supplying the full 2022 AAOMS position paper.
- Who it affects
- Patients taking antiresorptive medicines need individual assessment; a chatbot benchmark cannot weigh their dental findings, medicine history and jaw-risk factors safely on its own.
- What remains unproven
- Only 18 fictional, standardised scenarios
Keep in mind
Only 18 fictional, standardised scenarios
Cases, prompting material and scoring reference came from the same guideline
Only two specialist raters with consensus resolution
No external case set or real clinical records
No diagnosis, treatment, safety or patient outcomes
Results can change with model versions and settings
Source analysis · 1 linked source. Primary analysed source date: 26 October 2026.
Read the full report · 3 min The evidence, methods, implications and limitations
Supplying the guideline narrowed one model gap
A peer-reviewed study tested three chatbots on 18 fictional endodontic scenarios involving patients taking antiresorptive medicines, where treatment choices must balance tooth preservation against medication-related osteonecrosis of the jaw. ChatGPT, Claude and DeepSeek answered the cases first without and then with the complete 2022 American Association of Oral and Maxillofacial Surgeons position paper in context.
Two endodontic specialists independently scored anonymised answers across five clinical-decision domains on a 0–5 scale, then resolved disagreements by consensus. ChatGPT's baseline mean was 4.68, compared with 4.29 for Claude. After guideline prompting, Claude rose by 0.39 points to 4.68 and the earlier difference was no longer statistically significant. DeepSeek remained lower than the other two systems. Some domain-level sensitivity tests did not confirm the phase effect for consultation and medication management.[1]
Agreement with a supplied document is not clinical validation
The apparent gain is circular in an important way: researchers built the fictional scenarios and reference standard from the same guideline later supplied to the models. Better agreement therefore shows retrieval and application of that document under controlled conditions, not independent diagnostic reasoning. The same 18 cases were used in both phases, and there were no real patients, records, imaging findings, uncertain histories or consequences for treatment.
For dentists and patients, this is evidence that current guidance can make a chatbot's answer more guideline-concordant—not permission to delegate decisions. The safe use would remain a clinician-owned check with current local guidance, drug history, dental findings and specialist consultation where required. Confidence would require externally written cases, locked model versions, error analysis, prospective evaluation and evidence that support improves decisions without delaying care. The authors reported university support for the publication charge and no competing interests.[1]
The distinction that matters
What changed when the full guideline was supplied
The benchmark measured agreement with a guideline-derived reference, not patient outcomes.
Baseline
Models answered 18 fictional cases
Claude mean 4.29; ChatGPT 4.68
Guideline supplied
Same cases plus complete AAOMS paper
Claude rose to 4.68
Clinical care
No real patients or decisions
Safety and benefit untested