Four chatbots covered less guidance when simulated teenage depression was severe
A peer-reviewed audit used 1,600 scripted conversations to test ChatGPT-5, ChatGPT-4, Claude 4.5 and Gemini 2.5. Coverage was stronger for anxiety than depression and for moderate than severe presentations—but the benchmark did not test treatment, safety or real teenagers.
The short version
- What happened
- Researchers developed AMHART with clinician review, reducing 27 candidate items to an 18-item framework and using a 14-item yes/no/not-applicable grading rubric.
- Who it affects
- Teenagers may turn to general chatbots before speaking to an adult or clinician. This study gives evaluators a structured way to look for omissions, but it does not validate any chatbot as mental-health care.
- What remains unproven
- Synthetic personas and scripted conversations rather than real adolescents
Keep in mind
Synthetic personas and scripted conversations rather than real adolescents
Coverage benchmark, not a measure of diagnosis, treatment or symptom change
Automated evaluators were not statistically calibrated against the full audit set
Four named model versions at one point in time
No crisis outcome, downstream behaviour or clinician decision measured
US-developed clinician framework may not transfer unchanged across health systems and cultures
Source analysis · 1 linked source. Primary analysed source date: 28 October 2026.
Read the full report · 3 min The evidence, methods, implications and limitations
The benchmark asked whether replies included clinician-identified guidance
The researchers built the Adolescent Mental Health AI Response Tool, or AMHART, to make a narrow question measurable: when a teenager describes anxiety or depression, does a chatbot response cover elements clinicians consider relevant? Clinician review reduced 27 candidate items to an 18-item framework spanning seven domains. A 14-item yes/no/not-applicable rubric was then used to grade model replies.
The main test used eight fictional adolescent persona cards and generated 1,600 conversations with ChatGPT-5, ChatGPT-4, Claude 4.5 and Gemini 2.5. Each conversation contained ten chatbot responses across 20 turns. Coverage was greater for anxiety than depression and for moderate than severe presentations. That pattern matters because a fluent response may look supportive while omitting guidance that becomes more important as symptoms worsen.[1]
Coverage is not the same as safe or effective mental-health care
Three independent language-model evaluators graded outputs after an initial comparison with 84 human reference labels. None was fine-tuned, statistically calibrated or shown the human labels. Automated grading makes a large audit possible, but disagreements or shared model blind spots can still shape the scores.
The study did not involve real adolescents, clinicians making decisions, diagnoses or treatment outcomes. It therefore cannot show that one chatbot prevents harm, improves symptoms or should be used as a therapist. The practical lesson for families, schools and health services is narrower: product checks should test severe as well as moderate scenarios and inspect what is omitted, not only whether a reply sounds empathetic. Prospective studies with human review, crisis pathways and measured outcomes would change the assessment.[1]
The distinction that matters
What the AMHART audit measured
The study connected clinician-informed criteria to synthetic conversations, then stopped before clinical outcomes.
Framework
Clinicians reviewed candidate response elements
18 items across seven domains
Synthetic audit
Four chatbots answered fictional adolescent personas
1,600 conversations graded for coverage
Not tested
No real patient or treatment follow-up
Safety and clinical benefit remain unknown