Can AI explain bowel leakage safely? A 20-answer study found weak next-step guidance
Three clinicians rated answers from four chatbots to five commonly searched phrases. The median actionability score was 40%, and every model produced inaccurate or irrelevant statements.
The short version
- What happened
- Researchers tested five common US search phrases once in each of four free chatbots, producing 20 answers that three clinicians scored with DISCERN and PEMAT.
- Who it affects
- People who hesitate to discuss bowel leakage may turn to a chatbot first, but high reading levels and weak action steps can leave them unsure what to do or why professional assessment matters.
- What remains unproven
- Only five prompts and 20 one-shot answers
Keep in mind
Only five prompts and 20 one-shot answers
Short Google search phrases may not resemble conversational patient questions
Outputs were generated in February 2025 from model versions now replaced
Poor rater agreement for the understandability measure
No patient comprehension, behaviour, diagnosis or health outcome was measured
Source analysis · 1 linked source. Primary analysed source date: 26 October 2026.
Read the full report · 3 min The evidence, methods, implications and limitations
Clear-looking answers often lacked usable next steps
A peer-reviewed US study found that chatbot answers about fecal incontinence could look organised while still giving patients weak practical guidance. Researchers took five commonly searched phrases from Google Trends and entered each once into ChatGPT 4o mini, Perplexity sonar, Llama 3.2 and Gemini 2.0 flash on 24 February 2025, producing 20 answers.
Three pelvic-health clinicians rated the answers. Median quality was 44 out of 80 on DISCERN, while median PEMAT actionability was 40%. Readability ranged from a median US grade level of 9.4 for Perplexity to 13.0 for Llama—above the fifth- to sixth-grade target cited for consumer information. A colorectal surgeon separately recorded 25 inaccurate or irrelevant statements: three for Llama, four for Gemini, eight for ChatGPT and ten for Perplexity.[1]
A small baseline, not a ranking of today's chatbots
The apparent 77.8% understandability score needs care: agreement between raters was poor, so the authors said understandable content could not be concluded confidently. Model-level quality and readability differed overall, but the study had only five paired prompts; no pairwise comparison remained significant after correction. It cannot support a league table.
The queries were brief search terms rather than detailed patient questions, each answer was sampled once, and all four model versions are now superseded. No patients tested comprehension, followed advice or reported outcomes. For someone seeking help with an embarrassing condition, the useful message is practical: chatbot text may be a starting point for questions, not a substitute for clinical assessment or an authoritative action plan. Repeated testing of current models with patient-written prompts and measured comprehension would change the assessment.[1]
The distinction that matters
What the 20-answer evaluation found
Five short search phrases were entered once into four free model versions; clinicians scored the resulting text rather than testing patients.
Quality
Three raters used the 16-item DISCERN tool
Median 44/80
Actionability
Raters applied the PEMAT next-step criteria
Median 40%
Clinical review
A surgeon checked relevance and accuracy
25 inadequacies across 20 answers