Can a respiratory chatbot improve lay triage?
A preregistered Chinese trial found that 2,176 compliant participants answered simulated respiratory questions more accurately with a task-constrained GPT-4o chatbot than with ordinary web search. The largest gain was diagnosis; triage accuracy remained modest, and the trial did not measure real care-seeking or patient outcomes.
The short version
- What happened
- The trial randomized 2,400 adults across 24 Chinese healthcare systems; the primary compliant-case analysis included 2,176 questionnaires, split evenly between LungDiag assistance and conventional mobile web search.
- Who it affects
- A well-constrained chatbot could help people interpret respiratory symptoms more accurately than fragmented search results.
- What remains unproven
- The primary analysis excluded 224 non-compliant questionnaires and is therefore a modified intention-to-treat analysis, although all-randomized sensitivity analyses were directionally consistent.
Keep in mind
The primary analysis excluded 224 non-compliant questionnaires and is therefore a modified intention-to-treat analysis, although all-randomized sensitivity analyses were directionally consistent.
Participants answered simulated case questions; the study did not observe real care-seeking, diagnostic delay, service use or patient outcomes.
The control used conventional search with dedicated AI and generative-search functions disabled, so the result does not compare LungDiag with current AI-enhanced search products.
Four-category triage accuracy remained modest, and binary high-urgency specificity was 62.1% with LungDiag.
The exact prompt, retrieval configuration, knowledge layer and mini-program source were not publicly released, limiting full independent reproduction.
The trial was conducted in China through WeChat and may not transfer to other languages, health systems, digital-literacy levels or emergency pathways.
Source analysis · 3 linked sources. Primary analysed source date: 5 October 2026. Earliest linked supporting source: 17 April 2025.
Read the full report · 7 min The evidence, methods, implications and limitations
The trial compared a constrained chatbot with ordinary search
Researchers recruited 2,400 adults aged 18 to 75 without formal medical training across 24 Chinese healthcare systems between April and June 2025. Participants were randomly assigned one-to-one, stratified by centre, to use either LungDiag or conventional mobile web search. Dedicated AI and generative-search functions were disabled in the control arm. Outcome assessors were blinded, but participants, data collectors and analysts could not be because the two tools were visibly different.
LungDiag was not an unrestricted public chatbot. It used GPT-4o through a WeChat mini-program, a respiratory knowledge layer and task-specific routing for risk or aetiology, preliminary diagnosis and four-level triage. The study bank contained 88 simulated respiratory cases: 44 risk-behaviour vignettes and 44 diagnosis-and-triage vignettes, with the triage set evenly divided among self-care, care within one week, care within one day and emergency care. Trial cases and answer keys were kept out of the retrieval index.[1][2]
The primary result was an item-level accuracy difference
The prespecified compliant-case analysis retained 2,176 questionnaires, 1,088 in each arm. Researchers analysed correct and incorrect answers with generalized linear mixed-effects models that included random intercepts for participant and item, recognizing that repeated answers from one person and differently difficult questions are not independent. LungDiag's model-based overall accuracy was 70.0%, with a 95% confidence interval from 65.2% to 74.8%; the search arm reached 55.4%, with an interval from 49.7% to 61.1%.
That 14.6-percentage-point adjusted difference was statistically clear, with a 95% confidence interval from 12.2 to 17.0 percentage points and a risk ratio of 1.26. Sensitivity analyses included all 2,400 randomized participants under different assumptions for assigned-but-unanswered items, and the direction of the result remained. The main analysis was nevertheless a modified intention-to-treat approach because non-compliant questionnaires were excluded, so the cleanest interpretation is performance among participants whose data met the protocol's usability rules.[1][3]
Diagnosis improved more than triage
The aggregate result hides an important difference among tasks. Preliminary diagnosis accuracy was 81.2% with LungDiag and 54.5% with search, a 26.7-point gap. Risk or aetiology questions also favoured the chatbot, at about 78.5% versus 67.1%. Triage was the weak point: 53.4% with LungDiag and 47.1% with search. A tool can therefore produce a plausible disease label more often while still struggling to recommend the appropriate urgency.
Each participant in the primary analysis answered four triage items, producing 4,352 triage responses per arm. When the four categories were simplified into high urgency—within one day or emergency—versus lower urgency, LungDiag reached 85.6% sensitivity and 62.1% specificity. Search reached 80.9% sensitivity and 56.0% specificity. The chatbot found more urgent cases, but its moderate specificity implies many lower-urgency scenarios would still be escalated. This experiment did not measure whether that caution reassures users, crowds services or delays a different necessary action.[1]
Better answers took longer and remained simulated
LungDiag users took a median 527 seconds to complete the questionnaire, compared with 461 seconds for search. The additional 66 seconds is not necessarily a defect: a structured conversation may collect information that a quick search misses. But time, cognitive load and abandonment matter in consumer health tools, especially when someone is breathless or anxious. The paper reports task completion, not whether users understood uncertainty, trusted the advice appropriately or could act on it under real pressure.
The intervention also answered fixed vignettes rather than observing genuine symptom evolution, examination findings or care pathways. Participants knew they were completing a study exercise, not deciding whether to call emergency services for themselves or a relative. The trial did not measure subsequent attendance, clinician diagnosis, treatment, diagnostic delay, health outcomes or service use. Its strongest conclusion is that this constrained interface helped lay participants answer simulated questions—not that it safely diagnoses or triages patients.[1]
Reproducibility is partial rather than complete
The investigators released de-identified participant-level data, the primary mixed-model code, sensitivity analyses and figure scripts on GitHub and Zenodo. That is unusually useful for checking the statistics. They did not release the exact internal prompt, retrieval settings, chunking and reranking thresholds, the complete LungDiag knowledge layer or the trial-specific mini-program source. Those proprietary boundaries mean an independent team cannot recreate the intervention exactly from the public package.
The study was supported by Chinese public research grants. The interface was hosted by Xingyun and used paid API access routed through APIYI; the authors say neither vendor influenced design, conduct, analysis or reporting and report no employment, equity, patent, royalty or revenue-sharing relationship. The LungDiag-derived knowledge resource is owned and maintained by the authors' group. None of these disclosures invalidates the result, but they matter when the evaluated product cannot be fully reproduced or transferred unchanged to another health system.[1][3]
What would change the assessment
Confidence would increase with a locked, fully specified system tested against alternative AI-enabled search tools rather than a control from which generative functions were removed. Replication should use new case banks, languages, literacy levels and countries; separately report under-triage and over-triage; and verify performance after model or knowledge-base updates. Independent security, privacy and failure-mode testing would be essential for a messaging platform that handles health descriptions.
The decisive next step is a prospective real-world study with safety oversight. It should recruit people actually deciding what to do, compare advice with final clinician diagnoses and observed care pathways, and measure emergency escalation, delay, unnecessary attendance, anxiety, adherence and clinical outcomes. Until then, the trial is strong evidence that a constrained respiratory chatbot can improve simulated diagnostic recognition over ordinary search, but only limited evidence that it can safely guide real people to the right level of care.[1][2]