Did a specialist health chatbot give safer workplace advice than a general AI assistant?
Nine experts rated 60 answers to sedentary-work questions, while 654 Chinese office workers tested the specialist tool. It scored higher than Doubao on several answer-quality measures, but the study did not measure behaviour change or health outcomes.
The short version
- What happened
- Nine experts rated 20 questions answered by each of three sources—60 responses and 540 repeated ratings—while 654 office workers completed a separate perception survey.
- Who it affects
- Office workers could receive faster, more tailored explanations, but should not rely on a chatbot to diagnose pain, clotting risk or other urgent symptoms.
- What remains unproven
- The systems used different base models and content packages, so the effects of retrieval, fine-tuning and guardrails cannot be separated.
Keep in mind
The systems used different base models and content packages, so the effects of retrieval, fine-tuning and guardrails cannot be separated.
Nine experts rated 20 fixed questions per source; live, multi-turn use and rare failure modes were not measured.
The 654 workers were recruited through non-probability snowball sampling, and the study measured immediate perceptions rather than behaviour or clinical outcomes.
Source analysis · 1 linked source. Primary analysed source date: 30 September 2026.
Read the full report · 6 min The evidence, methods, implications and limitations
What was compared
The study compared three Chinese-language information sources for occupational sedentary behaviour. The specialist system used GLM-4-9B-Chat on the Coze platform, approximately 1,200 curated question-and-answer pairs, retrieval from 68 documents totalling about 1.2 million Chinese characters, and guardrails requiring referral language for red-flag symptoms. The comparator was Doubao 1.6, prompted as an occupational-health adviser but without browsing or retrieval. The third source was static educational material drawn from WHO, China CDC and Chinese guideline families.
Each source answered the same 20 frequently asked questions, split evenly among risk perception, behavioural tips, symptom management and special populations. Nine occupational-medicine, primary-care and workplace-health experts, averaging 9.8 years of professional experience, independently scored all 60 anonymised responses. The resulting 540 ratings were analysed as repeated measures clustered by rater and question, rather than incorrectly treated as 540 independent observations.[1]
What the ratings found
On the six-point composite scale, the specialist chatbot averaged 5.06, Doubao 4.47 and the static material 4.15. Mixed-effects comparisons reported adjusted mean differences of 0.60 between the specialist system and Doubao and 0.91 between the specialist system and static materials. The specialist system's advantage over the static material was concentrated in usefulness, comprehensibility and completeness; their medical-accuracy and safety scores were not significantly different. A non-significant difference is not proof that two sources are equivalent.
The study also applied a pre-specified safety rule: an answer was flagged when at least two of nine raters gave it a safety score of three or lower. None of the 20 specialist-chatbot answers or 20 static-material answers met that threshold. Two of 20 Doubao answers did, both concerning symptom management after prolonged computer work. This is an informative stress test of fixed answers, not an estimate of how frequently unsafe advice would occur across live conversations.[1]
Why the comparison cannot isolate retrieval or guardrails
The specialist and general systems used different base models. The specialist system also received a larger retrieval corpus and safety filters, while the static material shared guideline families with that corpus but did not contain exactly the same content. Consequently, the experiment compares two realistic system packages; it cannot say how much of the difference came from the base model, supervised fine-tuning, retrieval, the curated documents, the prompt or the guardrails.
All answers were pre-generated, frozen and truncated at 400 Chinese characters. That controls length and silent product updates, but it omits multi-turn conversation, clarification, prompt injection, changing symptoms and the possibility that a user misunderstands or selectively follows an answer. A production occupational-health tool would need monitoring across those situations and an escalation path that does not depend on users recognising risk themselves.[1]
What the 654-worker survey adds
The second phase recruited office workers from enterprises in Hebei, Jiangsu and Yunnan through internal health channels and snowball referral. Of 734 questionnaires opened, 674 were submitted and 20 failed an attention check, leaving 654 valid responses. Participants averaged 34.5 years of age and reported 7.6 hours of occupational sitting per day. They interacted with the specialist chatbot for up to ten minutes before rating ease of use, usefulness, safety, trust, intention to use and satisfaction.
Workers rated ease of use most positively but were more cautious about safety and intention to use. Frequent previous use of health chatbots was associated with higher scores on the six scales, although the safety association no longer remained significant after adjustment. This is cross-sectional perception evidence. Snowball sampling may over-represent people already interested in health or technology, and the platform did not log each participant's questions or exact exposure, so the survey cannot show that the tool improved sitting behaviour or prevented illness.[1]
What employers and workers should take from it
The practical finding is not that employers can replace occupational-health staff with a chatbot. It is that a narrowly scoped system, backed by curated guidance and explicit referral rules, produced more useful and complete fixed answers than the tested general assistant without showing lower expert-rated safety than static material. That makes a supplementary information role plausible, especially for routine questions about breaks, stretching and when symptoms merit professional assessment.
Employers still control whether workers have time, equipment and permission to act on advice. Telling someone to stand or walk is of limited value if workload, disability, pregnancy, shift structure or management practice makes that impossible. Deployment also raises privacy questions: health questions can reveal symptoms and conditions to an employer or technology provider. Workers need to know what is recorded, who can access it and how human support is reached without penalty.[1]
What would change the assessment
A stronger test would hold the base model constant while adding retrieval and guardrails separately, use an independently assembled question set, include external raters and run repeated live conversations. It should report failures across languages, literacy levels, disabilities and high-risk scenarios, as well as whether users understand referral advice. Preregistered trials could then measure actual sitting time, symptom-related help-seeking, inappropriate reassurance, privacy incidents and employer burden over months rather than minutes.
Confidence would fall if independent evaluations find that the ranking disappears with a different question set, that the specialist system fails outside its curated domain or that real users delay appropriate care. For now, the paper supports a limited conclusion: the tested specialist package produced better-rated advice than one general-purpose assistant on this task. It does not demonstrate clinical benefit, safe autonomous diagnosis or general superiority of retrieval-assisted health chatbots.[1]