Can a health chatbot pass an equity audit?
A peer-reviewed audit generated 1,350 mental-wellness responses from three open models. It found complex reading levels, uneven crisis language and almost no explicit clinician-referral wording—but three repeated prompts and no community validation sharply limit the claim.
The short version
- What happened
- Researchers ran three FNMWCF-informed persona prompts 150 times through each of three locally hosted open models, producing 450 responses per model and 1,350 in total; these are repeated stochastic generations, not 1,350 different patient scenarios.
- Who it affects
- People may receive fluent mental-wellness answers that remain too complex to understand during distress.
- What remains unproven
- The 1,350 outputs are repeated generations from only three fixed persona prompts, not 1,350 distinct scenarios or patients.
Keep in mind
The 1,350 outputs are repeated generations from only three fixed persona prompts, not 1,350 distinct scenarios or patients.
Province, sex and age changed together across the three personas, so apparent subgroup differences cannot be attributed to any one factor.
The human pilot had one rater and only 13 variable DeepSeek responses; it cannot compare models or estimate inter-rater reliability.
No clinical outcomes, patient comprehension, adverse events or care decisions were measured.
The personas were not Indigenous-governed or community-validated and contained no Indigenous-identity, Nation, language or location context.
All prompts were in English, and mutable model tags plus stochastic decoding prevent exact regeneration of every response.
Source analysis · 2 linked sources. Primary analysed source date: 5 October 2026.
Read the full report · 8 min The evidence, methods, implications and limitations
The denominator is 1,350 responses from three fixed prompts
Abbas Yazdinejad and Jude Kong asked whether a digital-health audit could look beyond conventional accuracy and score the language, safety signalling and cultural context of model-generated mental-wellness guidance. They constructed three persona profiles from variables in the 2019–2020 Canadian Community Health Survey public-use microdata file. The profiles described moderate-severity, non-emergency situations: a Saskatchewan woman aged 25–34, a Saskatchewan man aged 50–64 and a British Columbia woman aged 50–64. The profiles were checked against combinations that existed in 361, 1,551 and 2,612 survey records respectively, but they were not sampled or weighted to represent those populations.
Each persona became one prompt informed by the First Nations Mental Wellness Continuum Framework, with instructions concerning culture, community connection, land-based options, Elders, counselling, crisis lines and circumstances requiring urgent care. The researchers ran those three prompts 150 times through each of LLaMA-3.2, Mistral-7B and DeepSeek-r1:8B at temperature 0.7 and a 256-token limit. That yielded 450 responses per model and 1,350 overall. The large response count measures variation in repeated model generations; it does not provide 1,350 unique questions, patients, communities or clinical encounters.[1]
The automated audit found very different safety-language patterns
After removing non-user-facing reasoning traces, the team applied fixed regular expressions to full responses. Lexical cultural markers such as community, land, Elder and culture appeared in 100% of the outputs from all three models. Crisis-guidance language appeared in 99.6% of LLaMA-3.2 responses, 99.6% of Mistral-7B responses and 71.3% of DeepSeek-r1:8B responses. A narrower marker—either an explicit statement that the answer was not medical advice or a recommendation to consult a doctor, clinician or provider—was almost absent: zero for LLaMA and Mistral and 0.2% for DeepSeek.
Those figures do not establish that one model was safer. The prompt itself asked for crisis and urgent-care guidance even though the scenarios were non-emergencies, so near-universal crisis wording can reflect instruction-following or excessive escalation. The clinician-referral expression also ignores other onward support, including counsellors, Elders, community resources and crisis services. The paper appropriately treats the markers as observable text patterns, not judgements of clinical adequacy. It reports Wilson intervals conditional on the three prompts and their repeated generations; those intervals cannot support demographic or population inference.[1]
A display limit can hide late safety content
The researchers separately trimmed cleaned responses to 180 words for the user-facing display analysis. That choice materially changed one safety result. DeepSeek-r1:8B's crisis-guidance prevalence fell from 71.3% on full responses to 30.0% after trimming, a 41-point drop, because its guidance tended to appear later. LLaMA moved from 99.6% to 97.6% and Mistral from 99.6% to 96.9%. The study therefore measured safety and cultural markers on complete cleaned responses and used the trimmed text for readability and framing metrics.
This is the paper's most broadly useful design lesson. A health product that limits a model's response, collapses it behind a button or displays only the first screen can remove precisely the material a lab audit counted. Product teams should test the rendered interface, not only the unabridged generation. They also need to decide whether critical advice belongs in a deterministic safety layer, near the beginning of the answer, rather than hoping a stochastic model places it before a token or word cap.[1]
Fluent answers still demanded roughly secondary-school reading levels
Mean Flesch–Kincaid grade levels were 10.8 for DeepSeek-r1:8B, 9.8 for LLaMA-3.2 and 10.4 for Mistral-7B. All exceeded the grade 6–8 range the authors used as a target for patient-facing material. Mean sentence length ranged from 11.78 words for Mistral to 16.28 for DeepSeek. These are reproducible text-complexity estimates, not tests of whether a reader understood or acted on the guidance, but they reveal a practical mismatch: natural-sounding prose can remain inaccessible to people with lower literacy, limited English proficiency or cognitive burden during distress.
The proposed governance table translates this signal into action: require plain-language post-processing and re-audit before deployment, assign a clinical safety lead and health-literacy reviewer, and track model versions so an update triggers another assessment. Other proposed steps include calibrating crisis messages to scenario severity, adding explicit safety-netting, involving an Indigenous data-governance body or community advisory group, and creating incident pathways that can suspend a system after confirmed serious harm. The thresholds are illustrative, not clinically validated standards.[1]
The human evaluation is too small for a model ranking
A single rater scored responses on accuracy, cultural relevance, language accessibility and bias avoidance using a zero-to-three rubric. Only 13 DeepSeek responses had usable variable ratings; three LLaMA and three Mistral entries were present but identical and therefore uninformative for a descriptive summary. The DeepSeek means were 2.31 for accuracy, 1.77 for cultural relevance, 1.54 for language accessibility and 1.39 for bias avoidance. With one rater, no inter-rater reliability can be calculated, and the accuracy judgement was not anchored to an independent clinical reference standard.
Those scores are an exploratory pilot, not evidence that DeepSeek is better or worse than the other models. The automated analysis is also post hoc and descriptive. The paper did not test diagnoses, care decisions, patient outcomes, adverse events, comprehension or behaviour. It used mutable model tags without preserving the exact checkpoint digest for every original run, so the archived outputs and deterministic analysis can be reproduced, but byte-for-byte regeneration of the model responses cannot be guaranteed.[1]
A cultural framework is not community validation
The ethical limit is central rather than incidental. The Canadian survey file is a general-population dataset; it contains no Indigenous-identity variable, Nation or community affiliation, language, or urban, rural, remote or reserve context. The authors did not involve First Nations participants, Elders or communities in creating or validating the personas. They therefore describe the prompts as FNMWCF-informed, not representative, community-grounded or culturally safe. Frequent framework terms show that the models repeated requested vocabulary; they do not demonstrate cultural adequacy.
The study received support from Canadian research agencies and programmes, including NSERC, SSHRC's New Frontiers in Research Fund, Canada's International Development Research Centre, the UK Foreign, Commonwealth and Development Office, and a Canada Research Chair. The authors declared no competing interests. They also disclosed using AI-assisted tools for language and copy editing, code review, consistency checks, figure and table regeneration, and reviewer responses, with the authors reviewing the analyses and conclusions.[1]
What would change the assessment
Confidence would rise with a preregistered evaluation that varies one demographic or clinical factor at a time, uses many distinct scenarios and includes multilingual prompts reviewed by native speakers. Several independent clinical and community raters should score blinded outputs against an external reference, report agreement, and include First Nations governance from question design through interpretation. Exact model digests, prompts, interfaces and safety layers should be archived so another team can reproduce both the generation and analysis.
The consequential test is prospective and people-centred: can a governed system improve comprehension, access or appropriate escalation without increasing anxiety, delay, stereotyping or inappropriate care? Until such evidence exists, this paper supports a useful audit checklist and demonstrates that response trimming and reading level deserve deployment attention. It does not show that any tested model is culturally safe, clinically effective or suitable for patient-facing use, and it does not validate a one-number equity score.[1][2]