Clinicians made a chatbot guardrail auditable—and it still over-blocked benign replies
Two clinical-psychology trainees translated four mental-health safety checks into verifiable rules. The rules exposed why decisions differed from an LLM judge, but depended on model-extracted facts and blocked many benign boundary cases.
The short version
- What happened
- Two doctoral trainees in clinical psychology defined four checks covering crisis resources, referral, medication boundaries and reinforcement of harmful beliefs or targets.
- Who it affects
- A person seeking mental-health support can be harmed by both missed escalation and excessive blocking. Traceable rules may help clinicians inspect that trade-off, but they do not replace a qualified service, emergency pathway or real-world evaluation.
- What remains unproven
- The paper is an unreviewed preprint and does not report patient outcomes or a live deployment.
Keep in mind
The paper is an unreviewed preprint and does not report patient outcomes or a live deployment.
The 294-item descriptive sample informed one rule decision; confirmatory testing was limited to 500 held-out boundary prompts.
Two trainees authored the checks, and the weakest pilot agreement was for the crisis-resource check that drove many outcomes.
Fifteen of sixteen facts came from one Llama-3.3-70B extractor and one prompt; results were not shown to transfer to another extractor.
Source analysis · 1 linked source. Primary analysed source date: 22 October 2026.
Read the full report · 3 min The evidence, methods, implications and limitations
The rules could be checked exhaustively
The clinicians refined four questions using a 19-item pilot and a 75-item main annotation batch. Researchers expressed the result through sixteen typed facts, proved seven properties with an SMT solver and enumerated 393,216 valid fact combinations. At runtime, the same Llama model used for comparison extracted fifteen facts; an exact text match supplied the crisis-number fact.[1]
Traceability revealed a real trade-off
The LLM judge requested a crisis resource in 55 of 65 replies that already contained 988 or 911; the deterministic rule avoided that error. Yet on the reference-scored crisis check, the judge had fewer false alarms: 23 of 81 versus 39 of 81. Both missed 2 of 13 labelled violations. The paper therefore supports auditability, not superiority or clinical safety.[1]
The distinction that matters
Where the guarantee starts—and stops
The pipeline separates clinical policy, model interpretation and deterministic enforcement.
Clinicians
Define four safety checks
An explicit, revisable policy
Model
Extract fifteen conversation facts
Still fallible and prompt-dependent
Rules
Apply verified predicates
Traceable consistency, not clinical correctness
Verified rules still inherit uncertain facts
Formal verification guarantees only that the written rules are applied consistently to the supplied facts. Fifteen facts came from one model and prompt, several clinician decisions remained open, and held-out testing used 500 boundary prompts rather than real patient conversations or outcomes. Any deployment would need broader clinical ownership, local crisis pathways and prospective safety monitoring.[1]