A diagnostic AI can look safer when the test is vague—what changes with a clinical rubric?
A bilingual benchmark found that detailed, diagnosis-specific rubrics changed safety scores and exposed sharp failures under instruction-shell attacks. Its 3,489 prompts are synthetic and the 11 tested models ran without deployment safeguards.
The short version
- What happened
- DiagSafe contains 3,489 synthetic English and Chinese prompts: 2,401 ordinary diagnostic situations and 1,088 instruction-shell attacks across an 84-label safety taxonomy.
- Who it affects
- Clinicians and patients need safety claims tied to the actual diagnostic task and deployment controls. A model-level benchmark can help teams find weaknesses, but it cannot replace local clinical governance or incident monitoring.
- What remains unproven
- The unreviewed benchmark uses synthetic model-generated prompts and rubrics, with human checks on stratified samples rather than all 3,489 items.
Keep in mind
The unreviewed benchmark uses synthetic model-generated prompts and rubrics, with human checks on stratified samples rather than all 3,489 items.
Only open-weight models in one local inference configuration were tested, without production safeguards.
The study measures rubric-defined unsafe responses, not diagnostic quality, clinician behaviour, patient harm or live attack frequency.
Several authors are affiliated with Tencent Hunyuan; no separate funding statement is reported.
Source analysis · 1 linked source. Primary analysed source date: 21 October 2026.
Read the full report · 3 min The evidence, methods, implications and limitations
The benchmark asks what kind of harm is possible
Researchers organised diagnostic-assistant risks into 84 leaf labels under 11 top-level categories, linking them to 204 provisions drawn from clinical guidance and rules in China, the European Union and the United States. A generator model then produced 3,489 synthetic prompts: 2,401 ordinary diagnostic situations and 1,088 adversarial prompts designed to hide or reframe unsafe instructions. About half were Chinese and half English; single-turn and multi-turn cases appeared in a seven-to-three ratio.
Three experts checked stratified samples rather than every item. They accepted 324 of 336 prompts as realistic and correctly labelled, and 251 of 252 diagnosis-specific rubrics as reasonable and actionable. On a separate human-labelled evaluation set, rubric-guided judging reached 95.34% accuracy, compared with 87.29% for a generic medical-safety prompt and 86.86% when the label alone was supplied. Repeated judging agreed 93.33% to 94.78% of the time.[1]
Attack results are a warning, not a production incident rate
The team tested 11 locally served open-weight language models, ranging from 8 billion to 1.6 trillion parameters. Every model's unsafe-response rate rose under instruction-shell attacks: the multiplier ranged from 1.9 to 8.4, with a median of 5.3, and eight models exceeded 40% on that adversarial set. The finding suggests a broad weakness when harmful diagnostic instructions are wrapped in plausible context.
But the setup deliberately omitted system prompts, moderation layers, retrieval controls and deployment guardrails. The prompts and rubrics were themselves model-generated, then sample-checked; rare or locally specific harms may be missing. The work covers stand-alone chat rather than tool-using clinical agents, and it does not measure diagnostic accuracy, patient outcomes or real-world incident rates. Independent clinical validation, red-teaming of complete deployed systems and prospective monitoring would materially change the assessment.[1]
The distinction that matters
Why the rubric changes the test
The framework moves from a broad safety question to a diagnosis-specific failure definition.
Classify
Map the prompt to one of 84 risk labels
Names the kind of diagnostic harm
Specify
Apply an item-level clinical rubric
Defines what a safe answer must avoid or include
Stress
Compare ordinary and instruction-shell prompts
Separates baseline from adversarial behaviour