AI health assistants often endorsed self-diagnoses that needed caution—what did the test measure?

Matched questions tested both missed warnings and needless alarm

A new preprint finds that general-purpose AI assistants frequently accepted a user's benign explanation even when the written symptoms left a time-critical condition possible. The researchers built SelfDx from 100 matched pairs, split evenly between condition and medication questions. In each pair, a caution case leaves a serious alternative open; its partner keeps the same self-diagnosis but adds evidence intended to rule that alternative out. This paired design matters because a system that warns everyone can look safe while being unhelpful.

The 200 messages were written rather than collected from patients. Forty per cent of caution cases were graded emergent, 51% urgent and 9% chronic. A physician reviewed the pairs and grading criteria. Twenty-seven models from six labs answered each prompt three times. The automated judge scored only the first two substantive sentences, reflecting the authors' view that a warning buried later may arrive after the user has anchored on reassurance.

For 20 current-generation models, appropriate agreement on cases where reassurance was warranted ranged from 82% to 100%. Appropriate caution on condition cases averaged 46.6%, and even the strongest model reached 77.0%. Under the stricter requirement to name a specific serious alternative early, the best rate was 32.6%. These results describe response openings to constructed prompts, not diagnostic accuracy or clinical safety in use.[1]

Prompting helped caution but also produced false alarms

The team then tested three safety prompts and supervised fine-tuning on Llama 3.3 70B. Prompts raised strict caution on condition cases from 0% to between 56.3% and 74.1%, but reduced appropriate agreement by 9 to 35 percentage points. Fine-tuning on 128 separate examples lifted strict caution to 51.9% and agreement to 99.7% across three runs. That comparison is encouraging, but it used one open-weight model and synthetic training answers written by another model.

For patients, the finding is a reason not to treat conversational reassurance as triage. It does not show that any named model misses a particular emergency in real practice. The rubric was revised after review by one physician, judge–physician agreement was assessed on the same ratings used in that revision, and the automated judge came from a developer represented in the model panel. A prospective study with diverse clinicians, real user language, full conversations and measured care-seeking outcomes would substantially change the assessment.[1]