Why did five AI models keep naming the same drug products?
Across 177,120 synthetic cases, one product captured an average 97.5% of each model’s first-choice recommendations. The simulated pattern differed sharply from US dispensing data, but it did not test real prescriptions or patient care.
The short version
- What happened
- Five models produced about 883,000 usable answers from 177,120 synthetic vignettes spanning 123 disease–active-ingredient pairs and 1,440 demographic profiles.
- Who it affects
- Patients could see fewer equivalent options if model-generated product names are copied uncritically, especially where price, availability or regulatory status differs.
- What remains unproven
- Synthetic cases, not patient encounters
Keep in mind
Synthetic cases, not patient encounters
The active ingredient was supplied in the main experiment
MEPS distributions reflect clinical, payer, pharmacy and patient factors, so divergence does not itself prove error
Cost estimates were simulated and limited to 51 ingredients with complete data
No downstream prescribing, substitution, dispensing or health outcomes were measured
Source analysis · 1 linked source. Primary analysed source date: 26 October 2026.
Read the full report · 4 min The evidence, methods, implications and limitations
The test held the active ingredient constant
The researchers built 123 pairs linking a common disease to a prespecified active ingredient, then combined each pair with 1,440 profiles varying age, sex, race or ethnicity, LGBTQ+ identity and income. GPT-5.4, Claude Sonnet 4.6, Gemini 3.1 Flash, Llama 4 and MedGemma were each asked to list up to three FDA-approved product names. Four physicians standardized the outputs and checked them against the FDA Orange Book and Purple Book.
That design isolates product-name selection after the active ingredient has already been supplied. It does not ask a model to diagnose a patient or choose a therapy. Across roughly 883,000 usable responses, the most frequently named product captured a mean 97.5% of first choices. Despite an average 5.50 approved products per pair, the dominant name exceeded 90% of pooled recommendations in 96 of 123 pairs.[1]
Agreement between models was not proof of neutrality
Models usually converged on the same dominant product: Fleiss’ kappa was 0.895, with a 95% confidence interval from 0.855 to 0.931. Yet model distributions overlapped poorly with the Medical Expenditure Panel Survey’s real-world dispensing patterns. Median total-variation distance ranged from 0.93 to 0.98, where one represents no distributional overlap. Frequently prescribed products were therefore not necessarily the names models selected.
Visibility and market history were associated with output frequency. Products with more Google search results, a dedicated Wikipedia page or originator status were more likely to be named; originator status carried an odds ratio of 20.5. Regulatory-nonconcordant outputs ranged from 12.4% to 14.7%, mostly referring to products that had once been approved but were no longer marketed. Explicit demographic descriptors also shifted those error rates, although that does not show clinical harm in real patients.[1]
The distinction that matters
What the study measured
The experiment isolates product-name behaviour, not end-to-end care.
Input
Disease, active ingredient and demographic profile
177,120 standardized vignettes
Output
Up to three FDA-approved product names
Dominant name averaged 97.5%
Not tested
No live prescribing or dispensing
Patient and budget effects remain simulated
Prompting moved the winner but did not remove concentration
Prompts asking for comprehensive listings, the lowest cost or generic preference often changed which product led, and the change usually favoured a generic. But recommendations remained concentrated: the leading product still averaged 93.1%, 94.9% and 97.6% under those three instructions. A prompt can therefore steer the output without establishing commercial neutrality or regulatory reliability.
For clinicians and patients, the practical risk is a narrow menu presented with false consensus. The projected cost analysis found higher costs for 38 of 51 fully observed ingredients and estimated a large simulated annual difference, but it relies on normalized acquisition prices and prescribing volumes. Real formularies, dose, pharmacy substitution and clinical decisions intervene before a patient receives anything. Prospective workflow studies are needed before treating these outputs as prescribing evidence.[1]