A biomedical AI cited real papers—but did they support its claims?
In a 100-claim test, closed-book models produced 327 valid PubMed identifiers but an automated cross-family judge found only three relevant. Letting agents write their own searches improved retrieval, though the small accuracy gains were not significant.
The short version
- What happened
- The author sampled 100 of 300 labelled SciFact development claims and compared closed-book, one-shot PubMed retrieval, an agent-written multi-query loop and an oracle with gold abstracts across two model backbones.
- Who it affects
- Researchers and clinicians should open the cited paper and check the exact claim, not treat a resolving DOI or PMID as verification. Tool builders should bind citations to retrieved text and expose abstention.
- What remains unproven
- The unreviewed workshop paper evaluates 100 claims from one benchmark and two backbones.
Keep in mind
The unreviewed workshop paper evaluates 100 claims from one benchmark and two backbones.
Accuracy differences between realistic harnesses were not statistically significant at this sample size.
A cross-family LLM judge, not a complete human review, supplied the main citation-relevance audit.
The author reports no specific funding and no competing interests.
Source analysis · 1 linked source. Primary analysed source date: 21 October 2026.
Read the full report · 3 min The evidence, methods, implications and limitations
The harness changed what evidence the model saw
The study used a fixed random sample of 100 claims from the 300-claim SciFact development split because the benchmark's test labels are hidden. Each claim was labelled supported, contradicted or not enough information. Two efficient model backbones ran four conditions at temperature zero: closed book, one PubMed query built directly from the claim, a ReAct-style agent that wrote and refined queries, and an oracle supplied with the gold abstracts.
The single-shot query found the gold paper for only 6% of claims and mostly converted overconfident errors into abstentions. Agent-written queries tripled gold recall and produced the best realistic verdicts, at 62% accuracy against 49% to 59% elsewhere. Those differences were not statistically significant on 100 claims. When given the gold abstracts, the same backbones reached 79% to 86%, leaving a 17-to-24-point retrieval gap.[1]
A valid identifier did not make the citation valid
Closed-book models cited 335 PubMed identifiers; 327 resolved to real records. A cross-family model judge rated only three of those 327 as relevant to the claim. Search agents produced fewer citations but about 67% were judged relevant, and they almost never cited outside their retrieved set. The result warns that identifier-existence checks catch fabrication but not papers attached to the wrong proposition.
The denominator is small, all claims came from one benchmark and abstracts were truncated to 1,500 characters in retrieval conditions. Citation relevance relied on another language model rather than comprehensive human adjudication, so the striking three-of-327 figure needs independent review. The released cache, prompts and outputs support reproduction. Larger human-audited studies across databases, specialties and full text would change the assessment.[1]
The distinction that matters
Four ways to answer the same claim
The backbone and output format stay fixed while the evidence path changes.
Closed book
Answer from model memory
Can produce real but irrelevant identifiers
Search agent
Write and revise PubMed queries
Finds the gold paper more often
Oracle
Supply the benchmark's gold abstracts
Estimates remaining reasoning error