Can an AI agent audit cancer studies? A 20-paper test found useful triage—and many false alarms
An unreviewed benchmark reports that an agent reproduced 77.4% of 2,190 claims from US cancer-registry papers. But expert review rejected nine of ten sampled robustness warnings as consequential changes.
The short version
- What happened
- The benchmark covers 20 published SEER cancer-registry papers, 901 findings and 2,190 atomic claims; the agent reproduced 1,694 claims, or 77.4%, under the authors’ grading rules.
- Who it affects
- More efficient study audits could help clinicians and patients by exposing fragile evidence sooner. The present evidence supports expert triage, not replacing epidemiologists, statisticians, peer reviewers or clinical judgement.
- What remains unproven
- This is an unreviewed preprint from the team that created both the benchmark and the evaluation pipeline.
Keep in mind
This is an unreviewed preprint from the team that created both the benchmark and the evaluation pipeline.
The benchmark contains 20 SEER-based observational papers; transfer to trials, laboratory research, other registries or non-US data is unknown.
Experts reviewed ten of 55 automated warnings, and nine did not establish a consequential change; unflagged findings were not sampled for missed problems.
The paper reports no end-to-end comparison of expert time, financial cost, downstream corrections or patient outcomes.
Source analysis · 1 linked source. Primary analysed source date: 16 October 2026.
Read the full report · 4 min The evidence, methods, implications and limitations
The agent recovered most claims—but the denominator matters
An AI agent reproduced 1,694 of 2,190 numerical and qualitative claims drawn from 20 papers using the US National Cancer Institute’s SEER registry, according to a new unreviewed preprint. Another 76 claims were classed as close and 420 were not reproduced. The test is substantial: it covers 901 parent findings and a curated environment of 39 data tables, 198 million rows and 40 documentation resources. It does not show that 77.4% of cancer research is correct, or that the remaining claims are wrong.
The team treated a numerical claim as reproduced when the recreated value was within 5% of the paper’s value; qualitative claims received a yes-or-no judgement. Among the 420 misses, 251 were attributed to missing or changed data, 47 to under-specified analysis, 28 to unavailable-software approximations and 48 to agent errors. Historical registry coverage had also changed. Those categories show why a failed reproduction is a diagnostic lead, not a correction to the medical literature.[1]
Robustness warnings need human adjudication
The authors then asked several model-based analysts to alter defensible choices—such as cohort definitions or statistical specifications—and rerun analyses. At least one automated analyst linked a potentially major change to 55 of 579 fully reproduced findings, or 9.5%; two or more analysts agreed on 14. Yet 41 of the 55 were found by only one analyst, suggesting that model choice strongly shaped which warnings appeared.
That shortlist looked much less decisive after expert review. The paper reports examination of ten flagged cases: nine did not establish a consequential change under a defensible alternative, while one required further investigation. This is the study’s most important practical limit. The system may help concentrate scarce expert attention, but the sampled warnings had a high false-alarm burden and the experts did not review all 55 flags or a random sample of unflagged findings.[1]
The distinction that matters
What the audit actually measured
The stages answer different questions and should not be collapsed into one accuracy score.
20 papers
Split 901 findings into 2,190 claims
A claim-level benchmark
1,694
Recreated within the grading rule
77.4% reproduced
55 of 579
Flagged after alternative analyses
Candidates for expert review
10 checked
Experts examined sampled flags
Nine not consequential; one unresolved
Useful infrastructure is not an autonomous referee
For researchers and journal editors, the immediate value is an executable audit trail: code, outputs and alternative analyses attached to individual claims. That could make it easier to locate missing data, ambiguous methods and results worth checking. It should not become an automated pass-or-fail score. The benchmark uses one US registry family, the reproduction agent and grader share model infrastructure, and the work has not been peer reviewed or independently replicated.
Most authors are affiliated with Databricks AI Research or Stanford; the acknowledgements report support or resources from the Verdant Foundation, Hasso Plattner Institute, Itaú Unibanco, BMO Financial Group, Stanford HAI, the US NAIRR pilot and Microsoft Azure. No separate competing-interests statement appears in the version reviewed. Confidence would rise if an independent team reran the benchmark, experts blindly assessed all flags plus matched non-flags, and studies measured review time, cost and important errors avoided—not only claims reproduced. That evidence would show whether triage saves expert effort without hiding missed problems.[1]