Can AI tools review scientific literature reliably? A 102-paper benchmark found access and judgment gaps
Five tools collectively found 57 of 102 known papers before a controlled subscription test. Given every PDF, three tools found all 18 central papers but misclassified 40% to 79% of passing mentions as substantial. The manuscript is an unreviewed preprint.
The short version
- What happened
- The unreviewed study used a published 102-paper meteorology review as a benchmark, with 18 substantial papers and 84 passing mentions.
- Who it affects
- Researchers and students can use AI for candidate discovery and extraction, but must inspect access gaps, paper importance and claim fidelity.
- What remains unproven
- The manuscript is an unreviewed preprint using one English-language meteorology corpus, one run per tool and mid-2026 product versions.
Keep in mind
The manuscript is an unreviewed preprint using one English-language meteorology corpus, one run per tool and mid-2026 product versions.
Discovery conditions differed, and the subscription comparison also changed website blocking and browser state.
Only two experts rated syntheses; they differed substantially and one had authored the benchmark review.
The prompt prescribed ten themes, so the study did not test whether tools would independently discover the central issues.
Source analysis · 2 linked sources. Primary analysed source date: 3 November 2026.
Read the full report · 7 min The evidence, methods, implications and limitations
The answer: AI can assist a review, but its output hides its blind spots
An unreviewed preprint submitted on 3 November tests six AI research tools against a literature review whose 102 source papers and classifications were already known. The tools found and summarized useful material. They also missed papers because of access, overestimated how many treated the topic substantially and produced polished syntheses that two experts judged more accurate than insightful. The benchmark supports supervised assistance, not autonomous evidence synthesis.
The study separated four tasks researchers often compress into one prompt: discovery, interpretation, synthesis and proposing future work. That matters because a tool can quote text correctly while misunderstanding its importance, or write a coherent narrative from an incomplete search. The paper found both failures. None of the products revealed the size or pattern of its omissions in a way that would let an ordinary user recover them without a known benchmark.
The numbers belong to a narrow design. The corpus concerns the 'Spanish plume', a meteorological term used inconsistently over decades. Each product ran once per task using versions available in July and August 2026. Two experts scored three syntheses and disagreed on the ordering. The manuscript has not passed peer review, so it is a warning about workflow and access rather than a permanent product league table.[1][2]
A bounded corpus makes missing papers and bad weighting countable
The benchmark came from a published human review whose authors searched Google Scholar, Web of Science and meteorological publishers for the exact phrase. They assembled 102 peer-reviewed papers from 1968 to 2023 and classified 84 as passing mentions, 14 as informative and four as essential. The latter two groups formed the 18 substantial papers. The benchmark also recorded whether each paper meant an elevated dry airstream, a wider weather pattern or both.
A bounded set makes recall visible. In ordinary research, a tool may return twenty plausible sources and neither it nor the user knows whether twenty important papers are missing. Here, every named result could be checked. For interpretation and synthesis, all 102 machine-readable PDFs were supplied, so later errors could not be blamed on unavailable documents. The design separates access failure during discovery from judgment failure after access.
The topic is tricky because many papers mention the phrase without analysing it, while a smaller set uses it centrally. A useful review must distinguish those roles and recognize inconsistent terminology. That is closer to scholarly judgment than keyword extraction. But it remains one English-language topic with one expert-built gold standard. Other disciplines may have contested classifications, multilingual sources, unpublished evidence or different database coverage.[1][2]
The distinction that matters
Where the literature-review pipeline broke
The complete human-reviewed corpus made omissions and weighting errors measurable.
Find
Search for 102 known papers
57 found by at least one initial tool
Interpret
Separate 18 substantial from 84 passing
All 18 found; 34–66 passing misclassified
Synthesize
Write from all 102 PDFs
Mostly accurate, limited expert-rated insight
Propose
Identify research gaps
Some useful ideas, often too broad
Subscriptions changed discovery; full access did not solve judgment
Five tools completed the initial search. Codex named 42 benchmark papers, ChatGPT Deep Research 40, Gemini Deep Research 16, Claude Code 15 and Claude's web research interface seven. Together they found 57 distinct papers, leaving 45 not named. Gemini reported more screened records than it named, so those records could not be verified. These are not controlled scores: browser state, blocking and search interfaces differed across tools.
A paired follow-up tested the two coding agents without and with University of Manchester subscription access. Codex rose from 42 to 82 papers and Claude Code from 15 to 79. Login also let agents pass some website blocks, so the experiment cannot isolate entitlement from technical access. The tools found 78% of papers from 2020 onward but 48% from 1995 to 2019, showing how an apparently current review can lose the historical middle.
When Codex, Claude Code and NotebookLM received every PDF, all three identified all 18 substantial papers. Yet Codex misclassified 66 of 84 passing mentions as substantial, NotebookLM 65 and Claude 34. Precision for the substantial class was only 0.21 to 0.35. The tools rarely lost a central paper once supplied, but often failed to decide what deserved central weight. A system that labels almost everything important can achieve perfect recall without producing a discerning review.[1][2]
The prose was plausible; the expert judgments were mixed
The three tools received the same locked prompt and all PDFs. They had to synthesize ten predefined themes, cite a page for each claim, separate findings from inference and avoid unsupported consensus. Two meteorology experts scored anonymized outputs. One had led the benchmark review and the other was independent. NotebookLM wrote 3,624 words and cited 41 papers, Codex wrote 3,919 words and cited 27, and Claude wrote 7,480 words and cited 79. Length and citation count did not determine quality.
One reviewer ranked Claude first; the other ranked Codex first. Across every pairing, veracity scores exceeded insight scores. Both reviewers saw a tendency to restate rather than synthesize, and all three tools used stronger language about agreement than the corpus supported. Quotation auditing found the checked Claude quotations accurate, no direct quotations in Codex's synthesis and two problematic NotebookLM quotations. The study found no invented references in the scored outputs, but did find wrong titles, pages and source details.
The authors publish prompts, outputs, scripts and data on Zenodo, but limitations remain: one corpus, one run, one prompt per task, only two reviewers and product versions that will change. Reviewers agreed exactly on only 24% of their 150 ratings and ranked tools differently. Confidence would rise with preregistered replication across disciplines, languages and access conditions. The defensible conclusion is that discovery coverage, evidence weighting and synthesis insight are separate capabilities—and researchers must supervise all three.[1][2]