Can an LLM spot what a clinical-trial paper leaves out?
A peer-reviewed benchmark found that a retrieval-plus-LLM pipeline reached F1 0.822 across 119 reporting questions on 40 held-out trial articles. Better evidence retrieval lifted F1 to 0.893, showing both the promise and the main bottleneck.
The short version
- What happened
- Researchers converted 83 SPIRIT 2013 and CONSORT 2010 checklist items into 119 element-level questions and annotated 200 articles forming 100 protocol-results pairs.
- Who it affects
- More complete trial reports could help clinicians and patients understand what was done and judge whether results apply to them.
- What remains unproven
- The held-out evaluation contained 40 articles representing 20 randomized trials from one existing open-access corpus.
Keep in mind
The held-out evaluation contained 40 articles representing 20 randomized trials from one existing open-access corpus.
Only 50 of 200 articles were double-annotated; one expert annotated the remaining 150.
The framework derives from SPIRIT 2013 and CONSORT 2010 and does not fully cover their 2025 updates.
Expert-selected snippets produced markedly better results, so automatic evidence retrieval remains a major source of error.
The study measures reporting-element classification, not trial quality, risk of bias, truth of findings, patient benefit or editorial outcomes.
The publisher identifies the peer-reviewed article as an early accepted version that may receive further editorial changes before the final Version of Record.
Source analysis · 2 linked sources. Primary analysed source date: 6 October 2026.
Read the full report · 8 min The evidence, methods, implications and limitations
The answer: useful as a reporting screen, not as an evidence judge
A peer-reviewed paper published on 6 October reports that a two-stage AI pipeline could identify many specific details that randomized-trial papers did or did not report. On 119 element-level questions applied to a held-out set of 20 protocol-results pairs, the principal GPT-5 pipeline reached an F1 score of 0.822, sensitivity of 0.824, specificity of 0.889 and Gwet's AC1 agreement of 0.796 against expert annotations. The evaluation covered 40 articles, not 40 trials, because each trial contributed a protocol and a results paper.
Those figures support a narrow use: flagging passages and omissions for an author, reviewer or editor to inspect. They do not show that the system can decide whether a trial was well designed, whether its findings are true or whether a treatment helps patients. Reporting completeness and methodological quality overlap, but they are not interchangeable. A beautifully reported biased trial remains biased; a missed detail may reflect a retrieval failure rather than an absent method.[1]
The benchmark breaks broad checklists into 119 concrete questions
The researchers started with SPIRIT-CONSORT-TM, an earlier corpus of 200 open-access articles representing 100 randomized trials. Each trial had a protocol and a results publication annotated against 83 items drawn from SPIRIT 2013 and CONSORT 2010. Because one checklist item can contain several obligations, the team decomposed those items into 119 questions. A single question might ask whether a paper reports a dose, frequency and duration separately rather than treating any mention of dosing as complete.
Three experts in epidemiology and meta-research developed the questions and their instructions. Eighty-five questions required one answer and 34 allowed multiple selections; 26 depended on an earlier answer. Across the 200 articles, annotators made 9,892 explicit judgments out of 19,500 potential article-question combinations. Another 9,608 combinations were coded automatically because no relevant text was annotated or a question did not apply. The data and code are publicly available through the authors' Open Science Framework repository.[1][2]
Human agreement was substantial but not uniform
Two experts independently annotated 50 articles, covering 25 protocol-results pairs, and reconciled disagreements. One expert annotated the remaining 150 articles. Mean inter-annotator agreement across the questions was Gwet's AC1 0.782, with a standard deviation of 0.219; the median was 0.821. Some questions produced perfect agreement, while questions about whether adjusted analyses were pre-specified or post hoc fell to 0.33.
That spread matters because a model cannot be more objectively correct than an uncertain reference makes possible. It also shows why element-level questions are more informative than one broad completeness score. Straightforward presence checks may be suitable for automation; interpretive questions may need clearer guidance or human review. The study's reconciled labels create a useful benchmark, but most articles were seen by only one annotator, leaving possible annotation error in three quarters of the corpus.[1]
The evaluation separates retrieval from question answering
The automated pipeline first retrieves sentences relevant to a checklist item using a PubMedBERT model previously fine-tuned on the corpus. It then gives those snippets, the element-level question, expert-written instructions and permitted answer options to a large language model. The principal system used GPT-5; the comparison used the open-weight Qwen-2.5-72B-Instruct model in 8-bit form on one Nvidia H100 GPU. Both were tested under the same question-answering setup.
All reported evaluations used the pre-existing test split of 20 protocol-results pairs, while 80 pairs supplied training information and the majority-class baseline. That patient-like unit split matters because keeping paired papers together reduces leakage between a trial's protocol and results report. Performance was aggregated across 119 questions, and 95% confidence intervals were estimated with 10,000 bootstrap resamples. For the principal GPT-5 pipeline, F1 was 0.822 with a 95% interval from 0.794 to 0.847; agreement was 0.796 with an interval from 0.760 to 0.829.[1]
Retrieving the right evidence is the larger bottleneck
When the question-answering model received expert-annotated snippets instead of automatically retrieved ones, F1 rose from 0.822 to 0.893, sensitivity from 0.824 to 0.894, specificity from 0.889 to 0.935 and Gwet's AC1 from 0.796 to 0.877. The expert-snippet condition is an upper-bound experiment because a deployed checker would not normally receive passages selected by a human who already knows where the relevant evidence is.
Adding one illustrative example produced only modest gains. The best one-shot setting reached F1 0.830, an increase of 0.8 percentage points. More examples degraded performance in unreported experiments. GPT-5's aggregate results were slightly higher than Qwen-2.5's, but their 95% intervals overlapped; Qwen did better on 23 questions and tied on 39. That suggests the pipeline's evidence selection and question design may matter at least as much as choosing the larger proprietary model.[1]
The dataset is valuable, but it is small and historically bounded
The 200 articles came from paired publications in the PubMed Central Open Access Subset. That design enables full-text research and protocol-results comparison, but it may not represent subscription journals, specialties, languages, publication formats or trial designs outside the sample. The held-out denominator—40 articles from 20 trials—is especially important when interpreting an aggregate score across 119 questions. A system can look consistent across thousands of question decisions while still having little evidence about new journals or domains.
The framework is based on SPIRIT 2013 and CONSORT 2010 because those were the standards behind the source corpus. The authors mapped their questions against the 2025 updates and found partial rather than full coverage for many items. A production checker would therefore need additional questions, renewed expert validation and version control as reporting standards change. The published study also evaluates manuscripts already in a research corpus; it does not test whether showing flags to authors improves a submission or reduces research waste.[1]
What this could change in publishing
A carefully deployed checker could give authors concrete prompts before submission, help editors prioritise manual review and point reviewers to a suspected omission with its supporting passage. Because the system is built around permitted answers and evidence snippets, it offers a more inspectable workflow than asking a general chatbot whether a manuscript follows CONSORT. The public dataset also gives independent teams a shared target for testing retrieval, smaller models and human-AI review.
The safer design is advisory and traceable: show the exact question, retrieved passage, model answer and uncertainty; let a human correct the record; and never equate an absent passage with misconduct. Journals would need to protect unpublished manuscripts, disclose any external model processing and monitor whether error rates differ by specialty, language or writing style. The paper reports an estimated API cost of about $1.45 per article for GPT-5 and $0.45 for hosted Qwen-2.5, but privacy, integration, human review and governance would add costs not captured by token pricing.[1][2]
What would change the assessment
Confidence would rise with a locked external test across journals, specialties, languages and recent papers written to SPIRIT and CONSORT 2025, reported at the article and question level. Independent teams should reproduce the benchmark from the released data and code, then test whether better retrieval closes the gap to expert-selected evidence without creating new false omissions. Calibration, abstention and the time experts spend correcting flags matter more to practice than one pooled F1 score.
The decisive evidence would be a prospective editorial trial: randomly offer the checker to some authors or reviewers, measure which reporting elements improve, record false alerts and workflow time, and follow whether readers can verify methods more easily. The assessment would weaken if performance drops outside the 40-article test set, if confidential manuscripts cannot be protected or if automated scores become a gatekeeping shortcut. For now, this is a credible benchmark and a useful research tool—not a licence to automate peer review.[1][2]