Seventy-four medical GenAI trial reports left core reproducibility details missing
An audit covering 73 randomized trials and 24,150 participants found that no applicable report fully described how poor-quality or missing input data were handled.
The short version
- What happened
- The review screened 16,485 records and included 74 reports from 73 randomized trials involving 24,150 participants; the median trial size was 100.
- Who it affects
- A patient or clinician cannot judge a trial's relevance if the report does not specify which model, inputs and human checks produced the result. Reproducible reporting is a prerequisite for safe comparison and regulation.
- What remains unproven
- 74 reports from 73 RCTs
Keep in mind
74 reports from 73 RCTs
24,150 participants
Reporting audit—not effectiveness review
Author-developed signaling questions
English-language peer-reviewed trials
Mostly text-based systems
No risk-of-bias assessment
Source analysis · 1 linked source. Primary analysed source date: 28 October 2026.
Read the full report · 4 min The evidence, methods, implications and limitations
Trial reports often omitted what would let others reproduce the AI
Medical generative-AI trials frequently failed to state the model version, input rules, deployment setup or error behaviour needed to understand and reproduce an intervention, according to a peer-reviewed Journal of Medical Internet Research review published on 28 October. Among 74 reports, 17 fully reported the algorithm version, 21 described integration into the trial setting, 25 described human–AI interaction and 27 reported performance errors.
The largest gap concerned the data entering the system. Only three of 73 reports fully stated input-data eligibility and nine of 74 described how inputs were acquired or selected. None of 72 applicable reports fully explained how poor-quality or missing input data were handled. For a clinician or patient, these omissions matter because the same named model can behave differently when prompts, interfaces, access modes or incoming records change.[1]
The audit covered 73 randomized trials
Reviewers searched PubMed, CENTRAL and Embase for peer-reviewed English-language randomized controlled trials published from 2022 to 3 April 2026, then used citation searching. From 16,485 retrieved records, they included 74 reports representing 73 trials and 24,150 participants. The median trial enrolled 100 people, with an interquartile range of 50 to 247. Large language models accounted for 73 reports, and GPT-family systems for 45.
Two reviewers independently screened, extracted and rated each report against CONSORT-AI items, with disagreements resolved by a third. Agreement was substantial. The team also checked protocols and registrations for information missing from papers. This was a reporting-adherence study: it deliberately did not assess risk of bias or pool clinical effectiveness, so its findings cannot tell readers whether GenAI improved health, decisions or workflow.[1]
The distinction that matters
Four reproducibility details were often missing
The audit counted reports that fully satisfied each CONSORT-AI item.
Model version
17 of 74 reports
23.0% fully reported
Human–AI interaction
25 of 74 reports
33.8% fully reported
Poor or missing inputs
0 of 72 applicable reports
No full reporting
Existing guidance may need a GenAI-specific extension
Only nine of 74 reports explicitly cited CONSORT-AI. Citation accompanied better reporting for some interaction and data-sharing items, but major technical gaps persisted. The authors argue that conventional checklist language does not fully capture system prompts, inference settings, commercial access modes and changing third-party models. Their evidence supports a formal consensus process for a GenAI-specific extension; it does not itself create a new reporting standard.
The signaling questions used to score compliance were developed by the review team and have not been independently validated. The English-only, peer-reviewed RCT filter may also miss other randomized evaluations, and most included interventions were text based. The work was funded by the Republic of Korea's Ministry of Food and Drug Safety, with no funder role reported and no declared conflicts. Progress would mean validated criteria and trial reports that publish exact intervention specifications, inputs, failures and human roles.[1]