Can researchers rerun clinical AI?
A peer-reviewed scoping review found accessible analytical code in 12.2% of 3,967 prediction-model papers that cited TRIPOD or TRIPOD+AI. Even among shared repositories, dependency versions, tests and reusable structure were often missing, so code availability alone did not establish reproducibility.
The short version
- What happened
- The review included 3,967 open-access prediction-model articles that cited TRIPOD or TRIPOD+AI; 12.2% reported accessible analytical code.
- Who it affects
- Patients benefit when model errors can be found and local performance checked before a tool influences care.
- What remains unproven
- The sample was restricted to open-access papers that cited TRIPOD or TRIPOD+AI and may not represent the wider prediction-model literature.
Keep in mind
The sample was restricted to open-access papers that cited TRIPOD or TRIPOD+AI and may not represent the wider prediction-model literature.
The study measured reported code availability and repository features; it did not rerun all analyses or establish clinical validity.
Some stated code locations could not be independently retrieved but were counted to avoid underestimating reporting.
The LLM-assisted pipeline performed well on article selection but was less reliable for subjective repository qualities.
Repositories were inspected at the time of analysis and may have changed since publication.
Source analysis · 3 linked sources. Primary analysed source date: 9 October 2026. Earliest linked supporting source: 16 March 2026.
Read the full report · 8 min The evidence, methods, implications and limitations
The direct answer: usually not from the paper alone
Most clinical prediction-model papers in this review did not provide accessible analytical code. Among 3,967 eligible articles, 12.2% included a code-sharing statement that the review counted as accessible. The proportion rose over time and reached 15.8% for papers published in 2025, but that still left the large majority without code that another researcher could inspect or rerun.
Availability was only the first test. Of the repositories the team examined, 80.5% contained a README, 37.6% specified software dependencies, 21.6% pinned dependency versions and 42.4% were organised modularly. The paper also found infrequent licences, tests, citation metadata and reproducibility aids. A public link can therefore preserve files without supplying the environment, instructions or structure needed to reproduce an analysis.
The appropriate conclusion is not that 87.8% of models are wrong, nor that every shared repository fails. The study did not rerun every model or audit clinical validity. It establishes a reporting and scrutiny gap: in a large, guideline-aware sample, independent readers usually lacked accessible code, and many available repositories omitted practical features that make reuse credible.[1][2]
How the 3,967-paper sample was assembled
The researchers identified PubMed-indexed articles that cited the TRIPOD or TRIPOD+AI reporting statements by 11 August 2025. Eligible primary studies developed, updated or validated a multivariable prediction model using statistical or machine-learning methods. To keep full-text retrieval reproducible and free of subscription barriers, the review included only articles available through the PubMed Central Open Access API.
That produced 3,967 eligible papers. The frame is deliberately policy-relevant because these authors had acknowledged reporting guidance, but it is not representative of every prediction model. Papers behind paywalls were excluded, as were studies that did not cite the statements. The authors argue that a guidance-aware cohort may be more likely to share code, so the 12.2% estimate could be higher than the rate across the broader field.
The review followed PRISMA guidance for scoping reviews and registered a protocol. Code counted as shared when the retrieval tool successfully downloaded a repository containing at least one non-empty source-code file. Statements pointing to supplementary material or unsupported repository providers were also counted even when the tool could not independently inspect their contents, a decision intended to avoid underestimating reported availability.[2]
What the AI-assisted review did—and how it was checked
A structured pipeline used GPT-5.2 to screen full-text articles, identify code-availability statements, retrieve repository links and record the first author's country. Two independent reviewers manually annotated a random set of 500 articles, resolving disagreements through discussion. Against those labels, the automated selection achieved a weighted F1 score of 0.97 and repository-link retrieval accuracy of 92.3%.
The team then converted repositories into structured text and used a model to score 14 predefined features covering documentation, dependencies, testing, reproducibility, citation and sample data. On a manually annotated subset of 29 usable repositories, objective features such as the presence of a README, licence or tests performed strongly. More interpretive judgments were weaker: the reported F1 was 0.71 for sufficient code documentation and 0.67 for listing hardware requirements or linking to the paper.
Those validation results make the pipeline auditable, but they do not remove error. Automated misclassification could move the point estimates, particularly for subjective features. The authors therefore frame the numbers as indicators of overall practice rather than exact counts. They also published the review code and intermediate datasets, letting others inspect the method that was used to inspect everyone else's code.[2][3]
Why a repository link is not the same as reproducibility
Analytical code depends on more than source files. Another team needs to know which software and package versions were used, how data were transformed, what command starts the analysis and which outputs should appear. Random seeds, tests and a small synthetic dataset can expose hidden assumptions. A licence determines whether reuse is legally clear; an archived release protects against a living repository changing after publication.
The review found examples of good practice, including step-by-step instructions, sample or synthetic data, fixed random seeds, typed functions and persistent archives. It also found hard-coded file paths and loosely referenced external utilities during manual review. Those details explain why binary code-availability policies can be satisfied while meaningful rerunning remains difficult.
Reproducibility is not identical to clinical validity. A perfectly rerunnable model can still be biased, poorly calibrated or useless in another hospital. Conversely, restricted patient data can prevent full public execution even when researchers document the model carefully. The value of better code sharing is that it enables faster error detection, method comparison and adaptation; it does not turn a prediction model into a safe clinical product by itself.[2]
What this means for patients, researchers and journals
For patients, the effect is indirect but important. Prediction models can influence who is investigated, treated or discharged. If external teams cannot inspect preprocessing, missing-data handling and implementation details, errors may survive longer and local validation becomes harder. Code sharing is therefore part of the evidence chain, not a guarantee that an individual will receive better care.
For researchers, the paper supports planning reproducibility at the start of a project. That means separating confidential data from shareable analysis code, documenting dependencies, providing a minimal runnable example and archiving the exact release associated with the paper. Funders and institutions also need to recognise maintenance work rather than treating a repository as a last-minute publication attachment.
For journals and reporting-guideline groups, requiring a URL is too weak. Policies can ask whether the link resolves, whether dependencies are versioned, whether documentation describes inputs and outputs, and how restricted data can be replaced with synthetic examples. The authors are developing a TRIPOD-Code extension; this review is intended to supply an empirical baseline for that work rather than pre-empt its final recommendations.[1][2]
What would change the assessment
A broader assessment would sample prediction-model studies regardless of whether they cite TRIPOD, include subscription articles and stratify results by statistical versus machine-learning methods, clinical specialty, country and journal policy. Independent reviewers should repeat the repository scoring and publish uncertainty around prevalence estimates, especially for subjective documentation features.
The stronger outcome is successful execution. Future work could draw a preregistered sample of repositories, recreate environments from scratch and measure how many analyses produce the reported outputs without author assistance. It should record failure causes—missing data, unavailable packages, undocumented preprocessing or non-deterministic results—rather than collapse them into one pass-or-fail number.
Nature Medicine published the peer-reviewed version on 9 October 2026; the earlier manuscript appeared on 16 March. The authors' own public code makes the central recommendation testable. Until larger execution studies are available, the evidence supports a focused claim: accessible analytical code remains uncommon even among reporting-guideline-aware clinical prediction studies, and a public repository often needs substantially more documentation before another team can rerun it.[1][2][3]