AI papers share far more code and data—but are they really more reproducible?
A peer-reviewed analysis of 56,800 conference papers found large gains in open artefacts and documentation from 2014 to 2024. Its 64% reproducibility estimate is modelled, not the result of rerunning thousands of studies.
The short version
- What happened
- Researchers analysed 56,800 papers from five major AI conferences between 2014 and 2024; 52,328 were classified as empirical.
- Who it affects
- Researchers, reviewers and funders get more material with which to inspect and reuse AI results. But a repository link is not a guarantee that another team can reproduce a claim within realistic time and computing budgets.
- What remains unproven
- The study covers five leading conferences, not journals, workshops or the full global AI literature.
Keep in mind
The study covers five leading conferences, not journals, workshops or the full global AI literature.
A language model classified documentation indicators; manual evaluation found lower accuracy for several nuanced categories.
Estimated reproducibility relies on a small earlier reproduction sample and is not a direct rerun of the corpus.
The observational checklist analysis cannot isolate policy effects from existing trends or venue selection.
Funding came from the US National Science Foundation, the Research Council of Norway and an Alexander von Humboldt Professorship; the paper reports no commercial sponsor for the analysis.
Source analysis · 1 linked source. Primary analysed source date: 22 October 2026.
Read the full report · 4 min The evidence, methods, implications and limitations
Open artefacts and basic reporting improved sharply
The authors collected 56,800 papers from AAAI, ICLR, ICML, IJCAI and NeurIPS over 2014–2024, then identified 52,328 as empirical. They checked seven features: open code, open data, dataset-split details, pseudocode, hardware, software dependencies and experiment setup. Across the decade, the share reporting both open code and open data rose from 11% to 64%. Papers documenting at least five of the seven indicators increased from 8% to 43%. Hardware and setup reporting also improved, though unevenly across venues and topics.[1]
A language model did the large-scale classification
Gemini 2.5 Flash classified the PDFs after prompt development on 400 manually annotated papers. The researchers then evaluated the final classifier on 160 randomly sampled papers. F1 scores exceeded 90% for most indicators, but were lower for dataset splits (81.48%), dependencies (81.25%) and experiment setup (86.21%). That is adequate for broad trend analysis, not perfect measurement of every individual paper. The authors publish their paper-level indicators, code and analysis data on Zenodo, making the trend calculation open to reanalysis.[1]
The distinction that matters
What changed—and what was inferred
The study separates observable reporting from an estimated chance of reproduction.
Open code + data
Compare conference papers across a decade
11% in 2014 → 64% in 2024
Five or more indicators
Count seven documentation features
8% in 2014 → 43% in 2024
Estimated reproducibility
Map documentation to prior reproduction results
28% → 64%, not directly retested
The 64% figure is an estimate, not a reproduction rate
The study estimates that reproducibility rose from 28% in 2014 to 64% in 2024. It derives that estimate by linking documentation patterns to a much smaller prior reproduction exercise, where six of seven papers with open code and data were reproduced, compared with five of 15 that shared datasets alone. The new study did not rerun 56,800 experiments, verify that repositories still execute or test whether released artefacts reproduce claimed results. Documentation is a necessary signal, but it can overstate practical reproducibility when versions, licences, compute or hidden preprocessing are missing.[1]
Checklists did not show a clear additional acceleration
The paper compared trends around conference reproducibility checklists and found no statistical evidence of an overall acceleration after adoption: 15 of 35 indicator trends were higher after a checklist, with p=0.84 and a small negative aggregate effect size. That cannot establish that checklists do nothing. Adoption years differed, reporting was already improving and authors select which venues to enter. Stronger evidence would directly rerun representative samples, test whether artefacts remain usable years later and compare matched venues or policy changes. For now, the robust result is narrower: AI research became much more open and better documented, while actual repeatability remains less certain.[1]