AI agents reproduced 13 of 93 scientific tasks—what failed?
PaperBenchX replayed agent submissions across 12 research areas and scored only regenerated evidence. The best configuration fully completed 13 tasks, while validation remained the weakest stage; 81 tasks and most reference workflows stay private.
The short version
- What happened
- PaperBenchX contains 93 expert-constructed reproduction tasks from 93 papers in 44 venues, spanning 12 research areas and 3,168 scorable criteria.
- Who it affects
- Scientists could use agents to draft models, operate tools or diagnose runs, but weak validation can turn a plausible workflow into unsupported evidence. Researchers and reviewers still need inspectable files, domain judgment and responsibility for the final claim.
- What remains unproven
- Unreviewed preprint submitted 29 October 2026; no independent benchmark replication was identified.
Keep in mind
Unreviewed preprint submitted 29 October 2026; no independent benchmark replication was identified.
Only 12 of 93 tasks are public; 81 tasks and private reference workflows remain controlled by the benchmark team.
A GPT-5.6-sol evaluator applied the expert-written rubrics, so scores include a model-judge layer.
The paper reports one outcome per model–task configuration rather than repeated stochastic trials.
Three public tasks require separately licensed commercial software, limiting equal access.
The project includes industry contributors, and no separate funding or competing-interest statement was identified.
Multi-source analysis · 2 linked sources. Primary analysed source date: 29 October 2026.
Read the full report · 4 min The evidence, methods, implications and limitations
The benchmark requires executable evidence
An unreviewed preprint tests whether AI agents can reproduce published computational experiments outside the familiar software stacks of machine-learning research. PaperBenchX contains 93 tasks drawn from 93 papers in 44 venues, across electromagnetics, photonics, computational chemistry, materials science, bioinformatics, robotics and embodied AI. Experts converted each target experiment into a runnable environment and 3,168 scorable rubric criteria—a median of 38 per task.
An agent receives the paper, a public task specification and access to domain-native software. It must infer the intended model, run the computation and produce a workflow that supports the target claim. Evaluators delete mutable outputs, disable outbound network access and replay each submission from scratch. Only regenerated artefacts count. That design is stronger than checking whether an agent writes a plausible report or matches a reference chart visually.[1][2]
Partial progress rarely became a complete reproduction
Across ten model–scaffold configurations, the best—GPT-6-Astra with Codex—earned 59.08% of available rubric credit on average. It passed the benchmark's operational full-completion threshold of more than 90% on 13 of 93 tasks, or 13.98%. Across all 930 model–task outcomes, only 80 exceeded that threshold. Seventy-one tasks were never fully reproduced by any configuration, and another 11 were completed by no more than two.
The failure pattern is revealing. Macro-average scores were 62.73% for modelling, 61.83% for execution and 42.91% for validation. Agents often reconstructed part of the setup or ran scientific software, but struggled to establish that outputs actually supported the paper's claim. Tasks had time limits from four to 72 hours, with a seven-hour median, so this is a demanding autonomy test rather than a short question-answer benchmark.[1]
The distinction that matters
Where reproduction broke down
The agents made more progress constructing and running experiments than validating what the regenerated evidence meant.
Modelling
Recover assumptions, equations and experiment setup
62.73% macro-average score
Execution
Operate domain software and complete computation
61.83% macro-average score
Validation
Show regenerated evidence supports the target claim
42.91% macro-average score
The result is a warning, not a universal ceiling
The benchmark does not show that AI cannot assist scientists. It tests one demanding form of end-to-end autonomous reproduction, and partial credit may still represent useful work under expert supervision. Nor does a score above 90% prove a scientific claim true; it means the submission satisfied this benchmark's expert-written criteria. The same GPT-5.6-sol evaluator applied every rubric, introducing a model-judge layer even though the underlying criteria were expert validated.
Independent inspection is also limited. Twelve representative tasks are public, while 81 and their private reference workflows remain held out to reduce contamination. Three public tasks depend on commercial software that users must license. Because task selection, reference workflows and scoring choices shape the result, outside teams cannot yet audit most task-level failures or test whether the private set favours particular tools. The project lists contributors from UniPat AI, ByteDance Seed and Peking University, and the paper gives no separate funding or competing-interest statement. The assessment would change with repeated runs, blinded external expert scoring, broader public task access and independent replications on new papers. For now, the evidence says autonomous research agents need especially strong validation gates before their scientific outputs are trusted or published.[1][2]