AI agents improved 20 scientific benchmarks—but did they do science?
Four frontier coding agents beat a published result somewhere in every benchmark, but their gains mostly came from familiar machine-learning engineering. One run per agent-task pair and no human-researcher comparator make this a probe, not proof of automated discovery.
The short version
- What happened
- Agent4Sci packages 20 benchmarks, 25 tasks and 11 scientific domains into verified environments with train, validation and sequestered test splits.
- Who it affects
- Researchers could use this kind of agent to explore engineering variants or reproduce baselines, but scientific teams still need domain experts to judge novelty, plausibility and downstream consequences.
- What remains unproven
- This is an unreviewed preprint with one main run per agent-task pair for most comparisons and no complete estimate of run-to-run uncertainty.
Keep in mind
This is an unreviewed preprint with one main run per agent-task pair for most comparisons and no complete estimate of run-to-run uncertainty.
The comparator is the published best model, not a human researcher working with the same compute and time budget.
Better held-out metrics can reflect competent engineering rather than new scientific understanding.
The work was conducted through the Anthropic Fellows Program, and two authors are affiliated with Anthropic; the study includes Anthropic and competing models.
Multi-source analysis · 2 linked sources. Primary analysed source date: 20 October 2026.
Read the full report · 3 min The evidence, methods, implications and limitations
A broad test found progress, mostly through engineering
The researchers rebuilt published scientific machine-learning tasks inside controlled containers, reproduced each baseline and withheld the test split during hill-climbing. Four frontier agents then received the paper, code, training data and a target metric. Each main run had up to 12 hours on eight H200 GPUs; a separate agent evaluated the saved artifact, while another audited the trajectory for test contamination, hard-coding and other gaming.
Across 20 benchmarks and 25 tasks, at least one agent improved every published best result. Mean relative gains were 10.4% for Claude Opus 4.6, 9.6% for Sonnet 4.6, 6.8% for Gemini 3.1 Pro and 4.4% for Codex GPT-5.4. Yet the trajectory review said the agents usually reached for standard optimisers, augmentations and architecture swaps. They rarely formed or tested a genuinely domain-grounded hypothesis.[1][2]
A scientific benchmark gain is not a scientific discovery
Three plug-in skills prompted literature-grounded ideation, experiment management and scheduled self-checks. In the reported comparison, they raised Opus from 10.4% to 20.1% and Codex from 4.4% to 8.1%. The public repository exposes the runner, benchmark registry and protocol, which makes the setup inspectable, but it does not supply an independent reproduction of the headline results.
The main evidence remains high variance: most agent-task combinations had one long run, and only six benchmarks received repeat runs. There is no matched human-researcher baseline, and success on a held-out metric does not show that a result is novel, scientifically valid or useful outside the benchmark. The project is funded through Anthropic's Fellows Program and evaluates Anthropic models alongside rivals. Wider replication, repeated runs and expert review of claimed insights would change the assessment.[1][2]
The distinction that matters
How Agent4Sci closes common evaluation gaps
The benchmark separates model development, final scoring and misconduct checks.
Hill-climb
Agent edits code using train and validation data
Produces a candidate artifact
Evaluate
Fresh agent scores a sequestered test set
Measures transfer beyond the visible split
Audit
Another agent inspects the full trajectory
Looks for contamination and metric gaming