Can a saturated AI benchmark still reveal capability?

The answer: transcripts retained signal, but not a dependable safety verdict

Yes, in this retrospective experiment. When several models all received full marks on the easy portion of an evaluation, the way they reached the correct answer still contained information about how they would perform on harder items. A protocol that asked an AI judge to compare two correct transcripts predicted held-out capability substantially better than a baseline that knew only the saturated scores.

That does not turn old benchmarks into a permanent frontier test. The strongest protocol failed during a major change in model behaviour: judges calibrated on late-2024 systems underestimated the first reasoning models released in early 2025. The finding is therefore narrower and more useful than a claim that transcripts solve benchmark saturation. They can extend an evaluation's informative life, but only while the relationship between visible behaviour and capability remains stable enough to calibrate.

The paper is an unreviewed preprint submitted on 14 October by researchers affiliated with LASR Labs, the ELLIS Institute Tübingen, the Max Planck Institute for Intelligent Systems, the Tübingen AI Center, Sapienza University of Rome, Generality Labs and the UK AI Security Institute. Coefficient Giving funded the work through the LASR Labs fellowship. No independent replication was available when this article was checked.[1]

How the retrospective test was constructed

The researchers grouped model releases into six-month periods. For each prediction task, seven or eight models from the preceding period acted as references and eight models from the next period became targets. The main analysis used two target periods, producing 16 target-model predictions. Within each domain, the three strongest reference models defined saturation: an item counted as saturated only when all three answered it correctly. Any item at least one of them missed went into the harder, unsaturated set.

The protocol could see only correctly solved transcripts from the saturated items. It could not see the model's failures or the unsaturated questions. The researchers then fitted item-response models to the unsaturated set to create the capability target the protocols were trying to predict. Up to 100 saturated items were sampled per task, distributed across four item-difficulty quartiles. This design deliberately removed ordinary accuracy as a differentiator and asked whether process traces carried additional information.

The four domains combined established evaluations. Science included GPQA-Diamond, a fixed 200-question ARC-Challenge sample and science-tagged questions from a verified Humanity's Last Exam subset. Mathematics combined MMLU-Pro, AIME, APEX and HLE items. Coding used AgentBench OS, 50 SWE-bench Verified Mini tasks and 89 Terminal-Bench tasks. Cybersecurity used 39 CyBench challenges and 78 InterCode capture-the-flag challenges. The paper evaluated pairwise judging, single-transcript scoring, transcript embeddings and simple features such as length, hedging and tool-call counts.[1]

What improved, and where the information appeared

Across the 16 main target models, pairwise judging had mean squared prediction error of 0.26, with a reported 95% confidence interval from 0.18 to 0.34. The score-only baseline error was about 5.27 times larger. A likelihood comparison also favoured pairwise judging over the baseline (mean log-likelihood ratio 0.68, p=0.005). The feature, embedding and absolute-scoring approaches had lower error than the baseline numerically, but none produced a statistically significant likelihood advantage.

Ablations suggest that useful information was not merely polished writing. Removing visible reasoning increased error in coding, cybersecurity and science. Removing tool interactions increased cybersecurity error from 0.20 to 0.79, while making almost no difference in coding. Removing ordinary assistant messages did not significantly change error in any domain. This is evidence that the diagnostic channel depends on the work being evaluated, not that hidden reasoning is a universal measure of ability.

Pairwise judging is also expensive and reference-dependent. Each target transcript must be compared with calibrated reference transcripts, often in both display orders to reduce position bias. The appendix reports diminishing returns from adding more items when systematic judge bias dominates sampling noise. Strong anchor models in the reference set mattered, particularly for frontier targets.[1]

What this means for evaluators and public decision-makers

For evaluation teams, a ceiling score should not automatically consign a benchmark to the archive. Preserved transcripts can support a secondary analysis of strategy, verification, tool use and error recovery. That may buy time while harder tasks are designed and validated. It also argues for retaining structured execution traces, model and tool versions, prompts and item-level outcomes rather than publishing only a final percentage.

For regulators and safety institutes, the result is a warning against converting the new estimate into a pass-fail threshold without prospective validation. The measured target was performance on harder questions from the same broad evaluation ecosystem, not safe deployment, malicious capability or real-world reliability. A protocol that underestimates a new reasoning paradigm could create false reassurance precisely when model behaviour changes most.

A responsible use would keep the transcript estimate alongside unsaturated evaluations, expert review and uncertainty. It should be recalibrated when providers change reasoning visibility, tool interfaces or model architecture. Closed systems that hide internal traces also create an access problem: the main analysis favoured open-weight reasoning models with visible traces, although the paper includes separate transfer tests to closed models.[1]

Evidence limits and the test that would change the assessment

This was a retrospective simulation, not a prospective prediction challenge. The authors knew the historical models and could measure their later performance on held-out items. Item-response modelling supplies a useful common scale, but that scale still depends on the selected benchmarks, item parameters and model panel. Seven or eight reference models per period is a small calibration set, and judge behaviour can change with the judge model, prompt and visibility of reasoning.

The decisive next step is preregistered prospective testing. An evaluation body should freeze its items, reference models, judge protocol and uncertainty rules before a genuinely new model family arrives, then compare the prediction with results on independent harder tasks and expert red-team exercises. Repeating that across providers, hidden-reasoning systems and safety-relevant domains would show whether transcript analysis remains informative when it matters. Until then, it is a promising salvage method for saturated tests, not a substitute for new evidence.[1]