Can METR’s AI time horizons be read literally?
Not as a simple ruler, this statistical reanalysis argues. Across 228 software tasks and 26 AI systems, more flexible models predicted held-out task families better than METR’s shared logistic curve and suggested that equal multipliers in human task time do not represent equal jumps in difficulty. The overall capability trend may still be real.
The short version
- What happened
- The authors reanalysed 228 software tasks, 26 AI systems and 79 task families from METR Time Horizon 1.1.
- Who it affects
- A benchmark-hour estimate is not promised unattended work time.
- What remains unproven
- Unreviewed preprint analysing one public software-task dataset.
Keep in mind
Unreviewed preprint analysing one public software-task dataset.
Human times typically use about four attempts; 29% of tasks use expert estimates.
The suite covers software work and durations up to about 30 hours.
Comparisons depend on sampled families and statistical specifications.
Agreement between fitted ability estimates does not prove transfer.
Source analysis · 2 linked sources. Primary analysed source date: 8 October 2026.
Read the full report · 6 min The evidence, methods, implications and limitations
The direct answer: informative, but not a literal stopwatch
The preprint argues that METR's time-horizon graph should not be read as though human task duration were a uniform ruler of AI capability. A model's horizon is derived from the human completion time of tasks on which it is estimated to succeed at a chosen probability. That creates an appealing interpretation: a longer horizon sounds like an ability to carry out proportionally longer work. The audit finds that the mapping from human time to model difficulty is not that simple.
Using the same public task results, the authors fit more flexible models and predicted held-out task families better. Their curve is nearly flat across roughly two to 30 minutes of human time before becoming steeper. Moving from three to 30 minutes is therefore not equivalent to moving from 30 minutes to five hours, even though both are tenfold increases. The result challenges comparisons between multipliers; it does not show that the overall rise in model performance is fictitious.[1][2]
What data were reanalysed
The analysis uses public METR Time Horizon 1.1 data: 228 software-engineering tasks, 26 AI systems and 79 task families. A family groups related items, which matters because variants of the same problem may share difficulty features. Model success is observed over repeated attempts, while human completion time supplies the explanatory scale intended to make tasks comparable.
Human times are not exact constants. The paper says they were typically derived from about four attempts per task. For 29% of tasks, a valid human measurement was unavailable and an expert estimate was used. That adds uncertainty to the horizontal axis that a clean graph can hide. The meaning of an hour also depends on who performed the task, under what conditions and how estimation replaced observation.[1]
The baseline and two alternatives
The baseline is a shared-slope logistic model: each AI receives a position on a common curve linking log human time to success probability. The audit compares it with a shared monotone spline, which lets the curve bend while preserving the expectation that longer tasks are generally harder, and an explanatory item-response model. The latter gives tasks latent difficulty, models family structure and uses beta-binomial overdispersion.
If human time is a stable one-dimensional measure of difficulty, a simple shared curve should predict new families well. If family identity and a nonlinear relationship matter, a horizon can still summarise performance, but its numerical scale is a constructed benchmark quantity rather than a direct estimate of how long an autonomous system can work in any environment.[1]
How predictive performance was compared
The authors used five-fold cross-validation at task-family level. That is stronger than randomly holding out individual variants because it asks whether a model generalises to related groups it did not see during fitting. They evaluated 12 combinations of scoring rule and weighting: marginal log score, Brier score and smoothed elementary scores at 0.5 and 0.8 success thresholds, each under three weighting schemes.
Both proposed models beat the baseline on every displayed scoring combination. Consistency across metrics is more informative than one favourable number, although all comparisons still come from the same dataset and modelling choices. The spline and item-response approaches are not identical explanations; their shared advantage indicates that the simple curve leaves predictable structure unused.[1]
The alternative still recovers an overall ability trend
The item-response model yields an ability parameter for each system. After rescaling, those values have an R-squared of 0.996 with METR's estimates, according to the paper. The authors interpret this as evidence that the published plot may track a Rasch-like latent ability well even if the time label is not equally spaced in substantive difficulty. Ranking and broad progress can be robust while literal duration comparisons are not.
This prevents two overreactions. The audit does not warrant saying time-horizon research measures nothing. Nor does a strong correlation between two fitted summaries prove that either transfers to unrestricted work. Both are constructed from the same task suite, whose coverage and measurement choices bound what the latent ability means.[1]
Practical meaning, limits and what would change the assessment
A statement that horizons doubled or increased tenfold is incomplete unless readers know where on the fitted curve the change occurred, which task families support it and how uncertain the estimate is. Procurement and safety decisions should not use small differences as hard operational thresholds. If a decision depends on whether a system can handle work lasting a specific number of hours, it needs direct evaluation on representative tasks at that duration and in that environment.
The paper is an unreviewed statistical audit of software tasks up to about 30 human hours. It assumes the sampled tasks are useful probes and does not decide whether human time is the best measure for every kind of work. Confidence would rise with preregistered comparisons on new families, more measured human baselines, explicit modelling of time error, longer tasks and external tests beyond software.[1][2]