When AI work looks faster, who counts the mistakes? A preprint tests the missing ledger
New analysis today of an unreviewed 23 October preprint: reanalysis shows that one productivity dataset supports different slowdown estimates, while 40 paired AI-review trials found that an inflated reliability claim cut checking and let wrong figures through. This does not measure human worker productivity.
The short version
- What happened
- The unreviewed preprint reanalyses 246 software issues completed by 16 developers and finds that the same randomized dataset supports a 19% geometric-mean slowdown or a 24% increase in total hours, depending on the summary used.
- Who it affects
- Workers may be judged against speed metrics that stop before checking, rework or customer harm appears. Recording those outcomes can change which tool or workflow looks productive, but this preprint does not estimate the effect on human jobs or wages.
- What remains unproven
- Unreviewed preprint by one independent author
Keep in mind
Unreviewed preprint by one independent author
Claude used substantially in research and writing
Author works on an AI-software company not evaluated in the paper
One drafter model and one reviewer model
Seeded rather than naturally occurring errors in the decisive experiment
40 paired drafts in the over-trust comparison
Prompt-assigned time and error costs
No human reviewers or workplace deployment
Wide or model-dependent bounds in parts of the reanalysis
Source analysis · 1 linked source. Primary analysed source date: 23 October 2026.
Read the full report · 5 min The evidence, methods, implications and limitations
One dataset can produce more than one honest productivity headline
A new unreviewed preprint argues that an AI tool's visible time saving should be read beside the errors and later work it creates. The author reanalysed public data from METR's randomized study of experienced open-source developers: 246 issues completed by 16 people. METR's published specification estimated that access to early-2025 AI tools made work 18.8% slower. Using a different but defensible summary—the ratio of total mean time—the reanalysis estimates 24.0% more hours after adjusting for developers' forecasts, or 33.7% without that adjustment.
Those figures are not competing experiments. They describe the same observations with different estimands: the geometric mean asks about a typical proportional change, while the ratio of means is closer to total hours consumed. The preprint also calculates very wide partial-identification bounds: the data are compatible with AI helping on 2.6% to 82.1% of issues. That range has no sampling-error interval and should not be read as a precise success rate. It shows how little an average effect reveals about who or which tasks benefited.[1]
A small experiment made over-trust faster—and less reliable
The paper then tests one mechanism with AI agents. Draft briefs were built from US Securities and Exchange Commission filings dated after the models' training cutoff. A Claude Haiku model drafted the briefs and Claude Sonnet reviewed nine checkable figures. In the decisive preregistered comparison, the true draft-error rate was 30%. Across 40 paired drafts, the reviewer was either told that truth or told the drafter erred only 3% of the time.
Given the inflated reliability claim, the reviewer made 7.2 fewer lookups out of nine and allowed a wrong figure through in 22.5% of briefs, compared with none in the truthful condition. The reported difference was 22.5 percentage points, with a cluster-bootstrap 95% interval from 10 to 35 points. Every time the reviewer inspected a line exposing the seeded error, it corrected it; the failures arose because it checked less. A warning to treat the claimed track record as unverified did not restore the missing effort.[1]
The distinction that matters
Stopwatch versus ledger: what is being counted?
The paper separates visible task time from the checking and escaped errors that determine whether a workflow saved time overall.
Stopwatch
Time the visible task
Reports faster completion before downstream consequences
Verification
Record checks and corrections
Shows where review time is spent or skipped
Ledger
Add escaped-defect costs
Finds the defect cost that erases the apparent saving
The result is a measurement warning, not a verdict on workers
The experiment does not show that people behave like these agents or that workplace AI usually creates more errors than it saves. It tested one reviewer model, one structured financial task and costs written into the prompt. The defects were seeded; the planned comparison with natural errors could not run because the drafter made none in 140 briefs. The author also used Claude extensively to search, code, run the experiment and draft the paper, then reviewed the work. The paper is not peer reviewed.
Its useful proposal is procedural: before calling an AI workflow faster, name the time summary, record the checking performed and count mistakes that escape. Organisations can then ask a concrete question: how costly would each missed defect have to be before the apparent saving disappears? For procurement and workforce decisions, that ledger is more informative than a stopwatch alone—but it still needs testing with human reviewers, real consequences and independently replicated tasks.[1]