Coding agents finished the job—but did they preserve what they changed?
Six frontier models completed 86.1% of 205 executable tasks yet backed up before the benchmark's regret point only 15.2% of the time. A plug-in layer improved recoverability, but the benchmark and evaluator were built by the same team.
The short version
- What happened
- RegretBench turns 205 regret reports from GitHub, Stack Overflow and other public posts into executable multi-turn tasks spanning local and external state.
- Who it affects
- Developers and other workers can lose files, configuration or remote records when an agent overreaches. Default recovery points could reduce harm, provided people can control retention and sensitive data are protected.
- What remains unproven
- The unreviewed benchmark translates public regret reports into authored tasks and may miss agent-specific or domain-specific failures.
Keep in mind
The unreviewed benchmark translates public regret reports into authored tasks and may miss agent-specific or domain-specific failures.
The system's designers also built the benchmark and primary evaluator; independent reproduction is absent.
Character-level similarity can miss semantic or operational differences, while indiscriminate backup can create privacy and storage risks.
Samsung supported the work; the authors say the findings do not necessarily reflect the funder's views.
Source analysis · 1 linked source. Primary analysed source date: 20 October 2026.
Read the full report · 3 min The evidence, methods, implications and limitations
Capability did not imply reversibility
The team collected 205 public moments where people said they regretted not preserving prior state, then converted each into an executable multi-turn task. Changes included files, databases, messages and remote-service state. Six frontier models ran in coding-agent harnesses. They completed 86.1% of tasks on average but made a usable backup before the annotated regret action in only 15.2%.
That gap matters because a high task-completion score can hide irreversible side effects. The authors' audit of five mainstream harnesses found workspace or native-edit protection in several products, but no general coverage of state changed through external tools. RegretBench therefore scores both whether a backup occurred before the critical action and whether it held enough information to recover the affected state.[1]
An external backup layer improved recovery at a cost
BackUpper intercepts state-changing tool calls. Simple operations use rules; ambiguous calls go to a small backup subagent, with prior decisions distilled into reusable rules. On the 205 tasks it covered every annotated regret action and restored state to 92.2% average character-level similarity, versus 14.3% without restoration and 33.8% for post-hoc reconstruction. Reported overhead averaged $0.014 per task.
Coverage is not the same as complete recovery: regenerated identifiers and metadata caused residual differences, and unnecessary backups can create privacy, storage and retention costs. Human annotators confirmed 94% of judge-positive backups, while personalised scope was tested on 60 tasks. The benchmark derives from human regret reports that may not capture agent-specific failures, and the designers evaluated their own system. Independent, adversarial and longitudinal tests would change the assessment.[1]
The distinction that matters
Reversibility before a tool call
The safeguard sits outside the main coding agent and acts before state changes.
Inspect
Check each proposed tool call
Identify affected state
Preserve
Use a rule or backup subagent
Record enough state for recovery
Restore
Replay the backup after regret
Measure how closely the original returns