What breaks when an AI agent meets a messy customer brief?
Across 95 simulated delivery jobs, the leading configuration resolved only 14 without an incident. Finding the missing facts was necessary—but still not enough to deliver the right result.
The short version
- What happened
- FDE-Gym contains 95 tasks derived from 20 delivery scenarios across 18 sectors, with production-like systems and scripted stakeholders.
- Who it affects
- Workers may spend less time implementing routine changes but more time defining the real problem, checking constraints and owning deployment decisions.
- What remains unproven
- The tasks are controlled simulations with scripted stakeholders and no completed human baseline. The benchmark operator is also an interested platform developer.
Keep in mind
The tasks are controlled simulations with scripted stakeholders and no completed human baseline. The benchmark operator is also an interested platform developer.
Source analysis · 1 linked source. Primary analysed source date: 15 October 2026.
Read the full report · 4 min The evidence, methods, implications and limitations
Agents could build the requested thing and still miss the real problem
An AI agent can complete a polished piece of work and leave a customer no better off. In a new benchmark built around ambiguous enterprise projects, the strongest tested configuration resolved only 14 of 95 tasks. The main failure was not simply coding. Agents struggled to discover unstated rules, decide when a sponsor's request targeted the wrong problem and turn what they learned into a working improvement.
FDE-Gym adapts 20 delivery projects from China, Singapore and the United States into 95 tasks across 18 sectors. Each workspace contains a production-like system, documents, historical data and more than 130 scripted stakeholders. The agent can ask questions and alter the system. Its output is then deployed unchanged and scored on held-out data using the customer's own measure, such as cleared reconciliation breaks or correctly handled support cases.[1]
The test rewards business improvement, not literal compliance
The unchanged system scores zero and an expert reference scores one. Contacting stakeholders, running trials and waiting can reduce the score; an operational incident wipes out positive uplift. A task counts as resolved only at 0.8 or above without an incident. Thirteen configurations from seven providers each ran 152 task-and-instruction combinations three times, capped at 200 turns, 150 minutes or eight US dollars per run.
At the hardest instruction level, the leading configuration averaged 0.56, with a 95% bootstrap interval of 0.50 to 0.62. Five configurations resolved no tasks. Giving leading models the relevant rules and constraints raised their scores to 0.73–0.82; requiring them to uncover those facts reduced them to 0.39–0.62. Yet discovery was not the whole answer. On 23 tasks where the leading model obtained every key fact, only 10 achieved at least 0.8 of the reference business uplift.[1]
The distinction that matters
Why a completed deliverable can still fail
FDE-Gym separates the sponsor's request from the customer's underlying result.
Brief
Receive an incomplete or misdirected request
The visible task may not describe the real problem
Discovery
Question stakeholders and inspect systems
Rules and constraints become available
Delivery
Deploy the work on held-out data
Only measured customer uplift earns credit
This is a warning against removing the delivery team too early
For workers, the result suggests that requirements gathering, stakeholder judgement and accountable sign-off remain central skills even when an agent can edit code and operate tools. A manager should not read benchmark success on tidy tickets as evidence that a system can safely interpret a vague operational goal. The benchmark also found weaker models making some simulated customers worse off than the unchanged baseline.
The evidence is still a simulation. Stakeholders followed scripts, companies were de-identified or fictionalised, and the 75 variants and reference solutions are held out. A human baseline is being collected but was not available, so the study cannot say whether agents are close to experienced engineers. It also cannot show whether mixed human-agent teams would outperform either side alone. DeepLethe runs the benchmark and leaderboard, creating a platform interest the authors disclose. Independent replication with real delivery teams, live organisational constraints and audited incidents would change the assessment. For now, the practical lesson is to evaluate agents on the customer's outcome—and keep people responsible for discovering what the brief leaves unsaid.[1]