Which coding agent works best when a bug report unfolds over several turns?
In an unreviewed 72-task benchmark, the one-turn leader was not the eight-turn leader. The result suggests that coding-agent rankings depend on how long users can clarify a problem—but the users were simulated and the benchmark was built by Tencent researchers.
The short version
- What happened
- Tencent researchers evaluated 11 coding agents on 72 executable repository tasks: 22 two-issue tasks mined from open-source history and 50 tasks adapted from existing one-turn benchmarks.
- Who it affects
- Developers may experience one agent as decisive on a clear ticket and another as better after clarification. Procurement tests should therefore include the organisation's real conversational workflow, not only a one-shot leaderboard.
- What remains unproven
- This is an unreviewed preprint from Tencent Hunyuan researchers, not an independent comparison. No separate funding or competing-interest statement was reported.
Keep in mind
This is an unreviewed preprint from Tencent Hunyuan researchers, not an independent comparison. No separate funding or competing-interest statement was reported.
Only 72 tasks from 42 repositories were used. Fifty were adaptations of existing one-turn benchmarks, and the remaining 22 covered two linked issues rather than long projects.
Users were simulated from aggregate transition patterns in confidential commercial-assistant sessions. Real developers did not interact with the evaluated agents in this study.
Two model families helped judge and review results around executed tests; the adjudication protocol and equivalent-test rewrites can affect rankings.
Source analysis · 1 linked source. Primary analysed source date: 21 October 2026.
Read the full report · 5 min The evidence, methods, implications and limitations
The answer changes with the interaction budget
A newly posted preprint finds that there is no single coding-agent winner once a bug report becomes a conversation. On 72 repository tasks, Claude Fable 5.1 resolved 31.9% within the first turn, the best opening score. By the eighth turn, GPT-6-Astra had resolved 81.9%, narrowly ahead of Claude Opus 5 and Grok-4.6 at 80.6%. Claude Opus 5 led the paper's length-discounted score because it tended to finish successful tasks sooner.
That distinction matters for teams buying or evaluating coding assistants. A one-shot benchmark rewards an agent that acts quickly on a complete issue description. Real developers often start with an incomplete symptom, answer follow-up questions and correct a mistaken theory. The study's practical finding is therefore about evaluation design: rankings can change when agents get several chances to ask, edit and test—not proof that any named system is best for production software.[1]
How the researchers tried to imitate a developer conversation
The Tencent Hunyuan team assembled 22 tasks from linked issue histories in open-source repositories and converted 50 tasks from existing single-turn benchmarks. Every task included a repository state, a maintainer's patch and tests. The agent received information gradually, while tests ran after each turn. A task counted as resolved only when its required tests passed without breaking previously passing tests; an additional mechanism could recognise a functionally equivalent fix through a rewritten test that still failed on the original code.
The simulated user's wording came from eight persona patterns, such as a clear symptom with no suggested cause or a vague symptom paired with a wrong cause. Transition probabilities were calibrated on 20,859 sessions from an internally deployed commercial assistant; 13,530 of those sessions changed code. The researchers froze each task's information pool and persona sequence so competing agents faced the same trajectory. In a paired 50-task check, three of four selected agents scored better with an unrealistically clear, correct user; GPT-6-Astra's length-discounted score rose from 0.580 to 0.656.[1]
The distinction that matters
Why one leaderboard position is not enough
The same benchmark produces different leaders depending on what the evaluator rewards.
One turn
Measure who fixes the issue immediately
Claude Fable 5.1: 31.9%
Eight turns
Allow repeated clarification, edits and tests
GPT-6-Astra: 81.9%
Fewer turns
Discount successful runs that take longer
Claude Opus 5: highest LDPR, 0.640
Useful signal, but not a field trial
The benchmark also relies partly on AI to judge AI. Executed tests provided the primary evidence, but a Judge model wrote each verdict and a Reviewer from another model family could rewrite it. Reviewers changed 40.9% of scored-turn decisions, which shows that a single model judge would have materially altered results. It also means the leaderboard depends on this particular two-model adjudication design and its rules for equivalent patches.
The 72 tasks cover one maintenance episode, not months of collaborative development, architecture work, security review or deployment. The simulated users reproduce aggregate conversational patterns rather than identifiable people, and the commercial session data cannot independently verify that the personas represent other organisations. The paper is unreviewed and written by the company whose Hunyuan model is among those tested. A public release, independent replication and trials with real developers on longer projects would make the ranking claims more useful.[1]