Can an LLM selector make coding agents more reliable? Gains came with up to 8.7× the tokens
On 89 terminal tasks, generating four or eight candidate actions at every turn and letting a stronger model choose improved reported accuracy by 5–10 percentage points. Executed trajectories shortened, but total token use rose sharply and the preprint has not been peer reviewed.
The short version
- What happened
- The unreviewed study used 89 Terminal-Bench v2.1 tasks, two generator models, two selectors, best-of-four and best-of-eight settings, and four trials per task.
- Who it affects
- Software teams may get more reliable benchmark completion by spending substantially more inference at each decision.
- What remains unproven
- Unreviewed preprint on one 89-task terminal benchmark.
Keep in mind
Unreviewed preprint on one 89-task terminal benchmark.
Selectors were more capable than generators, complicating attribution.
No wall-clock, financial, energy, security or human-quality outcomes.
Behaviour analysis used an LLM judge on a subset rather than independent human adjudication.
Source analysis · 1 linked source. Primary analysed source date: 4 November 2026.
Read the full report · 8 min The evidence, methods, implications and limitations
The finding: choosing among several next steps helped, but it was not free
An MBZUAI Labs preprint reports that a coding agent became more accurate and more consistent when it generated several possible actions at every turn and delegated the choice to a stronger language model. On Terminal-Bench v2.1, the paper says turn-level best-of-four and best-of-eight selection improved task accuracy by about five to ten percentage points over the single-sample baseline. Executed trajectories also used roughly 40% to 50% fewer turns. The result suggests that some agent failures arise from a poor local choice rather than an inability to propose a useful action at all.
The efficiency claim needs its denominator. The agent executed fewer shell or editing steps, but it generated four or eight candidate actions before each selected step and paid for an additional selector prompt. Across the reported configurations, total token use rose to approximately 3.6 to 8.7 times the baseline. The experiment therefore trades parallel inference for shorter visible trajectories. That may be worthwhile when a failed task is expensive or candidate generation can run in parallel, but it is not evidence of lower operating cost, faster completion or lower energy use. Those outcomes were not measured.[1]
How turn-level best-of-N changes an agent loop
A conventional agent observes the current terminal state, generates one action, executes it and then observes the result. In the proposed loop, the generator produces N candidate actions from the same state. A separate selector sees the candidates and chooses one before any of them is executed. Only the chosen action changes the environment. The paper tested N equal to four and eight, with N equal to one as the baseline and an exploratory N equal to two that performed roughly like baseline. Selection was listwise: the selector compared the complete candidate set rather than scoring each option independently.
This architecture resembles inference-time scaling in reasoning systems, but the location of the extra computation matters. It is spent repeatedly at every interactive turn rather than once on a final answer. That lets the selector reject an early command likely to send the trajectory down a dead end, yet it also multiplies generation throughout a long task. The authors used Qwen3.6-35B and K2-Horizon-375B as generators, with Kimi-K3 and DeepSeek-v4.1-Flash as selectors. The selector was deliberately more capable than the generator, so the paper does not show that any extra model can provide the same benefit.[1]
The distinction that matters
Fewer actions did not mean less computation
The selector shortened the executed path by evaluating many actions that were never executed.
Observe
Read the current terminal state
One shared context
Generate
Propose 4 or 8 next actions
Inference multiplies
Select
A stronger model chooses one
Only one action executes
Trade-off
Repeat every turn
5–10 point gains, 3.6–8.7× tokens
The benchmark used 89 terminal tasks and four trials per task
The evaluation used Terminal-Bench v2.1, a set of 89 tasks performed in terminal environments, with mini-SWE-agent as the agent scaffold. Tasks can require inspecting files, writing or repairing code, running commands and satisfying an automated verifier. Each model and sampling configuration was run four times per task. Repetition allows the authors to examine both average task accuracy and consistency across runs. It does not produce 356 independent task types: the underlying challenge set remains 89, and repeated trials may share the same benchmark artefacts and scoring assumptions.
Terminal-Bench is useful because actions change state and mistakes can compound, making it more representative of agent operation than a static multiple-choice test. It is still one benchmark centred on software and terminal work. The paper does not cover office documents, web research, customer service, financial transactions, medical workflows or robots. It also does not evaluate whether a human engineer would accept the resulting patch, how often a technically passing solution is maintainable, or whether agents introduce security weaknesses. General claims about autonomous work should therefore remain separate from the measured terminal-task result.[1]
Shorter executed trajectories concealed much larger inference budgets
For the K2-Horizon generator, the baseline averaged 69.5 executed turns and about 72,000 tokens. With Kimi-K3 selecting four candidates, executed turns fell to 41.0, while the reported generator and selector totals were about 196,000 and 67,000 tokens—roughly 3.6 times baseline in all. At best-of-eight, turns fell to 33.9, but generation and selection rose to about 334,000 and 69,000 tokens, or 5.6 times baseline. DeepSeek selection produced a similar trajectory reduction and total-token multipliers of about 3.9 and 6.3 for N equal to four and eight.
The smaller Qwen generator started at 64.9 turns and about 50,000 tokens. Kimi selection reduced executed turns to 40.5 at best-of-four and 34.0 at best-of-eight, while total tokens reached roughly 5.2 and 8.2 times baseline. With DeepSeek as selector, the corresponding totals were about 5.7 and 8.7 times baseline. These are benchmark-level averages reported by the authors, not a price quotation. Provider pricing, batching, caching, parallelism and hardware utilisation would change financial and latency outcomes. The study did not convert tokens into money, wall-clock time or energy.[1]
Behaviour analysis suggests fewer repeated mistakes, with limited evidence
To understand the improvement, the authors examined 36 hard tasks and 360 trajectories with a language-model judge. They report that selection reduced unproductive repetition and made agents more likely to recover from an earlier mistake or choose a promising diagnostic action. This is consistent with the mechanism: when several alternatives are visible, the selector can prefer an information-gathering command or avoid repeating a failed approach. Fewer executed turns can therefore reflect better local decisions rather than simply more concise text. The analysis is descriptive and based on a subset of the benchmark, not an independently blinded human error study.
The design also leaves an important comparison unresolved. Because the selector is stronger, some of the gain may come from giving a better model control over each decision rather than from best-of-N diversity itself. The paper does not report the stronger selector acting as the generator under the same full harness and budget, nor does it test a selector weaker than the generator. Best-of-two was approximately baseline, suggesting that candidate count alone is not a smooth guarantee of improvement. Model-specific candidate diversity, calibration and the selector's ability to recognise good actions may all determine whether the method helps.[1]
What engineering teams can test—and what would change the assessment
For teams operating agents, the study offers a concrete design option for high-value steps: generate alternatives before an irreversible action and use an independent chooser. It may be more sensible to apply that selectively around destructive commands, uncertain patches or repeated failures than to multiply every turn. Operators should measure end-to-end completion, wall-clock latency, total tokens, monetary cost, security defects and human review time together. A shorter trace is not automatically a cheaper or safer trace, and a benchmark pass is not equivalent to a production-quality change.
Confidence would rise with preregistered tests on additional interactive benchmarks, direct comparisons against spending the same compute on a larger single agent, human review of patch quality, and measurements of latency, energy and cost. Ablations should test weak and equal-capability selectors, pairwise versus listwise choice, adaptive N, and selective use only when uncertainty is high. Field studies should ask whether developers resolve real tickets faster without accepting more defects. The accessible preprint reports no funding, acknowledgement or competing-interest statement and has not been peer reviewed, so independent reproduction is especially important.[1]