DeepSeek’s open AI update: what its agent benchmarks show
DeepSeek reports stronger agent and coding benchmarks. Its own model card supplies the results; independent checks and real-work performance remain open questions.
The short version
- What happened
- DeepSeek released MIT-licensed V4.1-Flash-1017 weights on 17 October and reports higher scores than its previous V4.1-Flash release on every benchmark in the new comparison table.
- Who it affects
- Engineers can test a permissively licensed model without sending every task to a closed provider, but employers should not turn a vendor benchmark into a promise of faster or safer work without testing real users, errors and operating costs.
- What remains unproven
- The central evidence is a developer-run comparison. Run counts, uncertainty, complete prompts, trajectories, cost controls and independent replication are absent; the technical report is an unreviewed company preprint for the underlying architecture rather than the 1017 update.
Keep in mind
The central evidence is a developer-run comparison. Run counts, uncertainty, complete prompts, trajectories, cost controls and independent replication are absent; the technical report is an unreviewed company preprint for the underlying architecture rather than the 1017 update.
Source analysis · 3 linked sources. Primary analysed source date: 17 October 2026. Earliest linked supporting source: 17 September 2026.
Read the full report · 5 min The evidence, methods, implications and limitations
The update improves DeepSeek's own comparison line
DeepSeek released V4.1-Flash-1017 on 17 October as a minor update to its V4.1-Flash model. The official repository supplies downloadable weights under the MIT licence and records the release commit at 05:46. DeepSeek says the update improves reasoning, coding, agentic work and visual tasks while shortening execution time and outputs. Its comparison table shows a higher score than the previous release on all 11 listed benchmarks: Terminal-Bench rises from 31.2 to 42.3, ProgramBench from 20.3 to 29.1 and Automation-Bench from 54.8 to 60.5. Those are benchmark movements, not measured gains in a workplace or public service.[1][2][3]
It beats named rivals on several tests, not all of them
The most striking company-reported results are on agent and software tasks. DeepSeek lists 77.0 on DeepSWE, against 73.0 for GPT-5.6 Sol and 74.0 for Opus 5; 60.5 on Automation-Bench, against 45.8 and 50.3; and 72.1 on Apex Bench, against 54.3 and 65.8. The pattern is not a clean overall win. On Humanity's Last Exam without tools, DeepSeek reports 42.6, below 44.5 for GPT-5.6 Sol and 56.3 for Opus 5. It also trails Opus 5 on Terminal-Bench, ProgramBench, NL2Repo-Bench, Chartography and BabyVision. Each benchmark measures a different task with its own scoring rules, so the rows should not be averaged into a general intelligence claim.[1]
The distinction that matters
Three benchmark rows show why the result is mixed
Higher is better within each row, but the tests are not interchangeable and the figures come from DeepSeek's own card.
DeepSWE
DeepSeek 77.0 · GPT-5.6 Sol 73.0 · Opus 5 74.0
DeepSeek leads this reported software-engineering result
Automation-Bench
DeepSeek 60.5 · GPT-5.6 Sol 45.8 · Opus 5 50.3
DeepSeek leads this reported agent-automation result
HLE, no tools
DeepSeek 42.6 · GPT-5.6 Sol 44.5 · Opus 5 56.3
DeepSeek trails both named rivals on this knowledge test
The missing denominator is repeated, independently controlled runs
The model card names the Claude Code harness for Terminal-Bench and Mini-SWE-V2 for DeepSWE, but it does not state how many repeated runs produced the displayed values, show uncertainty intervals, publish complete prompts and trajectories, or demonstrate identical inference budgets across every comparator. Benchmark scores can move with sampling settings, tool permissions, retry rules and harness versions. DeepSeek's September technical report describes the underlying 552-billion-parameter backbone, sparse mixture-of-experts architecture and training on a 45-trillion-token multimodal corpus, but it is a company preprint and predates the 1017 update. It cannot serve as independent validation of the new table.[1][3]
For developers, access is real but operation is not frictionless
The practical change is that organisations can inspect, modify and self-host the released weights under a permissive licence. That can widen experimentation and reduce dependence on a closed API. It does not make deployment free or simple: the repository lists a very large model, recommends a one-million-token context window and at least 256,000 output tokens for some uses, and currently shows no hosted inference provider. Teams still need specialist infrastructure, security controls and their own evaluation on representative tasks. Before procurement or workflow redesign, the useful next evidence would be independent reruns with fixed budgets, published traces, failure analysis and measurements of latency, energy, cost and human correction burden.[1]
What adopting the model would actually involve
Access to the weights is only one stage of making a working system. DeepSeek’s card supplies a Python reference implementation for prompt encoding, with tests for conversation history, tool calls, reasoning settings and images. It also links a maintained toolkit with Rust libraries and Python bindings. The card explicitly leaves model inference, tool execution and HTTP transport to the caller. An organisation therefore needs to assess the surrounding application as well as the model’s benchmark score.
Our interpretation is that a useful adoption test should separate those responsibilities. A model may propose an action while application code decides whether to execute it. Teams can compare representative tasks, the permissions granted, failed attempts and the human work needed to correct an answer. These are evaluation questions, rather than findings established by DeepSeek’s release. The published table provides a starting point for selecting tests; it does not show how a particular employer’s complete system will perform.[1]