DeepSWE
A benchmark designed specifically for evaluating large models on software engineering tasks. Core scenarios include automatic bug fixing in GitHub repositories, code generation, and multi-file refactoring. It corresponds to the SWE-bench task set and is more reflective of real-world development than general-purpose benchmarks.
Core Facts
The DeepSWE series of benchmarks is used to measure a model's autonomous end-to-end engineering capabilities in real-world software engineering scenarios.
DeepSWE is a code capability evaluation benchmark for software engineering scenarios, with tasks sourced from real GitHub Issue fix scenarios.
GPT-6.1 Sol performs best on the DeepSWE benchmark at the high reasoning effort level
Timeline (last 90 days)
The model card gives GLM-5.3-Flash's benchmark scores: 84.3 on Terminal-Bench 2.1 and 63.4 on DeepSWE.
Opus 5.5 scores 66.4% on TerminalBench, about 24 points higher than DeepSeek 1017; GPT 6.1 Sol leads 1017 slightly on DeepSWE at 75.2%.
Among the 11 benchmarks published by DeepSeek, 1017 surpasses the previous generation on all of them: Tools rose from 36.8 to 42.6, DeepSWE from 74.2 to 77.0, ProgramBench from 20.3 to 29.1, and TerminalBench 4.0 from 31.2 to 42.3.
DeepSeek-V4.1-Flash-1017 achieves 77.0% accuracy on the DeepSWE coding benchmark at an average cost of slightly under $0.20 per task (highest setting), with about 72% accuracy at about $0.10 in the low setting.
According to DeepSeek's chart, competing models cost between $2 and over $10 per task on the same DeepSWE benchmark.
The previous-generation model achieved 74.2% accuracy on DeepSWE at about $0.50 per task.
V4.1-Flash-1017 scores 77.0 on the DeepSWE benchmark.
There is a gap between benchmark scores and actual development experience, stemming from differences in task distribution (real requirements are vaguer), toolchain dependencies (building and testing in real projects is noisy), and output length constraints (short outputs may lose points in collaborative scenarios).
DeepSWE is a derivative or同类 benchmark of the software-engineering evaluation family like SWE-bench, and its score is usually the proportion of issues resolved.
DeepSeek v4.1 Flash 1017 has been released and has sparked widespread attention in technical communities, but official details disclosed are limited.
40 more timeline events
All Facts (20)
The DeepSWE series of benchmarks is used to measure a model's autonomous end-to-end engineering capabilities in real-world software engineering scenarios.
85%VerifiedDeepSWE is a code capability evaluation benchmark for software engineering scenarios, with tasks sourced from real GitHub Issue fix scenarios.
75%VerifiedGPT-6.1 Sol performs best on the DeepSWE benchmark at the high reasoning effort level
70%VerifiedV4.1 Flash surpassed Opus 5 on the DeepSWE benchmark by a narrow margin of 0.2 points.
70%VerifiedV4.1-Flash-1017 scores 77.0 on the DeepSWE benchmark.
65%VerifiedArgon achieved a score of 77.9% on the DeepSWE benchmark, setting a new record
65%VerifiedDeepSeek v4.1 Flash 1017 has been released and has sparked widespread attention in technical communities, but official details disclosed are limited.
80%Partially VerifiedDeepSeek-V4.1-Flash-1017 achieves 77.0% accuracy on the DeepSWE coding benchmark at an average cost of slightly under $0.20 per task (highest setting), with about 72% accuracy at about $0.10 in the low setting.
65%UnverifiedDeepSeek officially announced that V4 Pro scored as high as 87.9 on the Terminal Bench benchmark.
60%UnverifiedDeepSeek's model V4 Flash 0731 achieved a score of 82.7% on Terminal-Bench 2.1.
60%UnverifiedThe model card gives GLM-5.3-Flash's benchmark scores: 84.3 on Terminal-Bench 2.1 and 63.4 on DeepSWE.
50%UnverifiedAmong the 11 benchmarks published by DeepSeek, 1017 surpasses the previous generation on all of them: Tools rose from 36.8 to 42.6, DeepSWE from 74.2 to 77.0, ProgramBench from 20.3 to 29.1, and TerminalBench 4.0 from 31.2 to 42.3.
50%UnverifiedOpus 5.5 scores 66.4% on TerminalBench, about 24 points higher than DeepSeek 1017; GPT 6.1 Sol leads 1017 slightly on DeepSWE at 75.2%.
50%UnverifiedThe previous-generation model achieved 74.2% accuracy on DeepSWE at about $0.50 per task.
50%UnverifiedAccording to DeepSeek's chart, competing models cost between $2 and over $10 per task on the same DeepSWE benchmark.
50%UnverifiedDeepSWE is a derivative or同类 benchmark of the software-engineering evaluation family like SWE-bench, and its score is usually the proportion of issues resolved.
50%UnverifiedThere is a gap between benchmark scores and actual development experience, stemming from differences in task distribution (real requirements are vaguer), toolchain dependencies (building and testing in real projects is noisy), and output length constraints (short outputs may lose points in collaborative scenarios).
50%UnverifiedComparing BF16 and NVFP4 under the same harness, WebVoyager dropped from 77.72 to 66.31, DeepSWE from 24.65 to 18.2, and MMLU-ProX from 76.91 to 73.76, with other rows within the noise range.
50%UnverifiedDeepSWE focuses on complex, deeply dependent engineering problems; SWE-Marathon emphasizes long-chain, multi-stage tasks, simulating scenarios of sustained work over long periods.
50%UnverifiedThe SWE-Race evaluation protocol follows DeepSWE's 100-step setting, limiting the maximum number of action steps for the agent.
50%