ProgramBench
A benchmark for evaluating the ability of programming agents to build complete programs from scratch. It supports program generation task evaluation across system-level programming languages including C, C++, Go, and Rust.
Timeline (last 90 days)
ProgramBench is an evaluation benchmark for programming tasks that typically presents function-level or repository-level programming problems and measures results with metrics such as unit test pass rate.
The original report did not disclose Agensh's specific task pass rate on ProgramBench, its performance curves at different scales, or comparison data against centralized orchestration approaches.
Agensh's run on ProgramBench was constrained to a 6-hour time budget.
On ProgramBench, Agensh achieved scaling from 1 to 1,024 agents.
According to the official DeepSeek model card, the 1017 version's score on ProgramBench rose from 20.3 to 29.1.
On the four benchmarks ProgramBench, NL2Repo-Bench, Automation-Bench and Agents' Last Exam, V4.1 Flash 1017 scores 29.1, 68.2, 60.5 and 35.4 respectively, all higher than V4 Pro 0813's 15.5, 61.5, 43.2 and 25.7.
The new model improved on all 11 of the vendor's own benchmarks, with Terminal-Bench 4.0 up 11.1 points and ProgramBench up 8.8 points.
Among the 11 benchmarks published by DeepSeek, 1017 surpasses the previous generation on all of them: Tools rose from 36.8 to 42.6, DeepSWE from 74.2 to 77.0, ProgramBench from 20.3 to 29.1, and TerminalBench 4.0 from 31.2 to 42.3.
Wonder Full trails Opus 5.5 on ProgramBench (59.0 vs 70.5), Terminal-Bench 4.0, and HealthBench Pro.
Among the 14 benchmarks Meta itself published, Muse Wonder won 5, mainly in long-horizon coding, legal agent tasks, and Meta's internal memory tests; it lost 9, including ProgramBench (Opus leads 70.5 to 50.0).
4 more timeline events
All Facts (12)
ProgramBench is an evaluation benchmark for programming tasks that typically presents function-level or repository-level programming problems and measures results with metrics such as unit test pass rate.
50%UnverifiedThe original report did not disclose Agensh's specific task pass rate on ProgramBench, its performance curves at different scales, or comparison data against centralized orchestration approaches.
50%UnverifiedAgensh's run on ProgramBench was constrained to a 6-hour time budget.
50%UnverifiedOn ProgramBench, Agensh achieved scaling from 1 to 1,024 agents.
50%UnverifiedAccording to the official DeepSeek model card, the 1017 version's score on ProgramBench rose from 20.3 to 29.1.
50%UnverifiedOn the four benchmarks ProgramBench, NL2Repo-Bench, Automation-Bench and Agents' Last Exam, V4.1 Flash 1017 scores 29.1, 68.2, 60.5 and 35.4 respectively, all higher than V4 Pro 0813's 15.5, 61.5, 43.2 and 25.7.
50%UnverifiedThe new model improved on all 11 of the vendor's own benchmarks, with Terminal-Bench 4.0 up 11.1 points and ProgramBench up 8.8 points.
50%UnverifiedAmong the 11 benchmarks published by DeepSeek, 1017 surpasses the previous generation on all of them: Tools rose from 36.8 to 42.6, DeepSWE from 74.2 to 77.0, ProgramBench from 20.3 to 29.1, and TerminalBench 4.0 from 31.2 to 42.3.
50%UnverifiedThe effectiveness of ProgramGym has been validated on the ProgramBench benchmark.
50%UnverifiedProgram-Agent pushed the base model and the supervised fine-tuned model to 55.71% and 56.25%, respectively.
50%UnverifiedAfter fine-tuning using trajectories generated from building environments with Program-Env, the performance of the Qwen3.8-27B model improved from 34.87% to 42.84%.
50%UnverifiedDeepSeek的编程智能体DeepSWE在SWE-bench基准测试中从预览版的12.8分提升至62.7分
50%