Verified80% confidenceFactTime unknown
Pass rates of code generation models on benchmarks such as HumanEval jumped from 60% to over 90% between 2024 and 2025
6
Sources
80%
Confidence
Long-term
Relevance
6/1/2026
First Seen
Sources
Vibe Coding与Agentic Engineering边界模糊：Simon Willison的深度反思
rss
Related Entities
Related Claims
UnverifiedBenchmark overfitting is a core problem in current large model evaluation; some models achieve pass rates above 90% on HumanEval but perform far below expectations in real codebases.77% similarUnverified斯坦福HAI研究所2023年报告显示，最新LLM在HumanEval等代码基准测试中正确率已超过85%，远超2021年的不足50%74% similarUnverified自2023年GPT-4发布以来，AI在HumanEval等主流编程基准测试上的通过率从不足50%跃升至90%以上74% similarUnverified早期的Codex模型在HumanEval上的通过率约为28.8%，而到2024-2025年GPT-4级别的模型已达到90%以上的通过率67% similarUnverified2024 年的多项基准测试表明，经过严格安全对齐的模型在代码生成任务上的拒绝率比基座模型高出 15-30%，在创意写作任务上的自我审查率高达 40%66% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/32594API
curl https://kongchang.com/api/v1/knowledge/claims/32594MCP
get_claim(id=32594)