Unverified50% confidenceBenchmarkExact time
Research targeting HumanEval indicates that when a model has seen different code implementations of a problem, its pass@1 metric may be overestimated by 10–15%.
1
Sources
50%
Confidence
Long-term
Relevance
8/16/2026
First Seen
Sources
Terminal Bench 3发布：未被污染的AI终端能力基准测试
Related Claims
UnverifiedStandardized code benchmark tests such as HumanEval show that top models have already exceeded a pass@1 score of 90%.72% similarUnverifiedBenchmark overfitting is a core problem in current large model evaluation; some models achieve pass rates above 90% on HumanEval but perform far below expectations in real codebases.72% similarUnverified研究表明，即使能力相对较弱的模型，其pass@10或pass@20的表现往往能够接近强模型的pass@1水平67% similarUnverifiedIn a 2024 research paper, Anthropic quantified the phenomenon of model over-refusal, finding that safety-trained models had false positive refusal rates as high as 15–30% in certain scenarios65% similarUnverifiedWhen benchmark scoring metrics become optimization targets, a Goodhart's Law effect arises: models learn to score high on specific formats without necessarily truly improving their capabilities, and some models with HumanEval scores exceeding 95% vary dramatically in actual code generation ability.65% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/757003API
curl https://kongchang.com/api/v1/knowledge/claims/757003MCP
get_claim(id=757003)