Unverified50% confidenceFactExact time
SWE-bench requires models to read issues from real GitHub repositories, autonomously generate patches, and pass validation on a hidden test set, and is considered one of the benchmarks most closely reflecting real-world engineering capability.
1
Sources
50%
Confidence
Long-term
Relevance
9/23/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedSWE-bench requires models to autonomously complete code modifications starting from real GitHub issues and pass test cases, making it closer to real-world engineering scenarios than HumanEval.80% similarVerifiedSWE-bench uses real GitHub Issues directly as test problems, requiring models to locate and fix bugs within a complete code repository.78% similarVerifiedSWE-bench requires models to resolve real software defects on GitHub and is the authoritative benchmark for evaluating agent-level programming capabilities.77% similarUnverifiedSWE-bench要求模型在真实GitHub仓库中定位问题并生成可通过CI测试的补丁，以Resolve Rate作为核心指标77% similarUnverifiedSWE-bench evaluates a model's ability to resolve real GitHub issues, while LiveCodeBench uses competitive programming problems to prevent data contamination.76% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/943487API
curl https://kongchang.com/api/v1/knowledge/claims/943487MCP
get_claim(id=943487)