Expired60% confidenceBenchmarkExact time
In SWE-bench testing, the early GPT-4 resolution rate was below 2%, while Claude 3.5 Sonnet broke through 49%.
2
Sources
60%
Confidence
Medium-term (~90 days)
Relevance
7/16/2026
First Seen
This claim has expired. It is kept for the record and no longer describes the current state.
Valid until: 10/14/2026(expired)
Sources
Related Claims
UnverifiedEarly GPT-4 had a pass rate of only 1.7% on SWE-bench, Claude 3 Opus approximately 4.8%, and it was not until mid-2024 that the 20% mark was surpassed.77% similarUnverifiedTop paid models such as Claude 3.5 Sonnet and GPT-4o generally achieve resolution rates on SWE-bench that are 10 to 30 percentage points higher than free or lightweight models.75% similarUnverifiedIn the initial publicly released SWE-bench results, GPT-4's resolution rate was only approximately 1.7%, Devin claimed to reach 13.86%, and SWE-agent reached 12.5%.75% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/530699API
curl https://kongchang.com/api/v1/knowledge/claims/530699MCP
get_claim(id=530699)