Expired50% confidenceBenchmarkExact time
In the initial publicly released SWE-bench results, GPT-4's resolution rate was only approximately 1.7%, Devin claimed to reach 13.86%, and SWE-agent reached 12.5%.
1
Sources
50%
Confidence
Medium-term (~90 days)
Relevance
7/22/2026
First Seen
This claim has expired. It is kept for the record and no longer describes the current state.
Valid until: 10/20/2026(expired)
Sources
Related Claims
UnverifiedEarly GPT-4 had a pass rate of only 1.7% on SWE-bench, Claude 3 Opus approximately 4.8%, and it was not until mid-2024 that the 20% mark was surpassed.76% similarUnverifiedIn SWE-bench testing, the early GPT-4 resolution rate was below 2%, while Claude 3.5 Sonnet broke through 49%.75% similarUnverifiedThe median output token count per problem-solving attempt for Opus 4.7 is 60,000, while GPT 5.5 requires only 16,000.70% similarUnverified在GPT-4发布初期，主流模型在GPQA上的准确率约为35-40%，人类领域专家基准线约为65%，随机猜测为25%68% similarUnverifiedOn the HealthBench Professional benchmark, Mythos 5 scored 66.0% and GPT-5.6 Sol scored 60.5% (approximately 64% after length correction).68% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/589619API
curl https://kongchang.com/api/v1/knowledge/claims/589619MCP
get_claim(id=589619)