SWE-bench Lite
A streamlined version of SWE-bench containing 300 real GitHub issue tasks drawn from 12 Python repositories, used to evaluate the ability of language models to solve real software engineering problems.
Timeline (last 90 days)
The original SWE-bench contains 2,294 tasks covering 12 Python repositories; SWE-bench Lite has 300 curated tasks, and SWE-bench Verified has 500 human-verified tasks.
SWE-bench Lite is a trimmed subset of SWE-bench that retains about 300 tasks to reduce evaluation costs.
UC Berkeley research tested 7 frontier models on a variety of harnesses and found that for newer, smarter models, the choice of harness made no significant difference in SWE-bench Lite scores.
A UC Berkeley study ran seven frontier models on SWE-bench Lite across multiple harnesses such as Pi, Codex and Cloud Code, and found that on newer, smarter models the choice of harness had no significant effect on scores.
A UC Berkeley study tested multiple harnesses such as Pi, Codex and Claude Code with seven frontier models and found that on newer, smarter models the choice of harness had no significant effect on SWE-bench Lite scores.
All Facts (6)
SWE-bench Lite精选了300个更具代表性的子集
90%UnverifiedThe original SWE-bench contains 2,294 tasks covering 12 Python repositories; SWE-bench Lite has 300 curated tasks, and SWE-bench Verified has 500 human-verified tasks.
50%UnverifiedSWE-bench Lite is a trimmed subset of SWE-bench that retains about 300 tasks to reduce evaluation costs.
50%UnverifiedUC Berkeley research tested 7 frontier models on a variety of harnesses and found that for newer, smarter models, the choice of harness made no significant difference in SWE-bench Lite scores.
50%UnverifiedA UC Berkeley study ran seven frontier models on SWE-bench Lite across multiple harnesses such as Pi, Codex and Cloud Code, and found that on newer, smarter models the choice of harness had no significant effect on scores.
50%UnverifiedA UC Berkeley study tested multiple harnesses such as Pi, Codex and Claude Code with seven frontier models and found that on newer, smarter models the choice of harness had no significant effect on SWE-bench Lite scores.
50%