SWE-Lancer
A benchmark for evaluating the programming capabilities of large language models. It includes two evaluation modes—task selection and per-task—and tests a model's ability to solve real-world software engineering tasks.
Timeline (last 90 days)
Testers evaluated model programming capabilities using the SWE-Lancer benchmark in both task selection and per-task modes.
The standard Kimi-K3-REAP-512GB version completely failed on SWE-Lancer tasks 14294, 15815_1, and 15925.
The reap576_iq2xxs (478GB) version successfully resolved SWE-Lancer tasks 14294, 15815_1, and 15925.
Streaming decoding speed is slow, averaging approximately 2.5 hours per SWE-Lancer task.
The estimated cost of running all SWE-Lancer tasks using rented cloud computing on RunPod is as high as $1,800.
All Facts (4)
Testers evaluated model programming capabilities using the SWE-Lancer benchmark in both task selection and per-task modes.
50%UnverifiedThe standard Kimi-K3-REAP-512GB version completely failed on SWE-Lancer tasks 14294, 15815_1, and 15925.
50%UnverifiedThe reap576_iq2xxs (478GB) version successfully resolved SWE-Lancer tasks 14294, 15815_1, and 15925.
50%UnverifiedStreaming decoding speed is slow, averaging approximately 2.5 hours per SWE-Lancer task.
50%