LiveCodeBench
LiveCodeBench is a comprehensive coding evaluation benchmark for large language models that continuously collects problems from major competitive programming platforms such as LeetCode, AtCoder, and Codeforces to prevent models from overfitting to static test sets. Beyond code generation, the benchmark covers multiple programming task dimensions including code repair and test case generation, providing a dynamic and thorough assessment of model programming capabilities.
Core Facts
Timeline (last 90 days)
On the LiveCodeBench benchmark, Gemma 4 E2B scores 44, while Gemma 3 27B scores 29.
LiveCodeBench's problems come from new problems released within specific time windows on competitive programming platforms such as LeetCode and AtCoder, to reduce the risk of benchmark contamination.
Agent capability is usually measured through dimensions such as tool-calling accuracy, multi-step task completion rate, and instruction-following degree, with typical benchmarks including BFCL, τ-bench, and GAIA; Coding capability is measured through code generation and repair benchmarks (such as SWE-bench and LiveCodeBench).
The IQ3S tier scores 92.93 on GPQA Diamond (vs. BF16 baseline 91.92), 86.86 on LiveCodeBench V6 (vs. 87.43), with a three-task average of 93.26 vs. 93.12, and a file size less than one quarter of BF16 (approximately 22%).
Code capability benchmarks widely recognized in academia and industry include HumanEval, SWE-bench, and LiveCodeBench, but these benchmarks do not yet cover comprehensive tasks such as 3D game generation.
Version 2.0 shows declines in knowledge-based capabilities (MMLU-Pro 72.5 vs. base 78.5) and competition-level coding ability (LiveCodeBench 51 vs. 56).
SWE-bench and LiveCodeBench are relatively well-recognized industry benchmarks for evaluating code large language models
LiveCodeBench adopts a temporal cutoff strategy, using only competition problems released after a model's training cutoff date to prevent data leakage.
New-generation benchmarks such as SWE-bench and LiveCodeBench attempt to evaluate model capabilities in real-world scenarios, but still cannot fully cover engineering complexity.
LiveCodeBench continuously scrapes new problems from competitive programming platforms such as LeetCode, Codeforces, and AtCoder to prevent data contamination.
6 more timeline events
All Facts (17)
Mainstream benchmarks for evaluating programming large models include HumanEval, MBPP, SWE-bench, and LiveCodeBench
80%VerifiedQwen3 27B scored 90.3 on LiveCodeBench V6, slightly surpassing Opus 4.6 Max's score of 88.8.
65%UnverifiedLiveCodeBench continuously crawls new problems from competitive programming platforms such as LeetCode, Codeforces, and AtCoder to prevent models from gaming the benchmark, and its evaluation covers four dimensions: code generation, self-repair, code execution reasoning, and test output prediction.
60%UnverifiedLiveCodeBench uses continuously updated new problems from competitive programming platforms to avoid training data leakage.
60%UnverifiedSWE-bench evaluates a model's ability to resolve real GitHub issues, while LiveCodeBench uses competitive programming problems to prevent data contamination.
60%UnverifiedDynamic benchmarks such as LiveCodeBench and EvoEval reduce data contamination by continuously updating their problem sets or generating variant problems in real time.
60%UnverifiedOn the LiveCodeBench benchmark, Gemma 4 E2B scores 44, while Gemma 3 27B scores 29.
50%UnverifiedLiveCodeBench's problems come from new problems released within specific time windows on competitive programming platforms such as LeetCode and AtCoder, to reduce the risk of benchmark contamination.
50%UnverifiedAgent capability is usually measured through dimensions such as tool-calling accuracy, multi-step task completion rate, and instruction-following degree, with typical benchmarks including BFCL, τ-bench, and GAIA; Coding capability is measured through code generation and repair benchmarks (such as SWE-bench and LiveCodeBench).
50%UnverifiedThe IQ3S tier scores 92.93 on GPQA Diamond (vs. BF16 baseline 91.92), 86.86 on LiveCodeBench V6 (vs. 87.43), with a three-task average of 93.26 vs. 93.12, and a file size less than one quarter of BF16 (approximately 22%).
50%UnverifiedCode capability benchmarks widely recognized in academia and industry include HumanEval, SWE-bench, and LiveCodeBench, but these benchmarks do not yet cover comprehensive tasks such as 3D game generation.
50%UnverifiedVersion 2.0 shows declines in knowledge-based capabilities (MMLU-Pro 72.5 vs. base 78.5) and competition-level coding ability (LiveCodeBench 51 vs. 56).
50%UnverifiedSWE-bench and LiveCodeBench are relatively well-recognized industry benchmarks for evaluating code large language models
50%UnverifiedLiveCodeBench adopts a temporal cutoff strategy, using only competition problems released after a model's training cutoff date to prevent data leakage.
50%UnverifiedNew-generation benchmarks such as SWE-bench and LiveCodeBench attempt to evaluate model capabilities in real-world scenarios, but still cannot fully cover engineering complexity.
50%UnverifiedLiveCodeBench continuously scrapes new problems from competitive programming platforms such as LeetCode, Codeforces, and AtCoder to prevent data contamination.
50%UnverifiedQwen3 27B's performance on coding and agent benchmarks is: SWE-bench Pro 61.7, Terminal-Bench 2.1 73.0, and LiveCodeBench V6 90.3.
50%