MBPP
MBPP (Mostly Basic Python Problems) is a Python programming benchmark for evaluating code generation capabilities, released by Google Research. The benchmark contains hundreds of Python programming problems ranging from introductory to intermediate difficulty, each accompanied by a natural language description, a reference solution, and test cases. It is used to measure large language models' ability to interpret programming requirements and generate correct Python code, and is a widely used standard benchmark in the code generation domain.
Core Facts
MBPP (Mostly Basic Python Problems) was released by Google and contains approximately 1,000 entry-level programming problems.
Mainstream benchmarks for evaluating programming large models include HumanEval, MBPP, SWE-bench, and LiveCodeBench
Code large language models are evaluated using executable benchmarks such as HumanEval and MBPP, meaning the generated code is directly executed and the output is checked for correctness.
Timeline (last 90 days)
On MBPP (Evalplus version), the score of the model with the extended vocabulary dropped slightly from 65.3% to 64.6%.
Traditional unit test benchmarks such as HumanEval and MBPP differ from robustness evaluations, which require constructing adversarial inputs (messy naming, mixed responsibilities, comments that do not match the implementation, etc.).
The authors evaluate on two code generation benchmarks, HumanEval and MBPP, and the results show that both the practical decoding algorithms based on these decision rules and CTD outperform standard temperature sampling and adaptive decoding baselines.
MBPP (Mostly Basic Programming Problems) is similar in approach to HumanEval, with problems closer to entry-level everyday programming, at a scale of about one thousand problems.
Parsimony's evaluation dimensions form a complementary rather than substitutive relationship with existing benchmarks such as HumanEval, MBPP and SWE-bench.
After decontamination, the model achieved 22.6% on HumanEval, 21.3% on HumanEval+, and 35.0% on MBPP; compared with the published version, HumanEval dropped by 4.3 points, McNemar test p = 0.039.
On MBPP, only 10 problems matched removed data, while the two original-data runs differed by 11 problems; the published best of 38.9% is best understood as a lucky random seed, and the run-to-run variance on MBPP is about 4 points.
FrontierCode is closer to the complexity of production environments compared to single-function completion benchmarks such as HumanEval or MBPP.
CursorBench is a vertical-scenario evaluation benchmark, distinct from general-purpose code generation benchmarks such as HumanEval and MBPP.
Vertical benchmarks include HumanEval/MBPP (code), MATH/GSM8K (mathematical reasoning), and MMLU (multidisciplinary knowledge).
11 more timeline events
All Facts (20)
MBPP (Mostly Basic Python Problems) was released by Google and contains approximately 1,000 entry-level programming problems.
80%VerifiedMainstream benchmarks for evaluating programming large models include HumanEval, MBPP, SWE-bench, and LiveCodeBench
80%VerifiedCode large language models are evaluated using executable benchmarks such as HumanEval and MBPP, meaning the generated code is directly executed and the output is checked for correctness.
75%VerifiedThe most recognized evaluation benchmarks in the AI programming field include HumanEval (testing the generation of correct implementations from function docstrings) and SWE-bench (locating and fixing actual issues in real GitHub repositories).
65%VerifiedThe training data sources for specialized programming models include open-source code from GitHub, competitive programming solutions, technical documentation, and code review records, and the models are optimized for benchmarks such as HumanEval, MBPP, and SWE-bench.
65%UnverifiedHumanEval等传统代码评测基准通常只要求模型根据函数签名和文档字符串生成独立函数，本质上是受限的代码补全任务
90%UnverifiedCodex模型的评估通常使用HumanEval和MBPP等编码基准测试
85%UnverifiedHumanEval and MBPP primarily measure a model's code generation capability, while code review is an essentially different task, and the industry currently lacks standardized code review benchmarks.
60%UnverifiedOn MBPP (Evalplus version), the score of the model with the extended vocabulary dropped slightly from 65.3% to 64.6%.
50%UnverifiedTraditional unit test benchmarks such as HumanEval and MBPP differ from robustness evaluations, which require constructing adversarial inputs (messy naming, mixed responsibilities, comments that do not match the implementation, etc.).
50%UnverifiedThe authors evaluate on two code generation benchmarks, HumanEval and MBPP, and the results show that both the practical decoding algorithms based on these decision rules and CTD outperform standard temperature sampling and adaptive decoding baselines.
50%UnverifiedParsimony's evaluation dimensions form a complementary rather than substitutive relationship with existing benchmarks such as HumanEval, MBPP and SWE-bench.
50%UnverifiedMBPP (Mostly Basic Programming Problems) is similar in approach to HumanEval, with problems closer to entry-level everyday programming, at a scale of about one thousand problems.
50%UnverifiedOn MBPP, only 10 problems matched removed data, while the two original-data runs differed by 11 problems; the published best of 38.9% is best understood as a lucky random seed, and the run-to-run variance on MBPP is about 4 points.
50%UnverifiedAfter decontamination, the model achieved 22.6% on HumanEval, 21.3% on HumanEval+, and 35.0% on MBPP; compared with the published version, HumanEval dropped by 4.3 points, McNemar test p = 0.039.
50%UnverifiedFrontierCode is closer to the complexity of production environments compared to single-function completion benchmarks such as HumanEval or MBPP.
50%UnverifiedCursorBench is a vertical-scenario evaluation benchmark, distinct from general-purpose code generation benchmarks such as HumanEval and MBPP.
50%UnverifiedVertical benchmarks include HumanEval/MBPP (code), MATH/GSM8K (mathematical reasoning), and MMLU (multidisciplinary knowledge).
50%UnverifiedThe pass@k metric is widely adopted on standard benchmarks such as HumanEval and MBPP.
50%UnverifiedHumanEval and MBPP primarily evaluate single-function-level code generation and have relatively limited discriminating power for agent-type tools.
50%