HumanEval
HumanEval is a code generation benchmark released by OpenAI, used to evaluate a model's ability to produce functionally correct code from function signatures and docstrings.
Core Facts
MMLU evaluates a model's breadth of knowledge across 57 subject areas, HumanEval measures code generation accuracy, GSM8K tests mathematical reasoning, and HELM constructs a comprehensive multi-dimensional evaluation framework.
HumanEval was released by OpenAI and contains 164 hand-crafted Python programming problems, with pass@k as its core metric.
HumanEval was proposed by OpenAI in 2021 and evaluates function-level code completion.
Timeline (last 90 days)
The questions and answers of public benchmarks (such as MMLU and HumanEval) are usually publicly available, so model teams can specifically cover similar questions in their training data to obtain inflated scores on the leaderboards, that is, "benchmark gaming."
Compared with general code-generation benchmarks (such as HumanEval and SWE-bench), CursorBench is characterized by being close to actual editor workflows and incorporating cost as a first-class metric in its reports.
Unlike benchmarks that only examine functional correctness (such as HumanEval), EffiBench requires models to approach the resource consumption level of the reference solution as closely as possible while passing the test cases.
MMLU, GSM8K, HumanEval and C-Eval are common large-model benchmark suites, focusing respectively on multidisciplinary knowledge, elementary-school math reasoning, code generation and comprehensive Chinese ability.
The model with the extended vocabulary improved its pass rate on HumanEval from 45.7% to 52.4%.
Scores on code generation benchmarks (such as HumanEval) are already approaching saturation.
Traditional unit test benchmarks such as HumanEval and MBPP differ from robustness evaluations, which require constructing adversarial inputs (messy naming, mixed responsibilities, comments that do not match the implementation, etc.).
Benchmarks are standardized evaluation sets and metrics commonly used in machine learning, such as MMLU (multidisciplinary knowledge), HumanEval (code generation), GSM8K (mathematical reasoning), etc.
On HumanEval, Keera's pretrained model achieved pass@1 of 3/164 (1.8%), and after fine-tuning 9/164 (5.5%).
The author explicitly notes limitations in the HumanEval results: data contamination was not checked, the pretraining and fine-tuning evaluation settings differ (not a controlled measurement), and the saved fine-tuning per-question results file contains NUL bytes and is incomplete.
40 more timeline events
All Facts (20)
MMLU evaluates a model's breadth of knowledge across 57 subject areas, HumanEval measures code generation accuracy, GSM8K tests mathematical reasoning, and HELM constructs a comprehensive multi-dimensional evaluation framework.
90%VerifiedHumanEval was released by OpenAI and contains 164 hand-crafted Python programming problems, with pass@k as its core metric.
90%VerifiedHumanEval was proposed by OpenAI in 2021 and evaluates function-level code completion.
90%VerifiedHumanEval was proposed by OpenAI and contains 164 Python programming problems, primarily testing function-level code generation capability
90%VerifiedBenchmark testing is an important reference for verifying model capabilities, and common evaluation dimensions include mathematical reasoning, code generation, and scientific question answering.
85%VerifiedThe DeepSWE series of benchmarks is used to measure a model's autonomous end-to-end engineering capabilities in real-world software engineering scenarios.
85%VerifiedBenchmark contamination refers to the phenomenon where test questions may have already appeared in training data, causing high scores to not truly represent generalization ability.
85%VerifiedHumanEval was developed in-house by OpenAI and is specifically designed to test the functional correctness rate of code generation.
85%VerifiedStandard AI benchmarks include MMLU (a comprehensive knowledge test covering 57 subjects), GPQA (graduate-level professional question answering), SWE-bench (real-world software engineering tasks), HumanEval and MBPP (programming ability), and MATH and GSM8K (mathematical reasoning)
85%VerifiedMMLU contains approximately 15,000 multiple-choice questions across 57 subject areas; HumanEval, designed by OpenAI, contains 164 Python programming problems; MATH contains 12,500 competition math problems; and GPQA covers graduate-level science questions.
80%VerifiedCredible capability evaluation of models in the AI field relies on standardized benchmark tests; commonly used evaluation sets include MMLU, HumanEval, MATH, and GPQA.
80%VerifiedPass rates of code generation models on benchmarks such as HumanEval jumped from 60% to over 90% between 2024 and 2025
80%VerifiedOn mainstream evaluations such as MMLU, HumanEval, and GSM8K, the score gaps among leading large models are often only a few percentage points.
80%VerifiedHumanEval是由OpenAI于2021年发布的代码生成基准测试，包含164个手写的Python编程问题
80%Verified大语言模型通过在数千亿乃至数万亿个词元（Token）的文本数据上进行自监督预训练
80%VerifiedMainstream benchmarks for evaluating programming large models include HumanEval, MBPP, SWE-bench, and LiveCodeBench
80%VerifiedClaude 3.5 Sonnet is a model released by Anthropic in 2024 that surpassed the contemporaneous GPT-4o on the code generation benchmark HumanEval
80%VerifiedMainstream benchmarks such as MMLU, HumanEval, and GSM8K were originally designed primarily for academic comparison, focusing on measuring model accuracy on closed-form questions.
75%VerifiedBy mid-2024, top models had generally exceeded scores of 85–90% on MMLU, causing the benchmark's discriminative power to drop sharply.
75%VerifiedGPT-4 and Claude 3 achieve pass@1 rates exceeding 85% on the HumanEval code benchmark.
75%