MMLU
Massive Multitask Language Understanding (MMLU) is a large-scale multitask benchmark designed to evaluate the knowledge and reasoning capabilities of large language models across a broad range of academic subjects and professional domains.
Core Facts
MMLU (Massive Multitask Language Understanding) covers multiple-choice questions across 57 subjects, testing the breadth and depth of a model's knowledge from humanities and social sciences to STEM.
MMLU evaluates a model's breadth of knowledge across 57 subject areas, HumanEval measures code generation accuracy, GSM8K tests mathematical reasoning, and HELM constructs a comprehensive multi-dimensional evaluation framework.
Benchmark testing is an important reference for verifying model capabilities, and common evaluation dimensions include mathematical reasoning, code generation, and scientific question answering.
Timeline (last 90 days)
Scores on early widely used benchmarks such as MMLU and GSM8K are already highly concentrated in the face of new-generation models.
The questions and answers of public benchmarks (such as MMLU and HumanEval) are usually publicly available, so model teams can specifically cover similar questions in their training data to obtain inflated scores on the leaderboards, that is, "benchmark gaming."
After installing the corpus, Llama-3.3-70B's harmful decisions out of 300 decisions dropped from 70 to 0, while its MMLU benchmark remained unchanged.
Traditional evaluations such as MMLU and GSM8K usually only check whether the text of a model's single output is correct, amounting to a static input-output mapping.
MMLU, GSM8K, HumanEval and C-Eval are common large-model benchmark suites, focusing respectively on multidisciplinary knowledge, elementary-school math reasoning, code generation and comprehensive Chinese ability.
A benchmark refers to a task set and scoring criteria (such as MMLU and GSM8K), while a harness is the software framework that executes these tasks, responsible for loading questions, constructing prompts, calling model APIs, parsing outputs, and judging correctness.
Standard downstream benchmarks include MMLU, HellaSwag, ARC and GSM8K, which respectively assess knowledge question answering, commonsense reasoning, scientific reasoning and mathematical ability.
Most Vals Index questions come from real work scenarios and require models to complete multi-step reasoning, data verification, and compliance judgments, unlike the general-purpose MMLU or GSM8K.
As frontier models approach saturation on MMLU, MMLU's discriminative power has dropped significantly.
The score gaps among different vendors' large language models on general benchmarks (such as MMLU and GSM8K) are gradually narrowing, and open-source models are rapidly catching up with closed-source models.
40 more timeline events
All Facts (20)
MMLU (Massive Multitask Language Understanding) covers multiple-choice questions across 57 subjects, testing the breadth and depth of a model's knowledge from humanities and social sciences to STEM.
90%VerifiedMMLU evaluates a model's breadth of knowledge across 57 subject areas, HumanEval measures code generation accuracy, GSM8K tests mathematical reasoning, and HELM constructs a comprehensive multi-dimensional evaluation framework.
90%VerifiedBenchmark testing is an important reference for verifying model capabilities, and common evaluation dimensions include mathematical reasoning, code generation, and scientific question answering.
85%VerifiedBenchmark contamination refers to the phenomenon where test questions may have already appeared in training data, causing high scores to not truly represent generalization ability.
85%VerifiedStandard AI benchmarks include MMLU (a comprehensive knowledge test covering 57 subjects), GPQA (graduate-level professional question answering), SWE-bench (real-world software engineering tasks), HumanEval and MBPP (programming ability), and MATH and GSM8K (mathematical reasoning)
85%VerifiedMMLU (Massive Multitask Language Understanding) contains multiple-choice questions across 57 subjects and is used to test the breadth of a model's knowledge.
80%VerifiedMMLU covers over 57,000 multiple-choice questions across 57 subject areas, while HumanEval focuses on function-level correctness verification for code generation.
80%VerifiedMMLU contains approximately 15,000 multiple-choice questions across 57 subject areas; HumanEval, designed by OpenAI, contains 164 Python programming problems; MATH contains 12,500 competition math problems; and GPQA covers graduate-level science questions.
80%VerifiedCredible capability evaluation of models in the AI field relies on standardized benchmark tests; commonly used evaluation sets include MMLU, HumanEval, MATH, and GPQA.
80%VerifiedOn mainstream evaluations such as MMLU, HumanEval, and GSM8K, the score gaps among leading large models are often only a few percentage points.
80%VerifiedMMLU was released by the University of California, Berkeley in 2020 and contains approximately 15,000 multiple-choice questions spanning 57 subjects.
80%VerifiedMainstream benchmarks such as MMLU, HumanEval, and GSM8K were originally designed primarily for academic comparison, focusing on measuring model accuracy on closed-form questions.
75%VerifiedBy mid-2024, top models had generally exceeded scores of 85–90% on MMLU, causing the benchmark's discriminative power to drop sharply.
75%VerifiedAlignment training optimizes the model's statistical behavior in known scenarios rather than establishing formally verifiable security boundaries, and may fail when faced with rare instruction combinations or role-playing frameworks outside the training distribution.
75%VerifiedHumanEval was designed by OpenAI and contains 164 Python programming problems; GSM8K contains 8,500 elementary school math word problems; MMLU covers multiple-choice questions across 57 subject areas.
75%VerifiedBenchmark Contamination is a significant challenge in the current field of large model evaluation, where publicly available test questions may already be included in model training data, causing evaluation results to be distorted.
75%VerifiedA benchmark is a fixed set of questions used to score and horizontally compare models, similar to a mock exam, and is used to rank models.
70%VerifiedDistinguishing memorization from reasoning in AI evaluations of AI coding tools is regarded as the core challenge in understanding the true capability boundaries of models, as data contamination causes a model's reasoning to essentially become a replay of memorized training data
70%VerifiedSome models carry the risk of over-optimization toward test sets, causing leaderboard scores to deviate from real-world performance (benchmark saturation).
70%VerifiedAI model benchmarking typically evaluates a model's reasoning, coding, and mathematical capabilities on standard datasets such as MMLU, HumanEval, and GSM8K.
70%