GSM8K
GSM8K is a benchmark comprising approximately 8,500 grade-school-level math word problems, used to evaluate the basic mathematical reasoning capabilities of large language models.
Core Facts
MMLU evaluates a model's breadth of knowledge across 57 subject areas, HumanEval measures code generation accuracy, GSM8K tests mathematical reasoning, and HELM constructs a comprehensive multi-dimensional evaluation framework.
Benchmark testing is an important reference for verifying model capabilities, and common evaluation dimensions include mathematical reasoning, code generation, and scientific question answering.
Benchmark contamination refers to the phenomenon where test questions may have already appeared in training data, causing high scores to not truly represent generalization ability.
Timeline (last 90 days)
Scores on early widely used benchmarks such as MMLU and GSM8K are already highly concentrated in the face of new-generation models.
Two and a half years ago, when AI labs released a new generation of large language models, almost all of them put GSM8K scores in the most prominent position; today, this metric no longer appears in the core narrative of mainstream model releases.
When mainstream models' accuracy on GSM8K approaches 95% or above, the benchmark loses its discriminative power.
The design features of GSM8K are unique answers and automatic scoring, with the aim of testing a model's multi-step chain reasoning rather than mere memorization.
On GSM8K, dSpark achieves a 10.12x speedup on LLaDA-8B-Instruct relative to multiple baseline methods.
While achieving speedup on GSM8K, dSpark maintains or even improves task performance.
The gain from the sampler can be decomposed into a memory effect (present on both Sudoku and GSM8K) and a token-selection effect (present on GSM8K and most pronounced at small budgets).
On reasoning tasks (such as GSM8K), categorical refinement improves performance, whereas in unconditional text generation, updates that preserve randomness are more conducive to maintaining diversity.
Traditional evaluations such as MMLU and GSM8K usually only check whether the text of a model's single output is correct, amounting to a static input-output mapping.
MMLU, GSM8K, HumanEval and C-Eval are common large-model benchmark suites, focusing respectively on multidisciplinary knowledge, elementary-school math reasoning, code generation and comprehensive Chinese ability.
40 more timeline events
All Facts (20)
MMLU evaluates a model's breadth of knowledge across 57 subject areas, HumanEval measures code generation accuracy, GSM8K tests mathematical reasoning, and HELM constructs a comprehensive multi-dimensional evaluation framework.
90%VerifiedBenchmark testing is an important reference for verifying model capabilities, and common evaluation dimensions include mathematical reasoning, code generation, and scientific question answering.
85%VerifiedBenchmark contamination refers to the phenomenon where test questions may have already appeared in training data, causing high scores to not truly represent generalization ability.
85%VerifiedStandard AI benchmarks include MMLU (a comprehensive knowledge test covering 57 subjects), GPQA (graduate-level professional question answering), SWE-bench (real-world software engineering tasks), HumanEval and MBPP (programming ability), and MATH and GSM8K (mathematical reasoning)
85%VerifiedMMLU contains approximately 15,000 multiple-choice questions across 57 subject areas; HumanEval, designed by OpenAI, contains 164 Python programming problems; MATH contains 12,500 competition math problems; and GPQA covers graduate-level science questions.
80%VerifiedGSM8K is an elementary school math word problem benchmark released by OpenAI in 2021, containing approximately 8,500 problems that require 2 to 8 reasoning steps
80%VerifiedOn mainstream evaluations such as MMLU, HumanEval, and GSM8K, the score gaps among leading large models are often only a few percentage points.
80%VerifiedMainstream benchmarks such as MMLU, HumanEval, and GSM8K were originally designed primarily for academic comparison, focusing on measuring model accuracy on closed-form questions.
75%VerifiedGSM8K, released by OpenAI, contains approximately 8,500 elementary school math word problems designed to test multi-step reasoning ability
75%VerifiedHumanEval was designed by OpenAI and contains 164 Python programming problems; GSM8K contains 8,500 elementary school math word problems; MMLU covers multiple-choice questions across 57 subject areas.
75%VerifiedAI model benchmarking typically evaluates a model's reasoning, coding, and mathematical capabilities on standard datasets such as MMLU, HumanEval, and GSM8K.
70%VerifiedWhen test questions may have appeared in training data, high scores do not truly represent generalization ability — this is known as benchmark contamination.
65%VerifiedCommonly used AI benchmarks in the industry include MMLU (Massive Multitask Language Understanding), HumanEval (code generation), and GSM8K (mathematical reasoning), but each benchmark can only measure capabilities along specific dimensions.
65%VerifiedMMLU, HumanEval, and MT-Bench are standardized benchmark test sets widely recognized by the AI community.
65%UnverifiedOn mainstream benchmarks such as MMLU, HumanEval, and GSM8K, open-source models have reached 85–95% of the level of early versions of GPT-4.
70%UnverifiedBenchmarks for evaluating large models such as MMLU, HumanEval, and GSM8K have limitations including data contamination, overfitting, and insufficient ecological validity.
70%UnverifiedThe Self-Consistency method, proposed by Xuezhi Wang and others, improved accuracy by more than 10% on mathematical reasoning benchmarks such as GSM8K.
70%Partially VerifiedGSM8K是小学数学推理基准，HumanEval是代码生成基准，HellaSwag是常识推理基准
65%UnverifiedMMLU covers 57 subjects and nearly 16,000 multiple-choice questions; HumanEval was developed by OpenAI and contains 164 programming problems; GSM8K contains 8,500 elementary school math word problems.
60%UnverifiedHumanEval consists of 164 Python programming problems, and GSM8K contains 8,500 elementary school math word problems.
60%