MATH
MATH is a benchmark dataset for evaluating mathematical reasoning in machine learning models, constructed by Hendrycks et al. It contains thousands of problems drawn from high school mathematics competitions, spanning algebra, geometry, number theory, probability, and other domains, with problems stratified by difficulty level. Each problem requires the model to produce a complete solution process along with a final answer, making it a standard benchmark for assessing complex mathematical reasoning in large language models.
Core Facts
Benchmark testing is an important reference for verifying model capabilities, and common evaluation dimensions include mathematical reasoning, code generation, and scientific question answering.
Standard AI benchmarks include MMLU (a comprehensive knowledge test covering 57 subjects), GPQA (graduate-level professional question answering), SWE-bench (real-world software engineering tasks), HumanEval and MBPP (programming ability), and MATH and GSM8K (mathematical reasoning)
HumanEval focuses on code generation capability, testing models' logical reasoning and code correctness with 164 programming problems.
Timeline (last 90 days)
After GSM8K saturated, the community turned to MATH (competition-level mathematics), GPQA (graduate-level science Q&A), and agent-type evaluations that require executing code or calling external tools.
JOLT's absolute accuracy on MATH, the model scale at which it matches GRPO, whether the 6.5x token reduction comes at the cost of more training steps or a more complex teacher training phase, and whether there is a risk of capability degradation or mode collapse under shared parameters are all open questions still to be answered.
The currently public JOLT efficiency comparison is limited to the single MATH benchmark, so its performance on code generation, multi-turn dialogue, or open-domain reasoning cannot yet be inferred.
JOLT's currently public information covers only the MATH benchmark, and its generalization ability still needs more verification.
On the MATH mathematical reasoning benchmark, JOLT achieved the same accuracy as GRPO, but processed only about one-sixth as many training tokens as GRPO.
Common reasoning benchmarks include GSM8K (elementary school math word problems), MATH (competition-level math problems), StrategyQA (multi-hop commonsense reasoning), and others; these benchmarks require multi-step derivation and have automatically verifiable answers.
On a real LLM (DeepSeek), the researchers evaluated 50 samples each on the GSM8K, MATH, and BBH benchmarks, and GraphEvoResearcher achieved accuracies of 98%, 92%, and 82% respectively, the highest among all baselines.
GSM8K, MATH, and BBH are three commonly used reasoning benchmarks with a clear difficulty gradient: GSM8K consists of elementary school math word problems, MATH covers competition-level math problems, and BBH contains 23 task types requiring multi-step symbolic manipulation or strategic reasoning.
On the MATH dataset, GraphEvoResearcher improves by 24 percentage points over single-pass generation and by 12 percentage points over flat reflection.
Benchmark testing is an important reference for verifying model capabilities, and common evaluation dimensions include mathematical reasoning, code generation, and scientific question answering.
23 more timeline events
All Facts (20)
Benchmark testing is an important reference for verifying model capabilities, and common evaluation dimensions include mathematical reasoning, code generation, and scientific question answering.
85%VerifiedStandard AI benchmarks include MMLU (a comprehensive knowledge test covering 57 subjects), GPQA (graduate-level professional question answering), SWE-bench (real-world software engineering tasks), HumanEval and MBPP (programming ability), and MATH and GSM8K (mathematical reasoning)
85%VerifiedHumanEval focuses on code generation capability, testing models' logical reasoning and code correctness with 164 programming problems.
85%VerifiedSome models carry the risk of over-optimization toward test sets, causing leaderboard scores to deviate from real-world performance (benchmark saturation).
70%VerifiedAI model benchmarking typically evaluates a model's reasoning, coding, and mathematical capabilities on standard datasets such as MMLU, HumanEval, and GSM8K.
70%VerifiedCommonly used AI benchmarks in the industry include MMLU (Massive Multitask Language Understanding), HumanEval (code generation), and GSM8K (mathematical reasoning), but each benchmark can only measure capabilities along specific dimensions.
65%VerifiedArtificial Analysis Intelligence Index整合了MMLU、HumanEval、MATH、GPQA等多个基准测试的加权结果
65%VerifiedBenchmarks such as MMLU, HumanEval, and MATH are used to quantify differences in model capabilities.
65%VerifiedMMLU, HumanEval, and MT-Bench are standardized benchmark test sets widely recognized by the AI community.
65%UnverifiedThe industry has begun shifting toward more challenging benchmarks like GPQA and MATH as older benchmarks lose discriminative power
80%UnverifiedOpenAI's 2023 paper "Let's Verify Step by Step" demonstrated that process supervision (PRM) significantly outperforms outcome supervision (ORM) on mathematical reasoning tasks.
70%Partially VerifiedGSM8K是小学数学推理基准，HumanEval是代码生成基准，HellaSwag是常识推理基准
65%UnverifiedThere is often a significant gap between benchmark scores and real-world application performance; a model with a slightly lower MMLU score may perform better in specific vertical domains.
60%UnverifiedAfter GSM8K saturated, the community turned to MATH (competition-level mathematics), GPQA (graduate-level science Q&A), and agent-type evaluations that require executing code or calling external tools.
50%UnverifiedJOLT's absolute accuracy on MATH, the model scale at which it matches GRPO, whether the 6.5x token reduction comes at the cost of more training steps or a more complex teacher training phase, and whether there is a risk of capability degradation or mode collapse under shared parameters are all open questions still to be answered.
50%UnverifiedThe currently public JOLT efficiency comparison is limited to the single MATH benchmark, so its performance on code generation, multi-turn dialogue, or open-domain reasoning cannot yet be inferred.
50%UnverifiedOn the MATH mathematical reasoning benchmark, JOLT achieved the same accuracy as GRPO, but processed only about one-sixth as many training tokens as GRPO.
50%UnverifiedJOLT's currently public information covers only the MATH benchmark, and its generalization ability still needs more verification.
50%UnverifiedCommon reasoning benchmarks include GSM8K (elementary school math word problems), MATH (competition-level math problems), StrategyQA (multi-hop commonsense reasoning), and others; these benchmarks require multi-step derivation and have automatically verifiable answers.
50%UnverifiedGSM8K, MATH, and BBH are three commonly used reasoning benchmarks with a clear difficulty gradient: GSM8K consists of elementary school math word problems, MATH covers competition-level math problems, and BBH contains 23 task types requiring multi-step symbolic manipulation or strategic reasoning.
50%