GPQA
GPQA (Graduate-Level Google-Proof Q&A) is a professional knowledge question-answering benchmark at the graduate level, designed to evaluate large language models' deep reasoning and domain knowledge in natural sciences including biology, chemistry, and physics. Questions are crafted by domain experts, are of high difficulty, and are not easily answered even with the aid of search engines. The benchmark is commonly used to measure models' reasoning performance at the limits of specialized domains.
Core Facts
Benchmark testing is an important reference for verifying model capabilities, and common evaluation dimensions include mathematical reasoning, code generation, and scientific question answering.
Standard AI benchmarks include MMLU (a comprehensive knowledge test covering 57 subjects), GPQA (graduate-level professional question answering), SWE-bench (real-world software engineering tasks), HumanEval and MBPP (programming ability), and MATH and GSM8K (mathematical reasoning)
Credible capability evaluation of models in the AI field relies on standardized benchmark tests; commonly used evaluation sets include MMLU, HumanEval, MATH, and GPQA.
Timeline (last 90 days)
After GSM8K saturated, the community turned to MATH (competition-level mathematics), GPQA (graduate-level science Q&A), and agent-type evaluations that require executing code or calling external tools.
GPQA uses multi-round expert review and consistency voting to ensure question quality, with each question independently verified and cross-challenged by at least three domain experts.
Benchmarks are standardized evaluation sets and metrics commonly used in machine learning, such as MMLU (multidisciplinary knowledge), HumanEval (code generation), GSM8K (mathematical reasoning), etc.
Open-weight models such as Llama, Mistral, and Qwen may lag behind top closed-source models such as GPT-4 or Claude by a few percentage points on benchmarks like MMLU and GPQA, but they already have practical value for common academic research tasks such as literature summarization, coding assistance, hypothesis generation, and data cleaning.
Benchmark testing is an important reference for verifying model capabilities, and common evaluation dimensions include mathematical reasoning, code generation, and scientific question answering.
Commonly used evaluation benchmarks for measuring the gap between open-source and closed-source models include HumanEval/SWE-bench for coding ability, MATH and GPQA for reasoning ability, and MMLU for overall performance.
Artificial Analysis's Intelligence Index is a composite score derived from a weighted average of a series of existing benchmarks (such as MMLU, HumanEval, MATH, etc.).
Astra scored lower than small models such as GLM 5.3 Flash on the Artificial Analysis intelligence index, because that index is based on outdated and saturated benchmarks such as GPQA.
HumanEval and SWE-bench are commonly used to evaluate large model coding capabilities, MMLU and GPQA for comprehensive reasoning, and IFEval for instruction following.
Commonly used benchmarks for evaluating large language model capabilities include MMLU, HumanEval, MATH, and GPQA.
18 more timeline events
All Facts (20)
Benchmark testing is an important reference for verifying model capabilities, and common evaluation dimensions include mathematical reasoning, code generation, and scientific question answering.
85%VerifiedStandard AI benchmarks include MMLU (a comprehensive knowledge test covering 57 subjects), GPQA (graduate-level professional question answering), SWE-bench (real-world software engineering tasks), HumanEval and MBPP (programming ability), and MATH and GSM8K (mathematical reasoning)
85%VerifiedCredible capability evaluation of models in the AI field relies on standardized benchmark tests; commonly used evaluation sets include MMLU, HumanEval, MATH, and GPQA.
80%VerifiedDeepSeek V4.1 Flash scores lower than the previous version on GPQA, HLE, and other benchmarks.
75%VerifiedGPQA is a benchmark designed for PhD-level scientific question answering, with questions difficult enough that domain experts can only answer approximately 65% correctly.
70%VerifiedThird-party platforms such as Artificial Analysis normalize scores from multiple sub-benchmarks and aggregate them into a single quality index, enabling different models to be compared on the same scale.
65%VerifiedGPQA (Graduate-Level Google-Proof Q&A) focuses on questions requiring doctoral-level domain expertise.
65%VerifiedArtificial Analysis Intelligence Index整合了MMLU、HumanEval、MATH、GPQA等多个基准测试的加权结果
65%VerifiedThe three major benchmarks MMLU, GSM8K, and HumanEval have all experienced varying degrees of contamination, with researchers finding content in the training data of multiple mainstream models that overlaps highly with test questions
65%UnverifiedThe industry has begun shifting toward more challenging benchmarks like GPQA and MATH as older benchmarks lose discriminative power
80%UnverifiedGPQA (Graduate-Level Google-Proof Q&A) focuses on doctoral-level scientific reasoning problems.
60%UnverifiedGPQA is a graduate-level scientific question-answering benchmark, and SWE-bench targets real-world software engineering tasks.
60%UnverifiedGPQA was designed by a research team at New York University; non-experts using search engines achieve an accuracy of approximately 34%, while domain experts average approximately 65%.
60%UnverifiedAfter GSM8K saturated, the community turned to MATH (competition-level mathematics), GPQA (graduate-level science Q&A), and agent-type evaluations that require executing code or calling external tools.
50%UnverifiedGPQA uses multi-round expert review and consistency voting to ensure question quality, with each question independently verified and cross-challenged by at least three domain experts.
50%UnverifiedBenchmarks are standardized evaluation sets and metrics commonly used in machine learning, such as MMLU (multidisciplinary knowledge), HumanEval (code generation), GSM8K (mathematical reasoning), etc.
50%UnverifiedOpen-weight models such as Llama, Mistral, and Qwen may lag behind top closed-source models such as GPT-4 or Claude by a few percentage points on benchmarks like MMLU and GPQA, but they already have practical value for common academic research tasks such as literature summarization, coding assistance, hypothesis generation, and data cleaning.
50%UnverifiedCommonly used evaluation benchmarks for measuring the gap between open-source and closed-source models include HumanEval/SWE-bench for coding ability, MATH and GPQA for reasoning ability, and MMLU for overall performance.
50%UnverifiedArtificial Analysis's Intelligence Index is a composite score derived from a weighted average of a series of existing benchmarks (such as MMLU, HumanEval, MATH, etc.).
50%UnverifiedHumanEval and SWE-bench are commonly used to evaluate large model coding capabilities, MMLU and GPQA for comprehensive reasoning, and IFEval for instruction following.
50%