SimpleQA
A benchmark introduced by OpenAI to evaluate the factual question-answering capability of language models. It specifically tests model accuracy on short, unambiguous factual questions and is used to measure the degree of hallucination in model outputs.
Timeline (last 90 days)
SimpleQA derives from the SimpleQA dataset released by OpenAI in 2024, which contains about 4000 human-written factual questions with unique and short answers.
The SimpleQA Verified benchmark contains 1000 questions, selected from OpenAI's original SimpleQA dataset after reference-answer verification, removal of similar questions and topic balancing.
On the SimpleQA benchmark, o4-mini declines to answer only 1% of questions, with an error rate of 75%.
o4-mini and gpt-5-thinking-mini have almost the same accuracy on SimpleQA, but the latter makes roughly three times fewer errors than the former.
On the SimpleQA benchmark, gpt-5-thinking-mini declines to answer 52% of questions, with an error rate of 26%.
Benchmarks being led and established by the model's publishing party is a common phenomenon in the current large model field, with similar situations occurring in OpenAI's SimpleQA, Google's BIG-Bench, and other evaluations.
v4.1 Flash's expert weights are shipped natively in FP4 (4 bits per parameter) and no 8-bit version has ever been released; being close to the compression limit of this format, it cannot be further quantized for Agentic tasks.
SimpleQA is a benchmark introduced by OpenAI to evaluate the factual question-answering ability of language models, testing the accuracy of model responses to short, factual questions with definitive answers.
The developer claims MiAPI achieves 82% accuracy on the SimpleQA benchmark, surpassing Perplexity.
SimpleQA是OpenAI于2024年10月开发的评测基准，包含4326道有明确事实答案的简短问题
1 more timeline events
All Facts (11)
v4.1 Flash's expert weights are shipped natively in FP4 (4 bits per parameter) and no 8-bit version has ever been released; being close to the compression limit of this format, it cannot be further quantized for Agentic tasks.
60%UnverifiedSimpleQA derives from the SimpleQA dataset released by OpenAI in 2024, which contains about 4000 human-written factual questions with unique and short answers.
50%UnverifiedThe SimpleQA Verified benchmark contains 1000 questions, selected from OpenAI's original SimpleQA dataset after reference-answer verification, removal of similar questions and topic balancing.
50%UnverifiedOn the SimpleQA benchmark, o4-mini declines to answer only 1% of questions, with an error rate of 75%.
50%Unverifiedo4-mini and gpt-5-thinking-mini have almost the same accuracy on SimpleQA, but the latter makes roughly three times fewer errors than the former.
50%UnverifiedOn the SimpleQA benchmark, gpt-5-thinking-mini declines to answer 52% of questions, with an error rate of 26%.
50%UnverifiedBenchmarks being led and established by the model's publishing party is a common phenomenon in the current large model field, with similar situations occurring in OpenAI's SimpleQA, Google's BIG-Bench, and other evaluations.
50%UnverifiedSimpleQA is a benchmark introduced by OpenAI to evaluate the factual question-answering ability of language models, testing the accuracy of model responses to short, factual questions with definitive answers.
50%UnverifiedThe developer claims MiAPI achieves 82% accuracy on the SimpleQA benchmark, surpassing Perplexity.
50%UnverifiedSimpleQA是OpenAI于2024年10月开发的评测基准，包含4326道有明确事实答案的简短问题
50%UnverifiedSimpleQA是OpenAI开发的评测基准，专门测试模型在面对事实性问题时是否能准确判断自身知识边界
50%