BIG-Bench Hard
BIG-Bench Hard is a set of 23 of the most challenging tasks selected from the BIG-Bench benchmark, used to evaluate the complex reasoning capabilities of large language models.
Timeline (last 90 days)
Traditional judgment models output the final answer directly, skipping intermediate reasoning steps, and have limited accuracy on complex tasks such as BIG-Bench Hard.
BIG-Bench Hard is a subset of 23 high-difficulty tasks selected from the BIG-Bench benchmark, covering types such as multi-step arithmetic, logical reasoning, causal judgment, and date understanding.
On three selected BIG-Bench Hard tasks, nine independently constructed programs solved 450 executions of 150 development inputs.
ABACUS programs achieve 99.67% macro-average accuracy on the three BIG-Bench Hard tasks.
On Llama-3.2-3B, fine-tuning a brand-new dense model with the 5% of data selected by CACTUS reached 49.44% accuracy on BIG-Bench Hard; selecting data directly with the dense model gave 49.54%; data selected by the ALPS method could only reach 47.22%.
评估大语言模型推理可靠性的系统化基准测试包括GSM8K、MATH竞赛题集和BIG-Bench Hard
All Facts (6)
Traditional judgment models output the final answer directly, skipping intermediate reasoning steps, and have limited accuracy on complex tasks such as BIG-Bench Hard.
50%UnverifiedBIG-Bench Hard is a subset of 23 high-difficulty tasks selected from the BIG-Bench benchmark, covering types such as multi-step arithmetic, logical reasoning, causal judgment, and date understanding.
50%UnverifiedABACUS programs achieve 99.67% macro-average accuracy on the three BIG-Bench Hard tasks.
50%UnverifiedOn Llama-3.2-3B, fine-tuning a brand-new dense model with the 5% of data selected by CACTUS reached 49.44% accuracy on BIG-Bench Hard; selecting data directly with the dense model gave 49.54%; data selected by the ALPS method could only reach 47.22%.
50%Unverified评估大语言模型推理可靠性的系统化基准测试包括GSM8K、MATH竞赛题集和BIG-Bench Hard
50%UnverifiedOn three selected BIG-Bench Hard tasks, nine independently constructed programs solved 450 executions of 150 development inputs.
50%