ARC-Challenge
A scientific reasoning benchmark released by the Allen Institute for AI, consisting of approximately 1,172 multiple-choice questions selected because neither traditional retrieval methods nor word co-occurrence algorithms can solve them. The benchmark emphasizes genuine reasoning ability over surface-level pattern matching.
Timeline (last 90 days)
In the author's published 25-question small-sample evaluation, ZY-S1 improved over the base model on ARC-Challenge from 76 to 80 (+4 percentage points) and on MMLU from 48 to 52 (+4 percentage points).
A Reddit technical post claims that a model with only 0.8B parameters achieved a score of 42.15% on the ARC-Challenge benchmark, surpassing the untuned stock Qwen3.5-2B base model and outperforming a solution using SFT distillation with 25,000 data entries.
The Challenge subset of ARC specifically selects questions that both traditional IR systems and word co-occurrence models answered incorrectly, comprising approximately 1,172 test questions.
ARC-Challenge consists of four-choice multiple-choice questions, with a random baseline of 25%.
GPT-4-level models can achieve over 90% on the ARC-Challenge test, and many mainstream models above 7B can already exceed 70%.
The claim lacks support from a complete paper and open-source code, and key issues such as evaluation setup (0-shot or few-shot), fairness of comparisons, and reproducibility cannot be confirmed from the original content.
All Facts (3)
The Challenge subset of ARC specifically selects questions that both traditional IR systems and word co-occurrence models answered incorrectly, comprising approximately 1,172 test questions.
50%UnverifiedARC-Challenge consists of four-choice multiple-choice questions, with a random baseline of 25%.
50%UnverifiedGPT-4-level models can achieve over 90% on the ARC-Challenge test, and many mainstream models above 7B can already exceed 70%.
50%