ARC-AGI
ARC-AGI (Abstraction and Reasoning Corpus for Artificial General Intelligence) is an AI reasoning benchmark proposed by François Chollet to evaluate the abstract reasoning and generalization capabilities of AI systems. The benchmark presents visual pattern transformation tasks requiring models to induce rules from a small number of examples and infer answers, emphasizing human-like core knowledge reasoning rather than large-scale memorization or statistical learning. It is a reference standard for measuring progress toward artificial general intelligence.
Core Facts
ARC-AGI is a benchmark designed by AI safety researcher François Chollet specifically to measure fluid intelligence.
François Chollet emphasizes abstract reasoning and generalization ability, and proposed the ARC benchmark as a measure of these capabilities.
ARC-AGI was designed by AI researcher François Chollet to create a test set that is simple for human intuition but extremely difficult for language models that rely on data memorization.
Timeline (last 90 days)
ARC-AGI was proposed by François Chollet in 2019, with the design philosophy that each task is solvable by humans but difficult for machine learning, and humans can usually complete them within minutes.
ARC-AGI tasks consist of several pairs of colored grids, and the model needs to infer the transformation rule from the input-output demonstrations and apply it to a new input grid.
ARC-AGI was proposed in 2019 by former Google engineer François Chollet, whose original design intent was to provide a general reasoning evaluation set unaffected by the distribution of training data.
The ARC-AGI series of benchmarks is maintained by the Arc Prize Foundation.
ARC-AGI is a general intelligence benchmark proposed by François Chollet, comprising a series of colored grid transformation tasks that require a model to infer abstract transformation rules from very few examples and apply them to new inputs.
Compared with benchmarks such as ARC-AGI, this framework allows finer-grained control over the dimension of novelty—whether the rule itself is new, the rule combination is new, or the rule instance is new.
Chollet has repeatedly emphasized that ARC is merely a tool for measuring a certain type of abstract reasoning ability rather than the ultimate criterion for determining AGI, and that achieving a high score on ARC-AGI does not equate to achieving artificial general intelligence.
The ARC-AGI benchmark is designed to distinguish between two capabilities: 'performing well on tasks that have been seen before' and 'being able to flexibly handle entirely novel structural problems.'
Claude Opus's best score on the ARC-AGI private holdout set with native validation is 30%.
Prime Agent achieved approximately 78% on ARC-AGI using the GPT series and 95.5% using Opus.
34 more timeline events
All Facts (20)
ARC-AGI is a benchmark designed by AI safety researcher François Chollet specifically to measure fluid intelligence.
85%VerifiedFrançois Chollet emphasizes abstract reasoning and generalization ability, and proposed the ARC benchmark as a measure of these capabilities.
80%VerifiedARC-AGI was designed by AI researcher François Chollet to create a test set that is simple for human intuition but extremely difficult for language models that rely on data memorization.
80%VerifiedARC-AGI was proposed by AI safety researcher François Chollet in 2019.
80%VerifiedARC-AGI specifically measures fluid intelligence, whereas traditional large language models primarily exhibit crystallized intelligence.
75%VerifiedUnder the standard framework of ARC Prize, GPT-6 Astra's ARC-AGI-3 score is only 62.7.
65%VerifiedThe ARC-AGI benchmark was proposed by François Chollet in 2019 to evaluate AI's abstract reasoning and generalization capabilities when faced with entirely novel problems.
65%UnverifiedARC-AGI的每道题由彩色网格构成，要求模型从2-3个输入输出示例中归纳出变换规则再应用到新输入上
95%UnverifiedOnce a public benchmark attracts widespread attention, it risks being targeted for optimization, and high scores do not necessarily reflect generalized reasoning ability.
60%UnverifiedARC-AGI-2 was designed by AI safety researcher François Chollet; early GPT-4 scored less than 10% on this test, while the human average score is approximately 85%
60%UnverifiedARC-AGI was proposed by François Chollet in 2019, with the design philosophy that each task is solvable by humans but difficult for machine learning, and humans can usually complete them within minutes.
50%UnverifiedThe ARC-AGI series of benchmarks is maintained by the Arc Prize Foundation.
50%UnverifiedARC-AGI was proposed in 2019 by former Google engineer François Chollet, whose original design intent was to provide a general reasoning evaluation set unaffected by the distribution of training data.
50%UnverifiedARC-AGI tasks consist of several pairs of colored grids, and the model needs to infer the transformation rule from the input-output demonstrations and apply it to a new input grid.
50%UnverifiedCompared with benchmarks such as ARC-AGI, this framework allows finer-grained control over the dimension of novelty—whether the rule itself is new, the rule combination is new, or the rule instance is new.
50%UnverifiedARC-AGI is a general intelligence benchmark proposed by François Chollet, comprising a series of colored grid transformation tasks that require a model to infer abstract transformation rules from very few examples and apply them to new inputs.
50%UnverifiedChollet has repeatedly emphasized that ARC is merely a tool for measuring a certain type of abstract reasoning ability rather than the ultimate criterion for determining AGI, and that achieving a high score on ARC-AGI does not equate to achieving artificial general intelligence.
50%UnverifiedThe ARC-AGI benchmark is designed to distinguish between two capabilities: 'performing well on tasks that have been seen before' and 'being able to flexibly handle entirely novel structural problems.'
50%UnverifiedOn the ARC-AGI benchmark, the same model can jump in score from 30% to 95% or even 100% solely through different Harness designs.
50%UnverifiedClaude Opus's best score on the ARC-AGI private holdout set with native validation is 30%.
50%