AI Puzzles
An AI evaluation format situated between benchmarks and open problem lists. Through graded, laddered puzzle design, it requires machine-checkable proofs and avoids model memorization and benchmark saturation.
Timeline (last 90 days)
AI puzzles are a new evaluation format between a benchmark and an open-problem list, proposed by researcher Jarek Duda, built with the aperiodic order research field as a pilot to construct a tiered, ladder-style puzzle set.
The ladder design of AI puzzles arranges questions in the same research field by difficulty from human level (solvable with pen and paper) to ASI level (fully open), with each level genuinely harder than the previous one.
In the AI puzzles format, lower-level puzzles already have exact answers and the answers can be automatically verified by program, so they can be used to train and check model output, and variations of the questions are easy to generate to prevent memorizing the answers.
The AI puzzles format stipulates that only machine-checkable proofs count as solved; floating-point values are merely evidence, not proof.
The open levels of AI puzzles provide the complete materials needed to solve the problem: strong numerical evidence, known failed paths, relevant files, and a machine-checkable success criterion.
Solvers of the open levels of AI puzzles are credited, including the agents and models used, recorded in the next version of the paper, forming a traceable contribution attribution mechanism.
The advantage of the AI puzzles format over traditional benchmarks is resistance to saturation, because the open levels have no fixed answers; its advantage over open-problem lists is a clear success criterion—a machine-checkable proof.
All Facts (7)
AI puzzles are a new evaluation format between a benchmark and an open-problem list, proposed by researcher Jarek Duda, built with the aperiodic order research field as a pilot to construct a tiered, ladder-style puzzle set.
50%UnverifiedThe ladder design of AI puzzles arranges questions in the same research field by difficulty from human level (solvable with pen and paper) to ASI level (fully open), with each level genuinely harder than the previous one.
50%UnverifiedIn the AI puzzles format, lower-level puzzles already have exact answers and the answers can be automatically verified by program, so they can be used to train and check model output, and variations of the questions are easy to generate to prevent memorizing the answers.
50%UnverifiedThe AI puzzles format stipulates that only machine-checkable proofs count as solved; floating-point values are merely evidence, not proof.
50%UnverifiedThe open levels of AI puzzles provide the complete materials needed to solve the problem: strong numerical evidence, known failed paths, relevant files, and a machine-checkable success criterion.
50%UnverifiedSolvers of the open levels of AI puzzles are credited, including the agents and models used, recorded in the next version of the paper, forming a traceable contribution attribution mechanism.
50%UnverifiedThe advantage of the AI puzzles format over traditional benchmarks is resistance to saturation, because the open levels have no fixed answers; its advantage over open-problem lists is a clear success criterion—a machine-checkable proof.
50%