Concept
Benchmark Saturation
A phenomenon in which models approach the ceiling score on a fixed benchmark, making it impossible to distinguish differences in capability. It stems from fixed questions and answers that can be memorized through training.
Timeline (last 90 days)
Aug 30
Benchmark saturation refers to the loss of discriminative power of a benchmark when multiple models approach perfect scores on it, prompting the community to introduce new evaluation standards such as GPQA and SWE-bench.
Unverified50%