Benchmark Contamination

Benchmark contamination refers to a model having already seen test questions during training, resulting in inflated scores that fail to reflect true generalization ability.

Benchmark contamination refers to the phenomenon where test questions may have already appeared in training data, causing high scores to not truly represent generalization ability.

Benchmark data contamination refers to the phenomenon where, after a test set is made public, its questions may appear in the pre-training corpora of subsequent models, causing scores to reflect memorization rather than genuine reasoning ability.

Benchmark contamination refers to the situation where benchmark questions have entered a model's training data, causing the model to "recall answers" rather than genuinely reason, inflating benchmark scores and failing to reflect actual capability.

For code benchmarks the contamination risk is especially prominent, because contest solutions and discussion posts on code hosting platforms such as GitHub are often included in training corpora.

The paper recommends adopting dynamically generated test cases and private held-out sets, and reporting confidence intervals rather than single scores, to mitigate benchmark contamination.

Benchmark contamination refers to the possibility that after an evaluation set is made public, models may be trained specifically for it, so benchmarks need to be continuously updated or keep a private held-out set to maintain credibility

Common measures to mitigate benchmark contamination include using dynamically generated new questions, keeping the question bank confidential before release, or mixing evaluations across multiple dimensions to reduce the weight of any single dataset.

A study published on arXiv (arXiv:2609.02899) proposes that concerns about benchmark contamination should be distinguished into two separate questions: whether absolute scores are inflated and whether the ranking order of models is changed.