Data Contamination

When a model's training data contains benchmark test questions or highly similar content, the model may achieve high scores through memorization rather than genuine understanding.

Benchmark data contamination refers to the phenomenon where, after a test set is made public, its questions may appear in the pre-training corpora of subsequent models, causing scores to reflect memorization rather than genuine reasoning ability.

Benchmark contamination refers to the situation where benchmark questions have entered a model's training data, causing the model to "recall answers" rather than genuinely reason, inflating benchmark scores and failing to reflect actual capability.

Data contamination refers to test problems having already appeared in pretraining corpora, causing a model's high scores to more closely reflect memorization rather than reasoning.

Data contamination refers to a situation where a model has already seen data from the test set during training, causing benchmark scores to be inflated.

Data leakage falls into two categories—target leakage and train-test contamination—and leads to systematic overestimation of model performance.

The training corpora of large language models may include unpublished academic preprints and research notes, raising concerns about data contamination.

Training set contamination is the core controversy in the current AI benchmark evaluation field; models may answer questions through memorization rather than reasoning, leading to inflated results.

Mixing benchmark questions into training data causes models to score deceptively well, known in the industry as data contamination or benchmark tuning.

When different raters understand the same level very differently, the Likert scale leads to unstable inter-rater reliability, and a single number cannot carry information about specific good and bad dimensions.

Because extensive discussions of benchmark questions exist on the internet, models may have already seen the answers during training, causing test scores to be inflated—a phenomenon known as the 'contamination problem'.

The difficulty of building evaluation sets lies in covering edge cases, avoiding data contamination, and ensuring alignment between evaluation criteria and real business value.

Some researchers have attempted to mitigate data contamination by constructing "future datasets" or dynamically generating new questions, but such methods have not yet become an industry standard.

A perfect score on a benchmark may stem from three possible causes: a capability breakthrough, data contamination, or changes in evaluation configuration or version.

Using unpublished papers reduces the risk of data contamination from the model having seen the answers in its training data.

Evaluations based on open-source codebases suffer from data contamination issues, as these codebases may have been included in large model training data.

The degree of harm caused by data contamination is positively correlated with a model's memorization capacity; small models with limited capacity cannot benefit from contaminated data even when exposed to it.

The data contamination problem refers to the possibility that a model may have already encountered ideas or partial proofs during training that researchers have not yet formally published.

If the model memorized the answers to contaminated benchmarks during pre-training, information loss from compressing the context will not affect performance, rendering the reported 'lossless compression' conclusion meaningless.