Unverified50% confidenceFactExact time
Multiple studies in 2024 found that the strong performance of certain models on standard benchmarks resulted from benchmark test questions being mixed into the training data (data contamination).
1
Sources
50%
Confidence
Long-term
Relevance
7/24/2026
First Seen
Sources
Related Claims
VerifiedBenchmarks face contamination risks; if training data contains test questions or approximate variants, a model's high scores may partly stem from memorization rather than genuine capability.78% similarUnverified智能体的偏好受训练数据、提示词、版本迭代等多重因素影响，今天有效的优化手段可能在下一次模型更新后失效76% similarUnverifiedThe three major benchmarks MMLU, GSM8K, and HumanEval have all experienced varying degrees of contamination, with researchers finding content in the training data of multiple mainstream models that overlaps highly with test questions75% similarUnverifiedData annotation quality directly determines the upper bound of model training, and annotation in specialized domains requires the participation of domain experts.75% similarUnverifiedThe cleanliness of a benchmark is time-limited—once test questions are publicly circulated, future model training will very likely absorb them, causing fairness to decay accordingly.74% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/605909API
curl https://kongchang.com/api/v1/knowledge/claims/605909MCP
get_claim(id=605909)