Data Leakage

Benchmark contamination refers to the situation where benchmark questions have entered a model's training data, causing the model to "recall answers" rather than genuinely reason, inflating benchmark scores and failing to reflect actual capability.

The root cause of the issue in this case was data leakage in the data split, where random assignment placed nearly duplicate form templates on both the training set and regression test set sides simultaneously

Data leakage refers to future information or target-related information that should not appear in training data accidentally flowing into the feature set, commonly seen in time series misuse of future features or normalization performed before splitting.

Data leakage refers to test set information being indirectly seen by the model during training, causing evaluation metrics to be inflated; it is one of the most common and most concealed methodological flaws in machine learning research, and is especially dangerous in the medical prediction domain.

Data leakage falls into two categories—target leakage and training set leakage—and causes models to perform well during testing but fail after deployment.

Performing StandardScaler normalization on the entire dataset before the train/test split causes the mean and variance information of the test set to leak into the training set, constituting data leakage.

Machine learning evaluation assumes that evaluation samples are mutually independent and that no information from the evaluation set has flowed into the training process in any form; randomly splitting multiple dimensions of the same document breaks this assumption, causing data leakage and inflating evaluation scores.

The BUG:/DOC:/ENH: prefixes commonly found in Issue titles cause data leakage, because the model would directly "copy the answer" from the title prefix rather than learn from the report content, so these prefixes must be removed before training.

When there is speaker overlap between the training set and test set, an accent classification model may learn to recognize specific speakers rather than accent features, leading to inflated evaluation results.

The retention column in the dataset is only populated after a customer initiates a cancellation, making it a data leakage feature that should not be used for training.

The solution to the data leakage is to have the agent perform training on an isolated branch restored to the exact timestamp prior to the prediction window.

Other common sources of data leakage include computing mean/standard deviation for normalization using the full dataset, using future time-point information in feature engineering, and failing to apply cross-validation isolation during target encoding.

Data leakage almost never causes an error; the code runs normally and the metrics look great, making it one of the hardest-to-automatically-detect pitfalls in AI-generated code.

The way to avoid data leakage is to split data by individual — all augmented images derived from the same original image must go either entirely into the training set or entirely into the validation set.

If data augmentation is performed before splitting the dataset, and multiple images derived from the same original image end up in both the training set and the validation set, data leakage will occur.

In AutoML tasks, DeepSeek Flash can autonomously formulate plans, perform feature selection, run multiple sets of hyperparameter experiments, and proactively identify and eliminate features with data leakage.