Data Drift

Concept drift refers to the mapping relationship between input data and target labels changing over time, which is different from data drift where only the input distribution changes.

Distribution shift refers to the statistical distribution difference between the training environment and the deployment environment, and is the direct mechanism by which spurious correlations cause performance collapse.

Drift detection involves a trade-off: setting the threshold too sensitively leads to frequent false alarms and unnecessary retraining, while setting it too loosely may allow model performance to continuously degrade without being detected.

Data Drift refers to the continuous degradation of model performance without any errors when the statistical properties of real online data deviate from the training set.

Data drift or concept drift refers to the phenomenon where model performance degrades significantly when the real-world data distribution changes while the model is still trained on old data; the industry addresses this with continuous learning and active learning strategies.

模型漂移通常分为数据漂移和概念漂移两类，数据漂移指输入数据统计分布随时间变化，概念漂移指输入与输出关系本身发生变化

The core challenge of MLOps lies in model drift: as real-world data distributions change over time, model prediction accuracy gradually declines.

Performance drift refers to the phenomenon of gradual degradation in model output quality over time, which may stem from silent updates by the model provider, changes in the distribution of contextual data, and updates to the knowledge base of the RAG system.

The Reddit discussion shows that schema changes, feature pipeline failures and distribution drift are the main culprits behind sudden model degradation, with troubleshooting times ranging from several hours to a week.

Common triggers for a sudden drop in model performance include schema changes, feature pipeline failures, upstream data distribution drift, label errors and deployment configuration mistakes.

Data drift refers to a change in the distribution of input features P(X) while the relationship between features and labels P(Y|X) remains unchanged; concept drift is when P(Y|X) itself changes.

Data drift can usually be detected by monitoring statistical distances between the input distribution and the training distribution (such as PSI or KL divergence), whereas concept drift can only be caught through labeled online evaluation.

Most "sudden degradations" are the result of data drift accumulating near the model's decision boundary and then being exposed all at once, so monitoring should cover both feature distribution and prediction distribution; looking only at accuracy lags by days or even weeks.

The troubleshooting times mentioned in replies: schema changes 2-5 days, feature pipeline outages 1 day, label errors 3 days, deployment configuration errors several hours, upstream data distribution drift 5-7 days.

Under the declarative model, data drift (changes in upstream schema or distribution) manifests as contract validation failures at the framework level rather than hidden errors in procedural code.

Soft quality regressions in AI agents are often triggered by model version updates, prompt fine-tuning, tool interface changes, or retrieval data drift.

Model drift is a core problem that AI trading systems must continuously address, referring to the phenomenon in which a model's performance degrades over time in a production environment, and it is usually divided into two categories: data drift and concept drift.

State-drift is different from data drift or concept drift, which focus on changes in the model's input distribution or input-output mapping and can be detected with observability tools; state-drift concerns the expiration of stored beliefs, which can fail even when the input is entirely in-distribution.

Data drift does not trigger obvious errors; it only manifests slowly in business metrics.

It is necessary to regularly monitor the confidence distribution in production environments to prevent data drift from causing a previously well-calibrated model to become miscalibrated on new data.