Drift Detection
An engineering practice for detecting changes over time in the output of hosted model APIs. By running a fixed set of prompts repeatedly, hashing the outputs and comparing against a baseline, it statistically identifies drift in model behaviour.
Timeline (last 90 days)
Agents need to monitor their own performance and proactively discard outdated knowledge when performance degradation is detected.
In a drift canary, a Jaccard similarity below 0.90 or a length change exceeding 15% can serve as heuristic thresholds for flagging drift, but the thresholds need to be adjusted to the noise floor and are not laws.
The output-interpretation decision table states: when n_ok across multiple prompts is below K, fix access first (authentication, quota, or network issues), as this is not drift; when a constrained prompt's Jaccard is below 0.90, it may be a snapshot replacement, so pin a dated snapshot and rerun the evaluation.
A drift detector that produces false alarms is worse than no detector, because every false alarm trains operators to ignore the signal.
Drift detection should be measured against the agent's own baseline on comparable corpora (the same batch of tasks, the same corpus version), and a corpus version change should establish a new baseline rather than be treated as a drift signal.
True LLM reproducibility should be expressed through behavioral contracts, that is, downstream metrics of output hashes on a given corpus (pass rate, keyword hit rate, structured JSON parsability) should remain statistically stable rather than byte-for-byte identical.
The statistical premise of drift detection is that the two sets of observations being compared must be comparable samples from the same distribution; directly comparing raw counts across versions produces false positives.
Automated drift detection mechanisms can expose problems but cannot force people to act, so freshness files should be made public to compensate for human inertia through external pressure
All Facts (8)
Agents need to monitor their own performance and proactively discard outdated knowledge when performance degradation is detected.
50%UnverifiedIn a drift canary, a Jaccard similarity below 0.90 or a length change exceeding 15% can serve as heuristic thresholds for flagging drift, but the thresholds need to be adjusted to the noise floor and are not laws.
50%UnverifiedThe output-interpretation decision table states: when n_ok across multiple prompts is below K, fix access first (authentication, quota, or network issues), as this is not drift; when a constrained prompt's Jaccard is below 0.90, it may be a snapshot replacement, so pin a dated snapshot and rerun the evaluation.
50%UnverifiedA drift detector that produces false alarms is worse than no detector, because every false alarm trains operators to ignore the signal.
50%UnverifiedDrift detection should be measured against the agent's own baseline on comparable corpora (the same batch of tasks, the same corpus version), and a corpus version change should establish a new baseline rather than be treated as a drift signal.
50%UnverifiedTrue LLM reproducibility should be expressed through behavioral contracts, that is, downstream metrics of output hashes on a given corpus (pass rate, keyword hit rate, structured JSON parsability) should remain statistically stable rather than byte-for-byte identical.
50%UnverifiedThe statistical premise of drift detection is that the two sets of observations being compared must be comparable samples from the same distribution; directly comparing raw counts across versions produces false positives.
50%UnverifiedAutomated drift detection mechanisms can expose problems but cannot force people to act, so freshness files should be made public to compensate for human inertia through external pressure
50%