KL Divergence

The RLHF pipeline consists of three steps: training a reward model to predict human preference scores, adjusting language model parameters using the PPO reinforcement learning algorithm, and using KL divergence constraints to prevent the model from deviating from its original language capabilities.

KL divergence is asymmetric, i.e., D(p,q) ≠ D(q,p).

Common methods for detecting data drift include the KS test, PSI (Population Stability Index), and sliding-window-based statistical monitoring.

Cross-entropy = entropy of the true distribution + KL divergence.

从KL散度角度看，每一代合成数据与真实分布之间的散度都在单调递增，且误差会在迭代中累积

IDM uses measures such as KL divergence to minimize the difference between the student's reconstructed item distribution and the teacher's distribution.

Data drift can usually be detected by monitoring statistical distances between the input distribution and the training distribution (such as PSI or KL divergence), whereas concept drift can only be caught through labeled online evaluation.

Gaming KLD costs very little, requiring only adjusting quantization scale parameters on the evaluation data with almost no change to model weights; whereas gaming benchmarks requires fine-tuning the model, consuming large amounts of GPU compute, and is prone to exposing overfitting on held-out test sets.

In quantization evaluation, having the original model generate the dataset itself and computing KLD on these generated tokens with zero weight assigned to prompt tokens can better reflect real inference scenarios.

The relationship between KLD and downstream benchmark performance is not linear: when KLD is below a certain magnitude, downstream metrics are almost unaffected; only when KLD rises significantly does task performance begin to decline markedly.

Judging whether a quantized model is suitable for a specific task ultimately still requires evaluation in real scenarios and cannot rely on KLD alone.

Calibration data should be treated as training data for the quantized model, and any KLD computed on it or on data from the same distribution should be regarded as training error rather than generalization error.

KLD values themselves have no absolute good-or-bad threshold; they are meaningful only when two models are compared on the same data with the same calculation method, and KLD values reported in different papers are often not directly comparable across studies.

KLD measures relative entropy, i.e., the extra number of bits needed to encode the original model's output using the quantized model's distribution, so its unit is bits/token.

KLD is sensitive to the distribution tails: as long as the quantized model assigns extremely low probability to low-probability tokens, KLD will be significantly inflated, even if those tokens have little impact on generation quality.

KLD should be positioned as a diagnostic tool rather than a ranking basis; reasonable uses include observing distribution shift trends before and after quantization, comparing relative differences of different quantization configurations on the same data, and assisting in troubleshooting anomalies.

KLD (KL divergence) measures the difference between two probability distributions, with the formula D_KL(P||Q)=∑_x P(x)log(P(x)/Q(x)).

The KL divergence between SCEs and real execution is more than ten times lower than that of BashArena, indicating that the simulated environment closely matches the real environment in behavior distribution.

The paper derives an error bound between the squared distance of behavior map coordinates and the trajectory KL divergence under explicit assumptions.

The distillation loss often uses KL divergence to measure the difference between the student distribution and the teacher distribution.