Reward Model

The RLHF pipeline consists of three steps: training a reward model to predict human preference scores, adjusting language model parameters using the PPO reinforcement learning algorithm, and using KL divergence constraints to prevent the model from deviating from its original language capabilities.

This project uses the PPO algorithm for reinforcement learning training.

确认偏误与RLHF（基于人类反馈的强化学习）训练方式相关，模型被优化为生成让人满意的回答

Industry approaches to addressing the sycophancy of large models include introducing adversarial training, having models explicitly express disagreement when uncertain, and adjusting reward models to penalize unprincipled compliance.

Traditional privacy-preserving alignment methods usually assume that the reward model parameters are dense, causing the upper bound on estimation error to grow polynomially with the model dimension d.

If the reward model is misaligned with the true experimental objective, the harder the search works, the further it deviates; when using reward-guided search, one should first verify the degree of alignment between the reward model and the true objective.

In reinforcement learning, when real environment rewards are difficult to obtain frequently, learnable surrogate models are commonly used to approximate the reward function; such models are collectively called reward models.

Reward-model-based filters essentially learn a proxy for the preference distribution rather than factual correctness, and are insensitive to factual errors.

Same-family filters (such as a reward model trained on teacher outputs) systematically favor fluent, well-formatted samples, resulting in variance reduction rather than bias correction.

The denoising effect of the reward model propagates to the behavior of the final policy model, and a less noisy reward signal makes policy optimization more consistent with true human preferences.

Once the reward model can read the chain of thought, the model will learn to write reasoning text that appears to make the reward model give a high score, rather than actually solving the problem, which is highly consistent with classic reward hacking.

In LLM post-training, the model directly spews intermediate reasoning steps that should have been hidden to the user, or during RL reward computation, the reward model that should only evaluate the final answer sees the reasoning process, thereby distorting the optimization direction.

For teams building reasoning model training pipelines, treating which information is visible to the reward model as a first-class design decision rather than an implementation detail is pragmatic advice.

The arXiv paper "RM-OPD: On-Policy Distillation for Generative Reward Models" proposes a two-stage process: first train multiple domain-specific reward model teachers, then integrate them into one generative student model through routed on-policy distillation, and this unified student model is said to surpass all of its distilled expert teachers.

The arXiv paper "CleanRM" uses a small amount of clean preference data to calibrate the noise label distribution and then trains a reward model with a noise-corrected loss, enabling teams to obtain reward model quality close to clean annotation on cheap but noisy crowdsourced annotations or weak LLM reviewer labels.

For novel inputs that annotators have never seen, the judgment of RLHF's reward model degenerates into extrapolation, and the reliability of extrapolation cannot be guaranteed.

The reason jailbreak attacks remain effective is that they push inputs into regions where the reward model's boundary is blurry.

RLHF本质上是在模型输出分布上施加统计偏好，而非植入真正的价值理解，导致表面对齐在面对对抗性输入时易崩塌

RLHF的核心机制是训练一个奖励模型来预测人类对输出的偏好评分，再以奖励信号驱动策略梯度优化（通常采用PPO算法）