Score Centering

Score Centering is expected to become the default practice in RLHF training.

Score Centering does not attempt to eliminate engineering differences (quantization, asynchrony, etc.) but mathematically compensates for their effect on the gradient, so it applies to all the above types of mismatch.

With the corrected score, the drift term in the expected gradient is completely cancelled, leaving only the covariance term (the learning signal).

Score Centering remains stable under extreme quantization scenarios (such as 8-bit weights and 4-bit KV cache), whereas other methods collapse.

Score Centering is compatible with techniques such as truncated importance sampling and can be used in combination.

The key observation of Score Centering is that the drift term originates from the nonzero expectation of the score function under Q, so subtracting that expectation exactly cancels it without affecting the covariance signal.

Score Centering reduces the variance of gradient estimates by applying mean subtraction to the score function.

Score Centering places the effect of the baseline on the off-policy-corrected score term, rather than treating only the reward side.

If Score Centering is experimentally validated as effective, it will provide support for off-policy LLM training approaches that go beyond the PPO paradigm.

The paper 'Score Centering Stabilizes Off-policy Reinforcement Learning' was authored by researchers Marek and Ryabinin from Together.ai.