on-policy

ORL adopts on-policy residual learning: training data is generated in real time by the current model to ensure that the training distribution matches the inference distribution.

PPO is an on-policy algorithm, requiring that the data used in each update must come from the current policy.

Mainstream RL algorithms such as PPO require that the policy used to generate samples be as consistent as possible with the policy currently being optimized (on-policy); otherwise, off-policy data introduces bias, reduces sample efficiency, and can potentially cause training instability.

The core difference between reasoning models and ordinary LLMs lies in the training approach rather than the architecture; reasoning models introduce a reinforcement learning stage on top of pretraining and supervised fine-tuning.

Off-policy algorithms (such as SAC and DQN) can be trained on data collected by any policy, are capable of maintaining a replay buffer of millions of experiences for repeated learning, and have higher sample efficiency than on-policy methods.