LLM Reinforcement Learning: Is the Importance Sampling Ratio Really Necessary? LGPO Says No

LGPO replaces importance sampling ratios with likelihood gating, eliminating old-policy likelihood computation, cutting training time by ~18% and improving robustness.
The article introduces a study from RSS that questions the necessity of importance sampling ratios in LLM RL post-training. Traditional methods like GRPO require computing both old and current policy likelihoods to construct the importance sampling ratio and apply clipping, incurring significant double computation overhead. The study proposes LGPO (Likelihood-Gated Policy Optimization), which uses likelihood-based gating to implement a soft trust region constraint, completely eliminating old-policy likelihood computation. Empirical results show that removing importance sampling correction does not harm stability; what truly maintains stability is the trust region constraint itself. In fully on-policy scenarios, the importance sampling ratio is always 1, causing ratio clipping to fail and GRPO to potentially collapse, while LGPO remains stable. Under standard settings, LGPO achieves performance comparable to GRPO with approximately 18% average reduction in training time, achieving a win-win in performance and efficiency.
The Efficiency Bottleneck in RL Post-Training
Reinforcement learning (RL) has become a core component of post-training for large language models (LLMs). Among the many RL methods, GRPO (Group Relative Policy Optimization) is widely adopted for its simplicity and effectiveness. However, these methods have a somewhat inconspicuous yet costly step: to construct the importance sampling ratio, they must compute the likelihoods of both the old policy and the current policy. This dual computation introduces non-negligible overhead.
A recent study from RSS directly questions this design premise: Is the importance sampling ratio really necessary for stable RL training of LLMs?

GRPO (Group Relative Policy Optimization) is a reinforcement learning algorithm proposed by the DeepSeekAd team in the DeepSeekMath paper. Its core idea is to abandon the separate value network (critic) in PPO and instead use the average reward within a group of multiple sampled responses to the same question as a baseline to estimate the advantage function. This change significantly reduces memory usage and computational complexity, making RL training feasible at LLM scale. The loss function of GRPO retains the PPO-style probability ratio clipping term, so each parameter update still requires obtaining the log-probabilities of both the old policy at sampling time and the current policy for the same token sequence. This is exactly the source of the double likelihood overhead described in the article.
Importance sampling is a technique in statistics for estimating the expected value under one distribution using data sampled from another distribution. Its correction weight is the importance sampling ratio $r_t(\theta) = \pi_\theta(a_t|s_t) / \pi_{\theta_{old}}(a_t|s_t)$. In PPO-style methods, this ratio corrects the gradient bias caused by the distribution shift before and after the policy update, and constructs a trust region by clipping $r_t$ to the interval $[1-\epsilon, 1+\epsilon]$. In RL post-training for LLMs, this means the training engine must maintain a copy of the old policy that does not participate in gradient updates, or cache the log-probabilities from its forward pass, resulting in additional memory and computational overhead.
LGPO: Replacing Importance Sampling with Likelihood Gating
The study proposes Likelihood-Gated Policy Optimization (LGPO), whose core idea is to enforce a soft trust region constraint via likelihood-based gating, thereby completely eliminating the computation of old-policy likelihoods.
The role of the traditional importance sampling ratio is to correct the distribution shift between the old and new policies and to restrict policy updates within a trust region via clipping. The observation of LGPO is that these two things can actually be decoupled—the trust region constraint need not rely on the importance sampling ratio. Through likelihood gating, LGPO can maintain the stability of policy updates without requiring old-policy likelihoods.
Training Remains Stable Without Importance Sampling Correction
Empirical results yield a rather counterintuitive conclusion: removing the importance sampling correction term does not harm training stability; instead, removing the trust region mechanism can lead to training collapse. This indicates that in RL post-training for LLMs, what truly maintains stability is the trust region constraint itself, not the intermediate variable of the importance sampling ratio.
This finding has direct methodological implications: since the importance sampling ratio is not a necessary condition for stability, the double likelihood overhead incurred to compute it can, at least in some scenarios, be eliminated.
The trust region constraint is a class of stability mechanisms in policy gradient methods. Its goal is to limit the distribution difference between the new and old policies after each update, avoiding a sharp drop in policy performance due to excessively large step sizes. TRPO implements the trust region with a hard constraint on KL divergence, while PPO uses ratio clipping as an approximation. In RL training for LLMs, because reward signals are sparse and the policy space is extremely high-dimensional, the trust region constraint is particularly critical for preventing training collapse. However, how to enforce this constraint at lower cost has always been an engineering trade-off. LGPO's likelihood gating essentially uses the absolute likelihood level of the current policy on the sampled tokens to indirectly control the update magnitude, thereby bypassing the explicit dependence on the old policy distribution.
Why Ratio Clipping Fails in Fully On-Policy Scenarios
The study also reveals a more subtle failure mode. In fully on-policy training, the policy update is consistent with the sampling policy, and the importance sampling ratio is always 1. At this point, the ratio-based clipping mechanism is never activated—the trust region constraint becomes ineffective.
In other words, the ratio clipping that GRPO relies on fails precisely in the on-policy scenario where it is supposed to work. And once an effective trust region constraint is lost, training may collapse. This also explains why GRPO fails in fully on-policy training, while LGPO remains stable thanks to its likelihood-based gating mechanism.
On-policy and off-policy are two fundamental concepts in reinforcement learning that distinguish data sources. On-policy means that the data used to update the policy must be sampled in real time by the current policy, and the data is discarded after one use; off-policy allows the use of data generated by historical or even other policies, usually with an experience replay buffer. PPO is essentially a proximal policy optimization method that strikes a compromise between the two via the importance sampling ratio, allowing multiple epochs of mini-batch updates. When training is strictly on-policy and each data batch is used for only one update, the old policy and the current policy are completely identical, the ratio is always 1, the clipping term degenerates into an identity mapping, and the trust region constraint effectively no longer works—this is exactly the failure mechanism pointed out in the article.
A Win-Win in Performance and Efficiency
Under standard training settings where GRPO itself performs stably, LGPO achieves training stability and peak performance comparable to GRPO. More importantly, efficiency: compared to the standard implementation that requires recomputing old-policy likelihoods, LGPO reduces training time by approximately 18% on average.
The significance of this number lies in the fact that it is a pure efficiency gain obtained without sacrificing performance. For large-scale LLM post-training, the compression of training time directly translates into savings in computational cost. And in fully on-policy training scenarios where GRPO fails, LGPO not only remains stable but also opens up a more efficient training paradigm.
Implications for LLM RL Post-Training
The value of this work goes beyond a faster optimizer. It re-examines a design convention that is taken for granted in LLM reinforcement learning: the necessity of the importance sampling ratio, a classic component of PPO-style methods, deserves re-evaluation in the LLM context.
When the trust region constraint can be implemented in a lighter way, the entire computational pipeline built around the importance sampling ratio—the forward computation of old-policy likelihoods and the ratio clipping logic—has room for simplification. The empirical evidence from LGPO shows that this simplification is not only feasible but also maintains robustness across a wider range of training scenarios. For teams building RL post-training pipelines, this is a direction worth paying attention to.
Related articles

Why Microsoft CEO Nadella Still Backs Nvidia After Jensen Huang's Departure
Microsoft CEO Nadella publicly expresses confidence in Nvidia interim CEO Colette Kress. This article analyzes Nvidia's power transition after Jensen Huang's departure, Microsoft's compute dependency, and the trust logic of the AI supply chain.

Nadella: Critical Infrastructure Is 'Unpatchable' as AI Cybersecurity Arms Race Escalates
Microsoft CEO Nadella addresses AI cyberattack fears, saying much critical infrastructure is unpatchable, downplaying the Outlook outage, and analyzing AI's dual role in cybersecurity.

Musk Backs Jensen Huang: Firing Him Is Like Cutting Jordan
Musk says Jensen Huang is loyal to the U.S. and likens firing him to cutting Michael Jordan—highlighting politicization anxieties in the AI chip industry and defending talent value.