Entropy Is Not Equivalent: A New Framework for Reward-Conditioned Entropy Allocation in RLVR

In Reinforcement Learning with Verifiable Rewards (RLVR) training, policy entropy is often used to diagnose exploration and collapse, but there is no stable relationship between its numerical value and reasoning performance. A new paper points out that this ambiguity stems from global token entropy mixing the uncertainty of successful and failed trajectories, the diversity of correct solution strategies, and the variation between different implementations of the same policy. To address this, the authors propose Reward-Conditioned Entropy Allocation (RCEA), a hierarchical diagnostic framework that attempts to answer "what entropy is actually measuring" and "which part should we look at for entropy collapse."

The Blind Spot of Global Entropy: Why High Entropy Does Not Equal Good Exploration

In RLVR, the verifier gives a binary result, but policy entropy is averaged over all tokens. The paper argues that this global average mixes three types of uncertainty: the confusion in failed trajectories, the strategy diversity within correct solutions, and the random fluctuations from sampling the same policy. As a result, under different optimization objectives, entropy trajectories cannot form a stable ranking of reasoning accuracy—high entropy does not necessarily mean sufficient exploration, and low entropy does not necessarily mean collapse.

The core operation of RCEA is to separate full-trajectory entropy from average token entropy, condition the uncertainty on verifier results and process validity, and measure the retention of correct-solution modes. In this way, entropy is no longer a single scalar but a set of attributable quantities.

Policy Entropy in reinforcement learning usually refers to the Shannon entropy of the policy's output action distribution (in language models, the probability distribution over the next token), measuring the randomness or uncertainty of the policy. In the RLVR setting, the entropy of all tokens is averaged into a single global scalar to monitor the degree of exploration during training. However, this averaging ignores the different roles of tokens within a trajectory: some tokens determine key forks in the reasoning path, while others are merely syntactic filler. Global entropy therefore cannot distinguish between "meaningful diversity" and "meaningless noise."

Verifier in RLVR is an external program (such as a math answer checker or code executor) that gives a binary reward (correct/incorrect) for the complete trajectory generated by the model. It does not provide process rewards and only judges the final result, so the training signal is sparse and delayed. This leads to complex sources of uncertainty across different trajectories: failed trajectories may have high entropy due to model confusion, and successful trajectories may also maintain a certain entropy value because multiple correct solutions exist.

Experimental Findings: Failed Trajectories Actually Have Higher Entropy

The authors conducted a controlled sweep across two model scales and six RLVR objectives. One counterintuitive result is that token entropy on trajectories where the verifier fails is higher than on successful trajectories. This conclusion holds under both broad grouped aggregate estimates and paired analysis on common problems, with the latter's confidence intervals excluding the equality hypothesis in 10 of the 12 endpoints shown. In other words, failed samples often carry higher uncertainty, and global entropy is inflated by failed trajectories, thereby masking the true diversity changes within successful solutions.

Another finding comes from the local covariance approximation: it can predict the direction of entropy change between adjacent checkpoints, with an accuracy of 95.5% on non-failure transitions. The reward-stratified endpoint proxy further shows that positive chosen-token covariance is more concentrated in the tail within failure prefixes. This provides a mechanistic clue that "entropy change is driven by local covariance."

Local Covariance Approximation is a method for analyzing the drivers of entropy change. It approximates the instantaneous direction of entropy change by computing the covariance of policy distribution changes between adjacent checkpoints. Specifically, the change in entropy can be decomposed into a covariance term between the policy update and the current policy distribution; when this covariance is positive, entropy tends to increase, and vice versa. The paper uses this approximation to predict the direction of entropy change between adjacent checkpoints, achieving 95.5% accuracy on non-failure transitions, indicating that local entropy changes are mainly determined by the covariance structure of policy updates rather than random fluctuations in the global average. This provides a microscopic perspective for understanding the mechanism of entropy collapse: the decline in entropy may stem from systematically negative covariance at certain token positions rather than a uniform reduction in overall exploration.

Correct-Solution Mode Shrinkage: A More Concerning Collapse Signal

The paper audited all reportable trained endpoints and found that the breadth of correct-solution modes showed some degree of shrinkage in all cases. In other words, the model is indeed losing the diversity of correct solutions during training, but this shrinkage is not synchronized with the decline in global entropy. The authors cautiously note that the positive association between correct-solution mode breadth and large-k coverage remains exploratory and requires further validation.

This observation directly challenges the common practice of "judging collapse by looking at the entropy curve." If correct-solution modes are shrinking while global entropy remains flat due to the high entropy of failed trajectories, then entropy values alone will miss the true degradation signal.

Correct-Solution Modes refer to the set of all different correct solutions that the model can generate for a given problem. In RLVR, a math problem may have multiple correct solutions (such as different proof paths or calculation orders), and these solutions correspond to different modes in the policy distribution. The paper estimates the breadth of correct-solution modes (i.e., the number of modes or coverage) through clustering or diversity measures. Mode shrinkage means that the model gradually loses some correct solutions during training, retaining only a few or even a single mode, which is usually regarded as an early signal of policy collapse. Unlike global entropy, the breadth of correct-solution modes directly reflects the health of the model's output diversity and is not disturbed by the high entropy of failed trajectories.

Diagnostic Implications: Focus on Which Uncertainty Is Removed

The practical implication of RCEA is that entropy collapse should not be diagnosed by global entropy alone; instead, one should ask "which type of uncertainty has been removed." Is it the noise of failed trajectories being eliminated, or the strategy diversity within correct solutions being compressed? The former may be healthy learning, while the latter is the collapse to watch out for.

The hierarchical framework proposed in the paper places trajectory-level entropy, token-level entropy, verifier-conditioned uncertainty, and correct-solution mode retention in the same diagnostic table and establishes their connection to local covariance updates. For RLVR tuning and monitoring, this means more fine-grained metrics are needed instead of a single entropy curve.

Limitations and Open Questions

The authors explicitly label the association between correct-solution mode breadth and large-k coverage as exploratory; the sample covers two model scales and six objectives, and the generality of the conclusions still requires broader validation. The local covariance approximation performs well on non-failure transitions, but behavior on failure transitions has not yet been supported by equally strong evidence. In addition, as a diagnostic framework, RCEA does not directly provide a new training objective; how to translate hierarchical entropy signals into optimization constraints remains an open question.

Overall, this work decomposes "entropy" from a broad monitoring metric into attributable diagnostic quantities, reminding researchers to interpret entropy trajectories cautiously in RLVR.

One-line Summary

A new paper proposes the RCEA framework, decomposing the mixed uncertainty of global entropy in RLVR and pointing out that higher entropy in failed trajectories and correct-solution mode shrinkage are the key collapse signals.

Paragraph Summary

In Reinforcement Learning with Verifiable Rewards (RLVR) training, global token entropy is often used to diagnose exploration and collapse, but it lacks a stable relationship with reasoning performance. The paper points out that global entropy mixes the uncertainty of failed trajectories, the diversity of correct solution strategies, and sampling variation, so high entropy does not necessarily mean sufficient exploration. To address this, the authors propose the Reward-Conditioned Entropy Allocation (RCEA) hierarchical diagnostic framework, which separates trajectory entropy from token entropy and conditions on verifier results. Experiments find that token entropy on verifier-failed trajectories is actually higher than on successful trajectories, with global entropy inflated by failed samples; meanwhile, the breadth of correct-solution modes shrinks at all training endpoints, but not in sync with the decline in global entropy. The local covariance approximation can predict the direction of entropy change, with 95.5% accuracy on non-failure transitions. The paper concludes that entropy collapse should be diagnosed by "which type of uncertainty has been removed" rather than relying on a single global entropy metric.