KAPPO: Optimizing the Sharpening Effect in LLM Post-Training with @k

When RL Makes LLMs "Narrow": What Is the Sharpening Effect

In large language model post-training, reinforcement learning (RL) has repeatedly been shown to improve task accuracy. But behind the accuracy gains lies an easily overlooked cost: reduced behavioral coverage. The model gets increasingly good at problems it already knows how to solve, yet becomes increasingly unwilling to explore new solution paths—a phenomenon researchers call "sharpening."

The danger of sharpening is that it quietly slows progress on hard tasks. When the model concentrates its learning budget on high-confidence solutions to easy problems, those rare correct paths that require diverse sampling get squeezed out of the probability space. Prior work has found that repeated-sampling objectives (such as pass@k and max@k) can mitigate sharpening, but the underlying mechanism has long lacked a clear explanation. More vexing still, existing @k optimization methods are all bespoke algorithms that operate in isolation—there is no general framework for transferring mature @1 policy optimization methods to the @k setting.

pass@k and max@k are two commonly used metrics in LLM evaluation and optimization. pass@k measures the probability that, given k independent samples for the same prompt, at least one is correct; max@k typically refers to the reward or score of the best sample among k samples. Both are objectives from a "repeated sampling" perspective, naturally encouraging the model to preserve diverse high-reward paths in its distribution. In contrast, the standard @1 objective (such as the common RLHF/PPO single-sample optimization) only cares about the expected reward of a single sample, and easily concentrates probability mass on a few high-confidence solutions, thereby compressing the exploration space. Understanding the difference between these two metrics is key to understanding why "sharpening" is related to the number of samples.

What the Closed-Form Solution Reveals: @k Is Really Redistributing Learning Pressure

The first step of this research is to derive the closed-form optimal policy for pass@k and max@k objectives under KL regularization. The insight from the explicit solution is quite direct: @k optimization is essentially redistributing learning pressure according to problem difficulty, shifting the optimization focus from easy problems to hard ones.

A more critical finding is that the standard @1 objective, through "prompt-adaptive regularization" or "reward transformation," can achieve exactly the same optimal policy as @k. In other words, the sharpening-mitigation effect of @k objectives does not depend on some special algorithm, but stems from the different learning pressures it applies to prompts of varying difficulty. This provides an optimizer-agnostic explanation for why @k can suppress sharpening on easy tasks while prioritizing success rates on hard tasks.

The significance of this conclusion is that it breaks down the barriers between algorithms—since the optimal policy can be reproduced by the @1 framework, transferring from @1 to @k has a theoretical basis, without the need to design new optimizers from scratch.

KL regularization refers to adding a KL divergence penalty term against a reference policy to the optimization objective, typically in the form -β·KL(π||π_ref), to constrain the new policy from deviating too far from the initial model. It is a common technique in RLHF for stabilizing training and preventing reward hacking. In this article's context, the strength of KL regularization β acts as a "trust region" parameter: the larger β is, the more conservative the policy updates. The so-called "prompt-adaptive regularization" means letting this β or the equivalent penalty strength vary with prompt difficulty—applying stronger constraints on easy prompts to suppress sharpening, and relaxing constraints on hard prompts to preserve exploration. The reason a closed-form solution can be derived is precisely that the optimal policy under KL regularization has an exponentially weighted form, allowing the learning pressure for prompts of different difficulties to be analytically expressed and compared.

KAPPO: Seamlessly Upgrading @1 Methods to @k

Based on the above theory, the researchers propose @k Adaptive Pressure for Policy Optimization (KAPPO). Its workflow is quite lightweight: first perform a one-time reference calibration, after which existing @1 methods can be adapted to @k through prompt-adaptive regularization or reward transformation, without changing other parts of the original training pipeline.

This "minimally invasive" design allows KAPPO to be applied to a variety of RL methods, with experiments covering both mathematical reasoning and code generation tasks. Results show that KAPPO provides direct control over the degree of sharpening—users can proactively adjust the distribution of learning pressure between easy and hard problems, rather than passively accepting the coverage loss caused by sharpening.

For engineering practice, another highlight of KAPPO is its cost. The research proves that @k pressure can be fully realized through single completion training, matching the performance of multi-completion @k optimizers at only a fraction of the cost. This means teams do not need to pay multiplied compute overhead for multiple sampling to obtain the learning pressure signal needed to control sharpening.

Reference calibration is a one-time preprocessing step in the KAPPO pipeline: before formal training, the reference model (usually an SFT model or initial policy) is used to estimate the baseline difficulty or baseline reward level of each prompt over the task distribution, thereby providing a normalized baseline for subsequent prompt-adaptive regularization. Without this step, reward scales across different prompts are not comparable, and "allocating learning pressure by difficulty" becomes impossible. Calibration only needs to be done once and can then be reused throughout the training process, which is one reason KAPPO remains low-overhead.

Single completion training means that at each policy update, only one completion is sampled per prompt (rather than the k completions typically required by @k optimizers), yet the learning pressure of the @k objective is simulated through reward transformation or regularization design. Its principle comes from the equivalence of the closed-form solution discussed earlier: since the optimal policy of @k can be reproduced by the @1 framework, the pressure signal does not need to actually come from multiple samples, but can be constructed from a single sample combined with an adjusted objective function. This directly determines KAPPO's computational advantage.

Implications for Post-Training Practice

The value of this work lies not only in a new method, but in turning "sharpening" from an empirical phenomenon into something analyzable and controllable. The closed-form solution tells us that the root cause of sharpening is an imbalance in the distribution of learning pressure across the difficulty dimension, and the control mechanism offered by KAPPO turns this distribution into an adjustable parameter.

For teams currently doing RL post-training, several directions are worth attention: first, re-examine the prompt-level regularization settings in existing @1 pipelines, as they may already be inadvertently affecting the degree of sharpening; second, treat "behavioral coverage" as a monitoring metric alongside accuracy, rather than waiting until hard tasks stall to investigate; third, prioritize verifying whether the single completion approach can reproduce @k effects on your own task distribution, thereby gaining control capability at lower cost. Whether theoretical elegance can translate into stable gains in large-scale training still requires validation across more task types and model scales.

One-line Summary

Research reveals that the sharpening effect in RL post-training stems from imbalanced difficulty-based allocation of learning pressure, and proposes KAPPO to achieve @k control at low cost within the @1 framework.

Paragraph Summary

The article discusses the "sharpening" phenomenon in RL post-training of large language models: while accuracy improves, behavioral coverage declines, and the model becomes increasingly unwilling to explore new solutions, potentially slowing progress on hard tasks. By deriving closed-form solutions for pass@k and max@k objectives under KL regularization, the researchers find that @k optimization is essentially redistributing learning pressure by prompt difficulty, shifting focus from easy to hard problems; moreover, the standard @1 objective can achieve the same optimal policy as @k through prompt-adaptive regularization or reward transformation. Based on this, they propose KAPPO, which uses one-time reference calibration to seamlessly adapt existing @1 methods to @k without changing the training pipeline, and prove that @k pressure can be realized through single completion training, matching multi-completion optimizers in performance while greatly reducing cost. The article concludes with recommendations for post-training practice: monitoring coverage, reusing @1 pipelines, and prioritizing validation of the single completion approach.