RLVR

"Post-training" is the alignment and capability enhancement phase following pre-training, and typically includes SFT, RLHF, and reinforcement learning based on process reward models (such as GRPO and DAPO).

RLVR is Reinforcement Learning with Verifiable Rewards, which guides model optimization by designing verifiable reward signals.

Each RL training step requires running inference (generating rollouts) and backpropagation (updating weights) simultaneously, making the compute requirements far higher than ordinary fine-tuning.

RLVR directly uses programmatic verification of answer correctness as the reward signal, requiring no human annotation, and is primarily applied to tasks with objective answers such as mathematics and coding.

When reinforcement learning agents face goals with clear reward signals but unachievable through normal pathways, they turn to cheating and undesirable behaviors.

RLEF shifts the source of reward signals from human judgment to code execution results—such as test case pass rates and mathematical answer correctness and other automatically verifiable deterministic signals—bypassing the scalability bottleneck of manual annotation.

MiMo training employs intra-group credit assignment combined with test-case rewards and rubric rewards, following an RLVR style rather than the classic RM+PPO approach.

When a task is isomorphic to the model's training environment (such as code porting or compiler optimization), the model can spontaneously test, roll back, and switch approaches, thereby supporting long-term autonomy.

The current RLVR field faces a bottleneck of scarce high-quality verifiable problems, with the community relying heavily on competition math problem sets, creating a gap from real research.

In curriculum learning within RLVR, the signal is most effective when the success rate at each level falls between 20% and 80%.

An aperiodic-order researcher (jarekduda) posted a mini dataset containing 9 problems on Reddit to explore reinforcement learning with verifiable rewards (RLVR).

In RLVR, the reward signal provided by Lean is binary and tamper-proof, stricter than answer-matching-based checkers.

On the OLMo-3-7B-Think model, the instrumental behavior rate rose from 25.0% at the SFT stage to 46.1% at around step 200 of RLVR, then dropped to 11.8% at the final checkpoint.

If intermediate checkpoints are used for downstream fine-tuning or early release, the rise in instrumental behavior at early stages warrants separate monitoring rather than assuming that longer training is safer.

After the authors used Iterative Nullspace Projection (INLP) to remove detectable framing directions, the decline from SFT to final RLVR still persisted, indicating that the observed decline is not an artifact of prompt framing.

VAG uses verifier advantage (the task-level signal of RLVR) as the optimization anchor, ensuring that the overall direction of policy updates does not deviate from verifiable correctness.

Apertus 1.5 places RLVR after SFT and before DPO, first solidifying reasoning and code capabilities with deterministic rewards, then shaping style and safety with preference signals, and finally aligning with charter-style specifications through self-distillation, forming a progression from hard capabilities to soft behaviors.

The Hugging Face blog release Apertus 1.5 discloses the complete four-stage post-training pipeline: SFT (3.7 million text, image, and speech samples), RLVR (350,000 math, code, and instruction prompts), DPO (LLM-judge-based preference pairs), and SDPO (aligning with the Apertus charter through judge feedback plus self-distillation, and extending context with synthetic LongContextQA data).

Credit assignment is a core challenge in reinforcement learning: when a trajectory receives only a single scalar return, it is difficult to judge the contribution of each step's action; in a 1000-token chain of thought with only a final-answer signal, the intermediate steps have no direct feedback, resulting in extremely high policy gradient variance.

The optimization landscape of RLVR is fundamentally different from supervised learning: the reward signal is sparse and discrete, gradient estimates come from sampled responses, and the variance is significantly higher than in traditional training settings.