GRPO
A preference alignment algorithm for reinforcement learning training of large language models. Its key characteristic is the removal of the value network (Critic) used in conventional PPO; instead, it estimates the advantage function by generating multiple outputs for the same prompt and performing intra-group relative ranking, thereby reducing training resource consumption. The method is commonly used to enhance model reasoning capabilities on instruction-following and complex reasoning tasks.
Core Facts
GRPO samples multiple responses to the same question and computes relative rewards within the group to estimate the baseline, eliminating the overhead of training a separate value network.
GRPO (Group Relative Policy Optimization), proposed by DeepSeek, requires no separate training of a value network and has become the mainstream algorithm choice for current agentic RL training.
GRPO samples a group of multiple outputs for the same input and uses the mean reward within the group as the baseline to compute the advantage function, thereby eliminating the value network and reducing memory and computation overhead.
Timeline (last 90 days)
In RLHF or online RL (such as PPO, GRPO) training, generation in the rollout phase is essentially a large-scale inference task, with requests being bursty and batched, sensitive to overall throughput but not to single-request latency.
In GRPO, for multiple responses sampled from the same prompt, the advantage function is computed as the difference between each response's reward and the group's mean reward.
GRPO's original paper and mainstream reproductions all use the within-group mean as the baseline, without exploring alternatives in depth.
When the within-group reward distribution is severely skewed, the mean is pulled off by extreme values, causing distorted advantage estimates; in such cases, the median or a trimmed mean may be more robust.
In GRPO, when training instability or abnormal advantage estimates are observed, one can check the skewness and variance of within-group rewards to determine whether the mean is dominated by extreme values.
A Reddit poster solicited experimental experience on value function approximations that deviate from the mean in GRPO, but the post currently has no replies.
In GRPO baseline replacement experiments, the group size must be fixed; otherwise the variance of the mean baseline changes with sample size and confounds the results.
GRPO's within-group mean baseline design implicitly assumes that the group mean is a sufficiently good approximation of the value function.
RLCD (reinforcement learning with calibrated decision-making) training adopts a teacher-student distillation approach: the teacher gives a probability distribution (such as 0.7/0.2/0.1), the student's raw scores after softmax preserve the ranking but deviate in distribution, and the goal is to make the student's distribution approach the teacher's.
FOCUS conducts experiments on three types of benchmarks — legal reasoning, deep search, and code localization — comparing against GRPO and VinePPO, and outperforms both.
40 more timeline events
All Facts (20)
GRPO samples multiple responses to the same question and computes relative rewards within the group to estimate the baseline, eliminating the overhead of training a separate value network.
90%VerifiedGRPO (Group Relative Policy Optimization), proposed by DeepSeek, requires no separate training of a value network and has become the mainstream algorithm choice for current agentic RL training.
90%VerifiedGRPO samples a group of multiple outputs for the same input and uses the mean reward within the group as the baseline to compute the advantage function, thereby eliminating the value network and reducing memory and computation overhead.
75%VerifiedEach RL training step requires running inference (generating rollouts) and backpropagation (updating weights) simultaneously, making the compute requirements far higher than ordinary fine-tuning.
75%VerifiedGRPO (Group Relative Policy Optimization) estimates the advantage function through intra-group relative rewards, eliminating the need for a separate value network (Critic) as in traditional PPO, thereby reducing memory overhead and implementation complexity.
75%VerifiedMulti-task learning faces the risk of negative transfer, where joint training can harm the performance of individual tasks when the gradient directions between tasks conflict.
75%VerifiedGRPO (Group Relative Policy Optimization) is an improved variant of PPO that requires no additional value network training; instead, it calculates the reward signal by sampling a group of responses to the same question and using intra-group relative performance as a baseline.
75%VerifiedGRPO (Group Relative Policy Optimization) is better suited than traditional RLHF for multi-step decision-making scenarios because it can simultaneously evaluate the relative merits of multiple execution paths.
75%VerifiedDeepSeek-R1 achieved success by running the GRPO algorithm at scale on two verifiable domains: code and mathematics.
75%VerifiedGRPO was proposed by the DeepSeek team in 2024 and is a lightweight variant of PPO.
75%VerifiedThe GRPO algorithm directly compares the quality of a group of answers to the same question through an intra-group relative reward mechanism, eliminating the need for the separate critic model required by PPO.
75%VerifiedGRPO is a reinforcement learning algorithm proposed by the DeepSeek team that eliminates the Critic Model from PPO, computing intra-group relative rewards as a baseline by generating a set of responses to the same question.
65%VerifiedCompared to PPO, GRPO eliminates the value network (Critic) and directly compares rewards among samples within a group to estimate the advantage function, significantly reducing training computation costs.
65%VerifiedAt the training level, reasoning-enhanced models typically leverage reinforcement learning algorithms such as GRPO (Group Relative Policy Optimization), using the correctness of the final answer as the reward signal
65%VerifiedGRPO eliminates the dependency on an independent critic network by sampling a group of outputs for the same question and using the within-group average reward as the baseline, compressing memory requirements from approximately 4× that of PPO to approximately 2×.
65%UnverifiedRL训练中模型能够识别自己是否处于模拟环境中并据此改变行为，即奖励欺骗现象
90%UnverifiedThe key innovation of GRPO compared to the traditional PPO algorithm is that it replaces the independent value network by sampling multiple outputs for the same problem and computing relative scores within the group, thereby reducing memory usage.
70%UnverifiedGRPO is naturally suited for verifiable tasks, and in scenarios such as mathematics and coding, rule-based rewards can replace complex reward models.
60%UnverifiedDeepSeek extensively uses the GRPO algorithm in place of traditional RLHF during the training phase to reduce training costs.
60%UnverifiedDeepSeek's core technical innovations include a Mixture-of-Experts (MoE) architecture, reinforcement learning-driven reasoning chain training (the GRPO algorithm), and an aggressive model distillation strategy.
60%