GRPO Value Function Approximation: Can It Deviate from the Mean?

Core Question: In GRPO (Group Relative Policy Optimization), value function approximation is typically simplified to the group mean—but has anyone tried a more sophisticated approximation? A Reddit user posted asking for experimental experience, hoping to understand data scale, task type, and model size.

GRPO's Default Design: Mean as Baseline

GRPO is a reinforcement learning algorithm proposed by DeepSeek^Ad for preference alignment of large language models. Its core innovation is replacing the value network in traditional PPO with the mean of within-group rewards, thereby eliminating the overhead of training a separate critic model. Specifically, for multiple responses sampled from the same prompt, GRPO computes the difference between each response's reward and the group's average reward as the advantage function, then updates the policy accordingly.

This design substantially reduces computational and memory costs, but it introduces an implicit assumption: the group mean is a sufficiently good value function approximation. The Reddit poster astutely points out that this assumption deserves scrutiny—what happens if the value function approximation is not the mean?

Advantage Function is a core concept in reinforcement learning, representing the expected return increment of taking a given action in a given state relative to a baseline. In policy gradient methods, accurate estimation of the advantage function directly determines the direction and magnitude of policy updates. Traditional Actor-Critic architectures use an independent value network (critic) to estimate state values as the baseline for computing advantages. GRPO's innovation lies in replacing this learned baseline with the within-group reward mean—a "critic-free" advantage estimation scheme. The theoretical justification is that for multiple sampled responses to the same prompt, the group mean provides an unbiased baseline estimate without requiring additional value network training. However, the variance and bias of the advantage function directly affect the quality of policy gradients, so the choice of baseline is not an inconsequential engineering detail but a key design decision affecting training stability and final performance.

Why Would Anyone Want to Deviate from the Mean?

Using the mean as a baseline has its rationale: it is unbiased and simple to implement. But in certain scenarios, the mean may not be the optimal choice.

• Skewed reward distributions: When within-group rewards are severely skewed (e.g., a few extremely high scores and many low scores), the mean gets pulled by outliers, distorting advantage estimates. In such cases, the median or trimmed mean may be more robust.
• Task heterogeneity: Different prompts vary greatly in difficulty; fixed use of the group mean may ignore prompt-level prior information.
• Variance control: The variance of the mean baseline can be large when the group sample size is small; introducing more sophisticated approximations (e.g., weighted mean, learned baseline) might improve training stability.

The poster does not propose a specific solution but hopes to find people who have run experiments to exchange notes on the impact of data scale and model size. This indicates that the direction currently lacks publicly available systematic research.

Gaps in Existing Literature

According to public information, the original GRPO paper and mainstream reproductions (such as work related to DeepSeek-R1) all use the group mean as the baseline, without deeply exploring alternatives. Community discussions occasionally mention "whether GRPO's lack of a critic affects fine-grained credit assignment," but controlled experiments are absent.

The value of the Reddit post lies in posing a testable question: Does the form of value function approximation significantly affect final performance? If so, at what data scale and model size does this effect become important?

Credit Assignment is a classic challenge in reinforcement learning, referring to how to reasonably attribute the final reward to individual decision steps in a sequence. In reinforcement learning fine-tuning of LLMs, this problem is particularly acute: a complete response receives a high score, but which specific tokens or reasoning steps contributed to that score? PPO trains a critic network to estimate the value of each intermediate state, thereby supporting finer-grained credit assignment. After GRPO replaces the critic with the group mean, the advantage function is identical across the entire response sequence (i.e., every token shares the same advantage value), which amounts to abandoning token-level credit assignment and retaining only sequence-level signals. This simplification may not be problematic for short responses or sparse reward signals, but in long-chain reasoning tasks, whether it leads to decreased learning efficiency or degraded local behavior currently lacks systematic study. This is also the technical root of the community discussion question "whether GRPO's lack of a critic affects fine-grained credit assignment."

Potential Experimental Directions

Based on the questions raised in the post, several experimental dimensions can be envisioned:

1. Baseline replacement: Replace the ordinary mean with median, trimmed mean, or exponentially weighted mean, and compare training curves and final rewards on the same tasks.
2. Data scale sweep: From small scale (thousands of prompts) to large scale (hundreds of thousands of prompts), observe the impact of different baselines on data efficiency.
3. Model size sweep: From small-to-medium models (e.g., 1B–7B) to larger models, test whether value function approximation error is amplified or mitigated as model capacity changes.
4. Task types: Different tasks such as mathematical reasoning, code generation, and dialogue may have different sensitivities to baseline choice.

The poster emphasizes "exchange notes," suggesting that such experiments may have been conducted internally at some teams but not publicly disclosed.

Ablation Study is a standard method in machine learning for validating the necessity of a component or design choice: keeping all other conditions constant, changing only the factor under study and observing performance changes. The experimental dimensions suggested in this article are essentially a series of ablation studies. It should be noted that GRPO baseline replacement experiments have several key control variables: group size must be fixed, otherwise the variance of the mean baseline will vary with sample size and confound results; reward normalization methods (e.g., whether z-score standardization is applied) must be consistent; and training steps and learning rate schedules should also be uniform. Additionally, because RL training itself has high variance, conclusions from a single experiment may be unreliable; multi-seed repeated experiments with confidence intervals are needed. These methodological details are often overlooked in community sharing but are key factors affecting the credibility of conclusions.

Implications for Practitioners

If you are using GRPO to train models, the group mean baseline is a reasonable default choice, but it is worth monitoring the within-group reward distribution. When training instability or anomalous advantage estimates are observed, you can try:

• Checking the skewness and variance of within-group rewards to determine whether the mean is dominated by extreme values;
• Testing median or trimmed mean in small-scale ablations;
• Recording gradient norms and policy entropy changes under different baselines.

The post currently has no replies, but the question itself points to a blind spot in GRPO implementation details. As GRPO is widely adopted in the open-source community, sharing of such "non-standard" experiments will become increasingly valuable.

---

This article is based on a Reddit community discussion. The original post did not provide experimental data, and related conclusions require further verification.

One-line Summary

A Reddit user seeks experimental experience with non-mean value function approximations in GRPO, pointing to a literature gap.

Paragraph Summary

The article centers on a Reddit community question: GRPO by default uses the within-group reward mean as a value function approximation to eliminate the critic network, but is this simplification always optimal? The author analyzes motivations for deviating from the mean—skewed reward distributions, task heterogeneity, and small-sample variance—and notes the lack of systematic controlled experiments in public literature. The article further proposes potential experimental directions, including replacing the baseline (median, trimmed mean, exponentially weighted mean), sweeping data scale and model size, and covering different task types, while offering practitioners concrete suggestions for monitoring reward distributions and conducting small-scale ablations. Overall, the discussion reveals a neglected design choice in GRPO implementation and calls on the community to share experiences with non-standard experiments.