Unverified50% confidenceFactExact time
In PPO and GRPO algorithms, the importance sampling correction weight is proportional to the ratio of new and old policy probabilities exp(logprob_new - logprob_old); due to the amplification of the exponential function, a logprob discrepancy of 0.001 can produce a substantial weight error.
1
Sources
50%
Confidence
Long-term
Relevance
10/4/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedPPO等主流RL算法依赖rollout阶段采集的log概率来计算重要性采样比率，若推理与训练引擎的log-prob存在系统性偏差会导致策略梯度方差增大甚至训练发散75% similarVerifiedPPO通过引入裁剪目标函数将新旧策略概率比值限制在[1-ε, 1+ε]区间，替代TRPO的KL散度约束，简化了实现复杂度67% similarUnverifiedGRPO (Group Relative Policy Optimization) estimates the advantage function through intra-group relative rewards, eliminating the need for a separate value network (Critic) as in traditional PPO, thereby reducing memory overhead and implementation complexity.67% similarUnverifiedThe PPO algorithm requires computing the ratio of old policy probabilities to new policy probabilities (the importance ratio), and divergence between the two engines contaminates this ratio.66% similarUnverifiedError dependence is stronger among high-accuracy frontier model reviewers, even when those reviewers come from different vendors.62% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/972290API
curl https://kongchang.com/api/v1/knowledge/claims/972290MCP
get_claim(id=972290)