Unverified50% confidenceFactExact time
In RLHF or online RL (such as PPO, GRPO) training, generation in the rollout phase is essentially a large-scale inference task, with requests being bursty and batched, sensitive to overall throughput but not to single-request latency.
1
Sources
50%
Confidence
Long-term
Relevance
10/29/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedEvery step of RL training requires complete inference sampling, reward computation, and backpropagation, with rollout being the most time-consuming part.72% similarUnverifiedRL training often consumes enormous computational costs, and suspending training represents real costs and schedule losses.72% similarUnverifiedThe Max variant uses the Group Relative Policy Optimization (GRPO) training objective to mitigate the reward instability commonly seen in traditional RLHF.71% similarVerifiedTraditional RLHF requires first training a reward model and then optimizing with reinforcement learning algorithms such as PPO, making the process complex and unstable.71% similarUnverifiedStandard PPO requires training an additional value network (critic) of the same scale as the policy model to estimate the baseline, whereas the cost of GRPO is that multiple samples are needed per prompt, increasing sampling cost.69% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1110541API
curl https://kongchang.com/api/v1/knowledge/claims/1110541MCP
get_claim(id=1110541)