#GRPO
3 related articles
GRPO Value Function Approximation: Can It Deviate from the Mean?
Reddit discussion on whether value function approximation in GRPO can deviate from the group mean, exploring data scale, task type, and model size effects on baseline choice, highlighting literature gaps and potential experiments.
Perplexity and NVIDIA Launch JOLT: A New Teacher-Student Joint Reinforcement Learning Method
Perplexity and NVIDIA jointly propose JOLT post-training method, letting a single model serve as both teacher and student, achieving GRPO-equivalent accuracy on MATH with 6.5x fewer training tokens. This article explains its teacher-student joint RL mechanism, token efficiency implications, and open questions.
Why GRPO Can Train Reasoning LLMs But Can't Teach a Robot to Walk
GRPO made DeepSeek reasoning model training cheaper, but why can't it be used for robot control? A Reddit discussion points to two reasons: removing the critic brings back the credit assignment problem, and LLM-specific conditions like terminal rewards, replayable environments, and pretraining priors don't hold in control tasks. De Oliveira et al.'s controlled experiments show PPO wins across five classic control tasks.