Distill First, Then Fine-Tune: Get-Up Controller Success Rate Jumps from 71% to 94%

Why Training a Single Network from Scratch Failed

The reinforcement learning community often discusses the ideal of "end-to-end training," but in practice, directly training a large network from scratch to solve multi-stage tasks is often infeasible. A developer shared on Reddit his experience training a 2D ragdoll get-up controller with MuJoCo: a single policy never truly completed the get-up—0 successful get-ups out of 160 tests, and the learned policy even raised its head less frequently than a random bang-bang action.

This result is not surprising. The get-up task involves multiple physical phases: starting from a supine or prone position, rolling over, hand-off support, and finally standing and maintaining stability. The optimal action distributions for each phase differ greatly, and a single reward function can hardly guide all phases simultaneously. The developer's attempts confirmed this: training a large network from scratch repeatedly failed on the floor task.

Breaking Down the Task: Four Policies Plus a Rule-Based Controller

Faced with the failure of end-to-end training, he chose an engineering decomposition approach: splitting the get-up task into four independent policies, each with its own reward function and curriculum design—supine floor, prone floor, hand-off, and standing. Then he used a rule-based controller to chain them together, handling the switching timing and conditions between policies.

This combined approach gradually raised the success rate to about 71%, but then stagnated for a long time. Reward fine-tuning, retraining the standing policy, adjusting the hand-off timing gating—all attempts fell within the noise range and brought no substantial breakthrough. This is a typical "modular system bottleneck": each sub-policy works individually, but the switching logic of the rule-based controller becomes the performance ceiling.

DAgger (Dataset Aggregation) is an imitation learning algorithm used to distill the behavior of an expert policy (here, the rule-based controller) into a neural network. Its core idea is: during training, let the currently learned policy interact with the environment to generate new state distributions, then have the expert provide action labels on these states, and add these new samples to the dataset for retraining. Compared to simple behavior cloning, DAgger alleviates the "covariate shift" problem—where the learned policy deviates from the expert trajectory and encounters unseen states, leading to error accumulation. The specific process is: initialize the policy network, run the current policy to collect trajectories, label actions on these trajectories with the expert, aggregate data and retrain the policy, iterating multiple times until convergence. In this article, the developer used DAgger to compress the entire controller (including switching rules) into a 512x512 network, enabling the network to not only imitate sub-policy actions but also learn when to switch policies, providing a high-quality initial policy for subsequent PPO fine-tuning.

Distillation Plus Fine-Tuning: Compressing the Entire Controller into One Network

The turning point came from an unusual idea: instead of directly training an end-to-end policy, first use DAgger to distill the entire rule-based controller (including switching rules) into a 512x512 network. The distilled network achieved 74% at the same level as the rules, essentially replicating the rule-based controller's behavior.

The real leap came from the next step: using PPO to fine-tune this network end-to-end on a 20-second "standing" reward. The success rate jumped from 71% to 94.2%, with 97.7% in the supine scenario and 90.7% in the prone scenario, tested with 600 fresh starting states per side; the rule-based controller achieved only 70.8% on the same random seeds.

A detail worth mentioning is the network size experiment: even a small 64x64 network reached 85%, and 256x256 already matched the 512x512 performance. This shows that the benefits of distillation plus fine-tuning do not depend on model capacity, but come from the training path of "first learn rule behavior, then optimize globally."

PPO (Proximal Policy Optimization) is a policy gradient-based reinforcement learning algorithm that has become a mainstream choice for continuous control tasks due to its stability and ease of use. PPO avoids training collapse by limiting the magnitude of each policy update: it maximizes a clipped surrogate objective function that constrains the probability ratio between the new and old policies to the interval [1-ε, 1+ε] (typically ε=0.2), thereby preventing excessively large policy changes. Meanwhile, PPO uses generalized advantage estimation (GAE) to compute the advantage function, combining value function loss and entropy bonus to balance exploration and exploitation. In this article, the developer used the distilled network as the initial policy and fine-tuned it end-to-end with PPO on a 20-second "standing" reward. Since the initial policy already possesses reasonable get-up behavior, PPO does not need to start from random exploration, avoiding the exploration challenge under sparse rewards, and thus quickly raising the success rate from 71% to 94.2%.

Anti-Cheating Checks: The Results Are Not an Illusion

After the significant improvement in success rate, the developer proactively conducted a series of shortcut investigations. The unphysical root force actuator was never activated; moving the opponent's body away had no effect on the results; applying 150-500 N pushes after getting up did not change performance; the standing state remained stable with a torso height standard deviation of about 1 cm. These checks ruled out the possibility of the policy exploiting simulation loopholes or external assistance.

However, he also clearly listed the limitations: validated only in simulation, 2D scenarios, only one training run per configuration, starting states from the environment's built-in lying pose generator, and other poses not yet tested. These honest boundary statements make the conclusions more credible and remind readers not to over-generalize.

Shortcut Learning refers to the phenomenon where reinforcement learning agents exploit loopholes in the physics engine, flaws in the reward function design, or spurious correlations in the environment to achieve goals in unintended ways. Common forms include: applying external forces through non-physicalized actuators, obtaining high rewards by jittering or exploiting bugs, relying on specific initial states rather than truly learning the task, etc. In this article, the developer ruled out shortcuts through multiple checks: confirming the root force actuator was never activated (preventing illegal external forces), moving the opponent's body away did not affect results (ruling out dependence on external objects), remaining stable after applying 150-500 N pushes (verifying robustness), and torso height standard deviation of about 1 cm during standing (confirming posture stability). These measures ensure that the success rate improvement stems from the policy truly learning to get up, rather than exploiting simulation flaws.

Where Can This Method Be Reused

At the end of the post, the developer raised an open question: whether anyone has seen the same "break down first, then distill, finally end-to-end finish" pattern succeed or fail in other multi-stage tasks. From an engineering perspective, the value of this pattern lies in bypassing the most difficult part of end-to-end reinforcement learning—exploration. The rule-based controller provides a reliable initial behavior distribution, distillation transfers this distribution to a single network, and PPO fine-tuning performs global optimization on top of it.

For tasks such as robot manipulation, multi-stage navigation, and complex assembly, this approach may be more practical than pure end-to-end training. But its prerequisite is that you can first design a working rule system—if you can't even build a rule-based solution, distillation is out of the question. The code, documentation, and all failed experiments are open-sourced on GitHub and Hugging Face, and the developer also provided a demo script that runs in about 1.5 minutes for the community to reproduce and verify.

One-line Summary

Break down the task to train sub-policies, then distill the entire controller with DAgger, and finally fine-tune with PPO, raising the success rate from 71% to 94.2%.

Paragraph Summary

This article shares training experience for a 2D ragdoll get-up controller. Direct end-to-end training of a single policy completely failed (0 successes out of 160 tests), so the developer split the task into four sub-policies (supine, prone, hand-off, standing) and chained them with a rule-based controller, reaching about 71% before stagnating. The key breakthrough: first use DAgger to distill the entire rule-based controller (including switching logic) into a 512x512 network, then use PPO to fine-tune end-to-end on the standing reward, jumping the success rate to 94.2% (97.7% supine, 90.7% prone). Experiments show that even a 64x64 small network reaches 85%, indicating the benefit does not depend on model capacity. The developer ruled out simulation loopholes through multiple anti-cheating checks and noted limitations such as simulation-only 2D scenarios and single training runs. This method provides a practical paradigm for multi-stage tasks that bypasses the exploration challenge.