Imitation Learning

Behavior cloning transforms policy learning into a supervised learning problem, using environment states as inputs and expert actions as labels to train a mapping function from states to actions.

Behavioral cloning suffers from the distribution shift problem, where the model tends to make errors and accumulate mistakes on states outside the training distribution, a phenomenon referred to as compounding error in the robotics control field.

模仿学习的核心思想是让机器人通过观察专家演示来习得行为策略，而非依赖大量试错，主要技术路线包括行为克隆（BC）和逆强化学习（IRL）

Behavioral cloning is the simplest form of imitation learning, directly learning a mapping from states to actions rather than inferring the expert's reward function as inverse reinforcement learning (IRL) does.

Modern AI robots typically rely on reinforcement learning to train motion control policies.

The agent was trained entirely on reinforcement learning, without using any human data and without involving imitation learning.

For laboratory scenarios, the quality of demonstration data, the representation of the action space, and the alignment between vision and proprioception are the keys that determine whether imitation learning can be deployed.

Imitation learning is a branch of robot learning whose core idea is to let robots directly learn policies from expert demonstrations, rather than relying on hand-written control programs or pure trial-and-error reinforcement learning.

Jia Xiaosong believes that if training remains dominated by imitation learning, changing the architecture cannot bypass the inherent problems of imitation learning; valuable directions include filtering valuable data, going beyond brute-force imitation learning, designing more effective reward signals, and enabling models to reinforcement-learn tens of thousands of skills simultaneously.

Jia Xiaosong believes imitation learning can take a model from 0 points to 90 points and reinforcement learning can tackle the remaining 10 points, and the two are not substitutes for each other; reinforcement learning starting completely from scratch converges extremely slowly.

This kinematic prior appears under both reinforcement learning and imitation learning training paradigms, possibly reflecting common knowledge related to the humanoid embodiment rather than a product specific to a particular optimization process.

Compared with traditional reinforcement learning, imitation learning typically requires only tens to hundreds of demonstration trajectories for a robotic arm to learn grasping and placing, whereas reinforcement learning typically requires defining a reward function in a simulation environment and having the agent repeatedly trial and error, with sample sizes often in the millions.

Imitation Learning is a class of methods that lets robots learn policies directly from human demonstrations, mapping observations to actions through behavior cloning or diffusion policies, and its core advantage is high sample efficiency, but its out-of-distribution generalization ability is weak.

In robot demonstration collection, incorrect hand reconstruction will directly mislead the downstream imitation learning policy, causing the robot to learn actions that humans never performed, and errors in this kind of data itself will not be clearly exposed in the training loss.

Most existing methods for learning dexterous manipulation from video remain at the imitation level, strictly matching the action trajectories in the video, which leads to rapid failure when faced with new object initial poses, target poses, or grasp modes.

On the question of how humans align children, imitation learning may be more fundamental than reinforcement learning.

In imitation learning, data quality and demonstration consistency directly determine the performance of the policy.

FireFly's current training pipeline is primarily based on imitation learning/behavior cloning, training policies by recording screen observations and player actions.

The combination of imitation learning and reinforcement learning has transformed control policy development from writing rules one by one to learning from demonstrations.

Grok Bots' demonstration learning is technically close to a combination of program synthesis and imitation learning, automatically inferring reproducible operation scripts or tool-call chains by analyzing UI operation sequences from screen recordings.