Dreamer

Institutions such as DeepMind and Meta AI have produced representative world model works including DreamerV3 and JEPA

A world model is a type of model that learns the dynamic laws of an environment and can predict state evolution; typical examples include Ha & Schmidhuber's World Models, the Dreamer series, and generative environment models such as Genie and GameNGen.

Classic MBRL implementations (such as Dreamer and MuZero) typically require explicitly predicting the next frame observation or reward.

The appeal of latent world models lies in their high computational efficiency and suitability for long-horizon rollouts; the Dreamer series and methods based on the Joint Embedding Predictive Architecture (JEPA) both belong to this lineage.

World models advocate letting AI learn the laws of dynamic environmental evolution, with representative work including Dreamer, Genie, and others.

A world model is a general term for a class of methods in reinforcement learning and robotics, whose core idea is to have the agent learn an internal simulator, with typical representatives including Dreamer, PlaNet, and Genie based on video prediction.

The idea of the LFP head originates from the world model research tradition in self-supervised learning, such as the Dreamer series and JEPA (Joint Embedding Predictive Architecture).

In recent years, representative work on world models includes the Genie and Dreamer series and various video prediction foundation models.

Dreamer adopts the "KL balancing" technique, applying different gradient directions to the posterior and the prior respectively, so that the two move toward each other dynamically without collapsing.

Dreamer's policy learning adopts the Actor-Critic framework: the Actor outputs an action distribution based on the imagined latent state s_t, and the Critic estimates the value of that state; gradients are backpropagated through the world model's latent trajectory, but the world model parameters are usually frozen to avoid the policy optimization signal contaminating the world model's learning objective, thereby maintaining training stability.

The cost of freezing the world model training strategy is: if the world model itself has bias, the policy will overfit to that bias, i.e., the phenomenon of 'model exploitation' occurs.

In Dreamer reproduction practice, the stochastic component of the latent state requires appropriate KL constraints: too strong loses information, too weak leads to posterior collapse; the imagination trajectory length and batch size together determine gradient quality, and both need to be adjusted according to the task.

The Dreamer series adopts the approach of training policies within simulation, amortizing planning costs into the training phase.

The Dreamer series of work allows agents to learn policies entirely in latent space without rendering real pixels.

Representative early work on world models includes DeepMind's Dreamer series and the environment model used by OpenAI in Dota 2.

The Dreamer series (DreamerV1/V2/V3) pushed the world model framework toward more complex continuous control tasks by introducing discrete latent variables and improved world model loss functions.

The Dreamer series achieved performance comparable to or even better than model-free reinforcement learning on multiple Atari and 3D environment benchmarks.

A world model can be an explicit physics simulator (such as MuJoCo or Isaac Sim) or a learned implicit model (such as the recurrent state-space model used by the Dreamer series of algorithms).