Deep Reinforcement Learning

Deep reinforcement learning requires stabilization techniques such as experience replay and target networks to cope with the instability and divergence risks brought by function approximation.

Function approximation is the key concept bridging tabular reinforcement learning and deep reinforcement learning: when the state space is continuous or enormous, a parameterized function (such as a linear model or a neural network) is used to approximate the value function or policy.

Deep reinforcement learning combines deep learning and reinforcement learning, enabling robots to learn control policies directly from camera images.

Deep reinforcement learning lets a network learn expected returns rather than a single deterministic trajectory through interaction with the environment, making it suitable for scenarios with uncertain action outcomes.

Deep reinforcement learning uses neural networks to approximate policies or value functions, requiring only a single forward computation at inference time and responding quickly, but it requires a large amount of gameplay for training, and the training environment must be able to efficiently simulate sand-grain physics.

Deep reinforcement learning policies perform well in distribution, but neural networks have limited extrapolation capability when faced with OOD inputs, and easily output high-confidence yet erroneous predictions, leading to dangerous decisions.

Deep reinforcement learning cannot provide L-infinity (worst-case) guarantees because everything is based on sampling, and sampling is inherently incompatible with absolute maximum error.

The breakthrough in Atari deep reinforcement learning laid the methodological foundation for subsequent projects such as AlphaGo and AlphaStar.

DeepMind's approach trains a single neural network policy that directly maps plasma states to control commands for all magnetic field coils, enabling stable maintenance of plasma configurations previously difficult to achieve, such as negative triangularity cross-sections and snowflake divertors.

Reinforcement learning trains an agent to learn an optimal policy through repeated interaction with an environment guided by reward signals; Deep RL uses deep neural networks as policy approximators, enabling it to handle high-dimensional continuous state-action spaces and make control decisions on a microsecond timescale.

深度RL策略目前缺乏像李雅普诺夫稳定性分析那样为系统提供严格数学保证的理论工具

黑箱式神经网络策略难以提供稳定性保证，深度RL策略目前缺乏李雅普诺夫稳定性分析等理论工具

已有多个开源项目使用深度强化学习训练的 AI 代理在 Flappy Bird 上达到永不死亡的水平

The developer miwate used an agent trained with deep reinforcement learning to reach 63% of the height from the bottom in the game Getting Over It with Bennett Foddy, without using checkpoints.