AlphaZero

AlphaGo combines deep learning with Monte Carlo Tree Search (MCTS), using two key neural networks: a policy network and a value network.

DeepMind's AlphaZero was released in 2017 and, through self-play alone, achieved a level surpassing the world's top humans in chess, shogi, and Go.

DeepMind's AlphaGo learned Go strategies by playing millions of games against itself.

A world model refers to a robot's internal predictive capability of the physical environment—given the current state and an action, it can predict what will happen next.

AlphaProof is based on a combination of reinforcement learning and a formal proof system (the Lean proof assistant), exploring proof paths through algorithms such as Monte Carlo tree search.

The AlphaGo series improved its playing strength through self-play.

Some researchers mentioned that in recent years the success of combining MCTS with deep learning (such as AlphaZero) has partially obscured its theoretical flaws, but understanding its limitations is crucial for safety-critical applications.

AlphaZero further simplified this, reaching superhuman level using only self-play and MCTS, highlighting the generality of MCTS as a planning framework.

After the agent completed the map, the authors further improved the policy through an AlphaZero-style expert loop: combining search with distillation, distilling the optimal actions obtained from search into the policy network.

The AlphaZero-style expert loop refers to borrowing AlphaZero's training paradigm: first evaluating actions and selecting the optimal action through Monte Carlo tree search (MCTS) or a similar forward search, then distilling the (state, action) pairs produced by the search as supervision signals into the policy network, and this can be iterated.

In traditional AlphaZero implementations, both sides share the same network, because games such as chess are fully symmetric.

The "asymmetric" in asymmetric multi-head AlphaZero means that the two sides in a match have different action spaces, objectives, and even role definitions, breaking the symmetry assumption of classic AlphaZero.

AlphaZero's core technical combination includes MCTS for lookahead evaluation of moves, and a deep neural network that simultaneously outputs a policy head (prior probabilities for each action) and a value head (the expected win/loss of the current position).

In chess, the 'centaur' (human + AI) model was once regarded in the late 1990s to mid-2000s as the optimal solution for board game play, but with the emergence of self-learning systems such as AlphaZero, pure AI playing strength quickly opened a gap over any human combination, and by the 2010s this model had basically lost significance in top-level competition.

AlphaZero proved that in domains with clear rules and definable rewards, RL can learn strategies surpassing humans from scratch.

The mechanism of RL itself does not provide a source for goal generation; AlphaZero's game space is closed and its reward function is fixed.

PUCT（Predictive Upper Confidence bound applied to Trees）是 AlphaZero 系列算法中使用的一种树搜索策略，全称为 Predictor + UCT。

METIS-Core implements a natively running AlphaZero self-play loop, in which Monte Carlo Tree Search (MCTS) is executed in a multi-threaded manner in C++.

AlphaZero extended the same framework to multiple games including chess and shogi.

AlphaZero used no human chess records and surpassed Stockfish through just 4 hours of self-play.