MCTS

AlphaGo combines deep learning with Monte Carlo Tree Search (MCTS), using two key neural networks: a policy network and a value network.

Monte Carlo Tree Search (MCTS) is one of the core algorithms for Go AI to search game positions, and was the mainstream technology in Go programs before AlphaGo.

Harness Router supports escalating to MCTS (Monte Carlo Tree Search) for complex multi-step decision scenarios.

MCTS is the planning core of systems such as AlphaGo and MuZero, and its core loop consists of four steps: selection, expansion, simulation, and backpropagation.

MCTS is not a theoretically perfect algorithm; its convergence guarantees are conditional and its actual performance depends on engineering tuning; for online tasks requiring fast responses, it often needs to be combined with heuristic methods or learned policies.

UCT (Upper Confidence bounds applied to Trees) is the most commonly used tree policy in MCTS, and its core idea is to apply the UCB1 formula from multi-armed bandits to tree node selection.

The convergence of MCTS is asymptotic and conditional, rather than reaching robust optimality within a finite number of steps; in practice the number of samples is finite, so only approximate solutions can be obtained.

The convergence speed of MCTS is closely related to the size of the state space and the branching factor, and for continuous action spaces or partially observable environments, the theoretical guarantees are weaker.

AlphaGo's success demonstrated the practicality of MCTS in a huge state space (Go's branching factor is about 250), but it also exposed its dependence on computational resources—a large number of GPUs/TPUs must be invoked during play.

AlphaZero further simplified this, reaching superhuman level using only self-play and MCTS, highlighting the generality of MCTS as a planning framework.

MCTS has a large computational overhead, requiring a large number of simulations for each decision, so it may be limited in scenarios with high real-time requirements, and the community view is that it is better suited to offline planning or problems with low simulation cost.

The hyperparameter sensitivity of MCTS is reflected in the following: an exploration constant c that is too small leads to premature convergence to a suboptimal branch, while one that is too large wastes simulations on low-value regions; the simulation depth affects evaluation accuracy; and the quality of the simulation policy directly determines the convergence speed.

In community discussions, some users pointed out that the theoretical analysis of MCTS is less mature than value iteration methods such as Q-learning, but its empirical effectiveness cannot be ignored.

Some researchers mentioned that in recent years the success of combining MCTS with deep learning (such as AlphaZero) has partially obscured its theoretical flaws, but understanding its limitations is crucial for safety-critical applications.

MCTS has proven its practical value in multiple domains, including game AI, robot path planning, resource scheduling, and automated testing scenarios.

The AlphaZero-style expert loop refers to borrowing AlphaZero's training paradigm: first evaluating actions and selecting the optimal action through Monte Carlo tree search (MCTS) or a similar forward search, then distilling the (state, action) pairs produced by the search as supervision signals into the policy network, and this can be iterated.

Search-tree methods (such as MCTS and beam search) rely on precise simulation of game rules; their advantage is being interpretable and requiring no training data, but each step requires expanding a large number of branches, creating heavy real-time pressure.

AlphaZero's core technical combination includes MCTS for lookahead evaluation of moves, and a deep neural network that simultaneously outputs a policy head (prior probabilities for each action) and a value head (the expected win/loss of the current position).

The material heuristic fuses a hand-designed positional evaluation (degree of closeness to the target, spacing between the two drones, turret exposure angle) with the network value head, working together during the MCTS search.

The basic idea of MCTS is to repeatedly carry out the four steps of "selection-expansion-simulation-backpropagation", gradually building the game tree, usually using the PUCT formula to combine the network's output prior probabilities and accumulated value.