Monte-Carlo Tree Search as Regularized Policy Optimization

The combination of Monte-Carlo tree search (MCTS) with deep reinforcement learning has led to significant advances in artificial intelligence. However, AlphaZero, the current state-of-the-art MCTS algorithm, still relies on handcrafted heuristics that are only partially understood. In this paper, we...

Full description

Saved in:
Bibliographic Details
Published in:arXiv.org 2020-07
Main Authors: Jean-Bastien Grill, Altché, Florent, Tang, Yunhao, Thomas, Hubert, Valko, Michal, Antonoglou, Ioannis, Munos, Rémi
Format: Article
Language:English
Subjects:
Artificial intelligence
Exact solutions
Monte Carlo simulation
Optimization
Search algorithms
Searching
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:The combination of Monte-Carlo tree search (MCTS) with deep reinforcement learning has led to significant advances in artificial intelligence. However, AlphaZero, the current state-of-the-art MCTS algorithm, still relies on handcrafted heuristics that are only partially understood. In this paper, we show that AlphaZero's search heuristics, along with other common ones such as UCT, are an approximation to the solution of a specific regularized policy optimization problem. With this insight, we propose a variant of AlphaZero which uses the exact solution to this policy optimization problem, and show experimentally that it reliably outperforms the original algorithm in multiple domains.
ISSN:2331-8422