Adaptive Multistage Sampling Algorithm: The Origins of Monte Carlo Tree Search

1. Adaptive Multistage Sampling Algorithm:
The Origins of Monte Carlo Tree Search
Ashwin Rao
ICME, Stanford University
December 9, 2019
Ashwin Rao (Stanford) Adaptive Multistage Sampling December 9, 2019 1 / 7

2. Monte Carlo Tree Search (MCTS)
MCTS was popularized a few years ago by Deep Mind’s AlphaGo
It is a simulation-based method to identify the best action in a state
MCTS term was first introduced by Remi Coulom for game trees
Each round of MCTS consists of four steps:
Selection: Successively select children from root R to leaf L
Expansion: Create node C as a new child of L
Simulation: Complete a random playout from C
Backpropagation: Use result of playout to update nodes from C to R
Ashwin Rao (Stanford) Adaptive Multistage Sampling December 9, 2019 2 / 7

3. The Selection Step in MCTS
Selection involves picking a child with “most promise”
This means prioritizing children with higher success estimates
For estimate confidence, we need sufficient playouts under each child
This is our usual Explore v/s Exploit dilemma (Multi-armed Bandit)
Explore v/s Exploit formula for games first due to Kocsis-Szepesvari
Formula called Upper Confidence Bound 1 for Trees (abbrev. UCT)
Most current MCTS Algorithms are based on some variant of UCT
UCT is based on UCB1 formula of Auer, Cesa-Bianchi, Fischer
However, MCTS and UCT concepts first appeared in the
Adaptive Multistage Sampling algorithm of Chang, Fu, Hu, Marcus
Adaptive Multistage Sampling (AMS) is a generic simulation-based
algorithm to solve a finite-horizon Markov Decision Process (MDP)
AMS can be considered as the “spiritual origin” of MCTS/UCT
Hence, this lecture is dedicated to AMS
Ashwin Rao (Stanford) Adaptive Multistage Sampling December 9, 2019 3 / 7

4. The Setting for the AMS Algorithm
MDP with finite number of time steps t = 0,1,...,T
State denoted st ∈ S, where S is very large
Action denoted at ∈ A, where A is fairly small
Reward rt ∈ R, with E[rt (st,at)] provided as a function R(st,at)
Next time step’s state st+1 can be generated by invoking a random
sampling function SF(st,at), i.e., st+1 = SF(st,at)()
Discount factor denoted as γ, and rT = 0
The problem is to calculate the Optimal Value function V ∗
t (st)
Unlike tabular backward induction where state transition probabilities
are given, here only a sampling function (for next state) is given
Armed with the sampling function, can we do better than backward
induction for the case where S is very large and A is small?
Ashwin Rao (Stanford) Adaptive Multistage Sampling December 9, 2019 4 / 7

5. Outline of AMS Algorithm
AMS Algorithm is based on a fixed allocation of action selections for
each state in each time step
Denote number of action selections per state in time step t as Nt
Denote ˆV Nt
t (st) as the AMS Algorithm estimate of V ∗
t (st)
Let Nst ,at
t be the number of selections of at for st (∑at ∈A Nst ,at
t = Nt)
Proportions of Nst ,at
t based on Explore v/s Exploit UCT formula
For each of the Nst ,at
t selections of at, one next-state st+1 is sampled
Each st+1 = SF(st,at)() sample leads to recursive call ˆV Nt+1
t+1 (st+1)
Optimal Action Value Function Q∗
t (st,at) estimated as:
ˆQNt
t (st,at) = R(st,at) + γ ⋅
∑
N
st ,at
t
j=1
ˆV Nt+1
t+1 (SF(st,at)())
Nst ,at
t
V ∗
t (st) = maxat Q∗
t (st,at) approximated as:
ˆV Nt
t (st) = ∑
at
Nst ,at
t
Nt
⋅ ˆQNt
t (st,at)
Ashwin Rao (Stanford) Adaptive Multistage Sampling December 9, 2019 5 / 7

6. The AMS Algorithm
Algorithm 0.1: OptVF(t,s,Nt)
if t == T return (0)
comment: Initialize VALS and CNTS by selecting each action once
for a ← A
do {
VALS[a] ← OptVF(t + 1,SF(s,a)(),Nt+1)
CNTS[a] ← 1
for i ← A to Nt − 1
do
⎧⎪⎪⎪⎪⎪⎪⎪
⎨
⎪⎪⎪⎪⎪⎪⎪⎩
comment: Pick action based on UCB1 Explore v/s Exploit formula
a∗
← argmaxa∈A(R(s,a) + γ ⋅
VALS[a]
CNTS[a] +
√
2 ln i
CNTS[a] )
VALS[a∗
] ← VALS[a∗
] + OptVF(t + 1,SF(s,a∗
)(),Nt+1)
CNTS[a∗
] ← CNTS[a∗
] + 1
return (∑a∈A
CNTS[a]
Nt
⋅ (R(s,a) + γ ⋅
VALS[a]
CNTS[a] ))
comment: Nt next-state samplings and Nt recursive calls to OptVF
Ashwin Rao (Stanford) Adaptive Multistage Sampling December 9, 2019 6 / 7

7. Running Time, Bias, Convergence and Code
Let N = max(N0,N1,...,Nt−1) and assume N > A
Running time of AMS Algorithm is of the order of NT
⋅ A
Compare this versus backward induction running time of S 2
⋅ A ⋅ T
So AMS is more efficient when S is very large (typical in real-world)
AMS paper proves the estimate ˆV N0
0 (s0) is asymptotically unbiased
lim
N0→∞
lim
N1→∞
... lim
NT−1→∞
E[ ˆV N0
0 (s0)] = V ∗
0 (s0) for all s0 ∈ S
AMS paper also proves that the worst-possible bias is bounded by a
quantity that converges to zero at rate O(∑T−1
t=0
ln Nt
Nt
)
0 ≤ V ∗
0 (s0) − E[ ˆV N0
0 (s0)] ≤ O(
T−1
∑
t=0
lnNt
Nt
) for all s0 ∈ S
Here’s some Python code for the AMS Algorithm you can play with
Ashwin Rao (Stanford) Adaptive Multistage Sampling December 9, 2019 7 / 7