Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model
RePEc: Research Papers in Economics, 2019 · paper · AI and machine learning
Presents MuZero, which combines tree-based search with a learned model of reward, policy and value, without knowing the environment's dynamics. It set a new state of the art on 57 Atari games and matched AlphaZero in Go, chess and shogi.
Checked: PDF from arxiv.org, fetched 4 Oct 2026. Page numbers are those of that PDF.
3 findings
Substance Text says MuZero outperforms R2D2 "in 42 out of 57 games", but Table S1 shows MuZero strictly higher in 41 games, with 2 ties (pitfall 0.00/0.00, pong 21.00/21.00) and 14 games where R2D2 is higher.
- p. 5 Section 4 Results, Atari paragraph
model-free approach) in 42 out of 57 games
- p. 17 Table S1, R2D2 and MuZero columns
# best
- p. 5 Section 4 Results, Atari paragraph
in 42 out of 57 games
- p. 17 Table S1, R2D2 [21] and MuZero columns
Evaluation of MuZero in Atari for individual games with 30 random no-op starts
Only the authors can say what is right here.
Judge's note: I compared the R2D2 and MuZero columns of Table S1 row by row against the page 17 image and the layout text. R2D2 is higher in 14 games: amidar, asterix, bank heist, gravitar, ice hockey, kung fu master, montezuma revenge, qbert, seaquest, solaris, star gunner, venture, video pinball and yars revenge. Pitfall and pong are ties. That leaves 41 strict wins; counting both ties as wins gives 43. No counting rule in the paper produces 42. The sentence cites Table S1, which is the 30 no-op evaluation, and Table S2 has no R2D2 column.
Second check: another model agreed
Report a problem with this finding
Substance Figure S2 gives the losses as "l^p(π, p) = π^T log p", "l^r = φ(u)^T log r" and "l^v = φ(z)^T log q" without a minus sign, so they are log-likelihoods (≤ 0), not the cross-entropy losses to be minimised in Eq. (1).
- p. 19 Figure S2, Losses
lp(π, p) = πT log p
Judge's note: Viewed p.19. For probability vectors, π^T log p ≤ 0. The loss l_t(θ) in Eq. 1 / Figure S2 is minimised and also contains +c||θ||² and (z − q)², which are minimised terms, so maximisation cannot be the intended convention. Footnote 1 (p.4) calls these cross-entropy losses, which are −π^T log p. The formulas as printed have the wrong sign.
Second check: another model agreed
Report a problem with this finding
Reference or label Appendix A cites "(see Figure 1 A)" for AlphaZero's simulator plus network planning and "(see Figure 1 B)" for MuZero's learned model in the search, but Figure 1A shows MuZero planning with its learned model and Figure 1B shows MuZero acting in the environment.
- p. 11 Appendix A, paragraphs 2–3
simulator (see Figure 1 A)
- p. 11 Appendix A, paragraph 3
neural network (see Figure 1 B):
- p. 11 Appendix A, paragraph 2
produced by the simulator (see Figure 1 A)
- p. 11 Appendix A, after list intro
learned by a neural network (see Figure 1 B)
Only the authors can say what is right here.
Judge's note: Viewed p.3 (Figure 1 and its caption) and p.11. The caption says '(A) How MuZero uses its model to plan', '(B) How MuZero acts in the environment', '(C) How MuZero trains its model'. No panel shows AlphaZero or a simulator, and the learned model inside the search is panel A, not B. Neither pointer reaches the content it describes, which suggests they were carried over from an earlier figure layout.
Second check: another model agreed
Report a problem with this finding
20 minor (typos and the like; most checked by the judge only)
Reference or label Cross-references "(see Methods)" (p.4) and "(see Methods for further details)" (p.5) point to a section that does not exist; the material is in Appendices B and F.
- p. 4 Section 3
intermediate rewards (see Methods)
- p. 5 Section 4
(see Methods for further details)
- p. 4 Section 3, search description
generalized to allow for single agent domains and intermediate rewards (see Methods)
Judge's note: I searched the text for section headings. The paper has Sections 1–6 and Appendices A–I, with no section titled 'Methods'. The architecture is in Appendix F, 'Network Architecture', and the search is in Appendix B. The pointers are probably left over from the journal format.
Second check: another model agreed
Report a problem with this finding
Reference or label Figure 2 Atari panel y-axis is labelled "Reward" although the caption says it shows "mean (full line) and median (dashed line) human normalized scores across all 57 games".
- p. 5 Figure 2, Atari panel y-axis label
For Atari, mean (full line) and median (dashed line) human normalized scores
- p. 5 Figure 2 caption / Atari panel y-axis
mean (full line) and median (dashed line) human normalized scores across all 57 games are shown on the y-axis
- p. 5 Figure 2 Atari panel y-axis vs caption
human normalized scores across all 57 games are shown on the y-axis
Judge's note: Page 5 image: the Atari panel axis label is 'Reward'. The R2D2 orange lines sit at about 4000 and 1900, which match Table 1 (4024.9% and 1920.6%) as human-normalized percentages. The data look consistent with the caption; only the axis label is wrong.
Second check: another model agreed
Report a problem with this finding
Reference or label Figure 3B y-axis is labelled "Mean Reward" while the caption says it shows "final human normalized mean score in Atari".
- p. 7 Figure 3B y-axis and caption (B)
Scaling of final human normalized mean score in Atari with the number of simulations per search
Judge's note: Page 7 image: the panel B axis reads 'Mean Reward'. The R2D2 dashed line at about 4025 matches the R2D2 mean of 4024.9% in Table 1, so the plotted quantity is the human-normalized percentage. Panel C uses the same label for raw Ms. Pacman reward. This is a different figure from C009, so it is kept as a separate cluster.
Second check: another model agreed
Report a problem with this finding
Reference or label The Supplementary Materials list offers "Data for Figures 2, 3, S2, S3, S4 ... in JSON format", but Figure S2 contains only equations, and the data figure S1 (repeatability curves) is not listed.
- p. 11 Supplementary Materials list
Data for Figures 2, 3, S2, S3, S4 and Tables 1, S1, S2 in JSON format.
- p. 19 Figure S2 caption
Equations summarising the MuZero algorithm
Only the authors can say what is right here.
Judge's note: Viewed p.11 (the list) and p.19 (Figure S2 is 'Equations summarising the MuZero algorithm', with no plotted data). Figure S1 (p.15/16 text) is a repeatability plot. The 'S2' is probably meant to be 'S1', but this cannot be checked without the ancillary files.
Second check: another model agreed
Report a problem with this finding
Wording Eq. (1) sums "Σ_{k=0}^{K} l^r(u_{t+k}, r^k_t) + ..." from k = 0, but r^k is produced only by the dynamics function g for k ≥ 1 (s^0 = h(o_1..o_t) has no reward), so the k = 0 reward term is undefined.
- p. 4 Equation (1)
lr(ut+k, rk t )
- p. 3 Section 3
k = 1...K steps
Only the authors can say what is right here.
Judge's note: Page 4 Eq. (1) and Figure S2 on page 19 both sum from k=0 with a reward term in every summand. Section 3 (page 4) defines r^k, s^k = g(s^{k-1}, a^k) and s^0 = h(...), so there is no r^0. Figure 1C shows rewards r^1 and r^2 only. The k=0 reward term is presumably dropped in practice, but the formula as printed contains it.
Second check: another model agreed
Report a problem with this finding
Wording Footnote 1: "For chess, Go and shogi, the same squared error loss as AlphaZero is used for rewards and values" conflicts with Appendix G ("in board games without intermediate rewards, we omit the reward prediction loss") and Figure S2 (l^r(u,r) = 0 "for games").
- p. 4 Footnote 1
the same squared error loss as AlphaZero is used for rewards and values
- p. 15 Appendix G
in board games without intermediate rewards, we omit the reward prediction loss
- p. 19 Figure S2 Losses
for games
Only the authors can say what is right here.
Judge's note: Page 4 footnote 1, page 15 Appendix G and page 19 Figure S2 all checked. The footnote says a squared-error loss is applied to rewards in board games. Figure S2 sets the reward loss to 0 for games, and Appendix G says it is omitted. The authors should say which is correct; the footnote is probably loose wording.
Second check: another model agreed
Report a problem with this finding
Wording Appendix B states "In two-player zero sum games the value functions are assumed to be bounded within the [0, 1] interval", but Section 3 defines board-game outcomes as "rewards u_t ∈ {−1, 0, +1}" and Appendix A refers to "terminal rewards of ±1".
- p. 12 Appendix B, after Eq. 4
In two-player zero sum games the value functions are assumed to be bounded within the [0, 1] interval.
- p. 4 Section 3
are treated as rewards ut ∈{−1, 0, +1}
Only the authors can say what is right here.
Judge's note: Viewed p.4 (crop showing {−1, 0, +1}), p.11 (±1) and p.12 ([0, 1]). The paper describes no rescaling of board-game values from [−1, 1] to [0, 1]; the only normalisation described is the min-max Q̄ in Eq. 5. The paper leaves the range unclear, so the authors need to say which convention applies.
Second check: another model agreed
Report a problem with this finding
Wording Figure 2 caption: Elo "established by playing games against AlphaZero using 800 simulations per move for both players"; Appendix I: tournament between MuZero iterations and "a baseline player: either Stockfish, Elmo or AlphaZero respectively", where "Baseline players used an equivalent search time of 100ms per move".
- p. 5 Figure 2 caption
established by playing games against Alp- haZero using 800 simulations per move for both players
- p. 15 Appendix I
either Stockfish, Elmo or AlphaZero respectively
Only the authors can say what is right here.
Judge's note: I read the Figure 2 caption on page 5 and Appendix I on pages 15–16 (text layer lines 789–798). The opponents differ: AlphaZero only, versus Stockfish/Elmo/AlphaZero. The baseline search setting also differs: 800 simulations for both players, versus 100 ms for the baseline. The authors should say which protocol produced Figure 2.
Second check: another model agreed
Report a problem with this finding
Wording "outperforming the previous best model-based approach SimPLe [20] in all games (see Table S1)" follows "across the 57 games", but Table S1 caption states "SimPLe was only evaluated on 36 of the 57 games".
- p. 5 Section 4, Atari paragraph
SimPLe [20] in all games (see Table S1)
- p. 17 Table S1 caption
SimPLe was only evaluated on 36 of the 57 games
- p. 5 Section 4 Results, Atari paragraph
outperforming the previous best model-based approach SimPLe [20] in all games (see Table S1)
Judge's note: Table S1 on page 17 has SimPLe scores for 36 games, and MuZero is higher in each of them. The claim holds for the games where SimPLe was evaluated. Read in a paragraph about 57 games, though, 'all games' overstates it; the restriction is stated only in the Table S1 caption.
Second check: another model agreed
Report a problem with this finding
Wording Text "our model-free algorithm achieved identical results to R2D2" and Fig. 3C caption "reaches the same final score as R2D2", while the plotted Q-Learning curve touches the R2D2 line (~43k) only near 0.65M steps and ends at ~27k at 1.0M steps.
- p. 6 Section 4, model-free comparison paragraph
our model-free algorithm achieved identical results to R2D2
- p. 7 Figure 3 caption (C)
Our Q-Learning implementation reaches the same final score as R2D2
Only the authors can say what is right here.
Judge's note: Page 7, Figure 3C image: the R2D2 dashed line is at about 42–43k. The Q-learning curve peaks at about 43k near 0.65M steps and ends at about 25–27k at 1.0M steps. 'Same final score' or 'identical' fits the peak but not the plotted end point. The authors should say which value is meant.
Second check: another model agreed
Report a problem with this finding
Typo Figure 1 caption (C) writes the predicted reward as "r_{t+k} ≈ u_{t+k}" (subscript), while the adjacent p^k and v^k and the rest of the paper (caption A, Section 3, Eq. 1) use the superscript r^k / r^k_t.
- p. 3 Figure 1 caption (C)
and reward rt+k ≈ut+k
Judge's note: I checked the crop and page 3. The caption has 'p^k ≈ π_{t+k}, value function v^k ≈ z_{t+k}, and reward r_{t+k} ≈ u_{t+k}'. Part (A) of the same caption uses r^k, and Section 3 uses r^k_t. This is a notational inconsistency only.
Second check: none (judge only)
Report a problem with this finding
Typo Page 4 names the benchmark "Atari Learning Environment [2]"; page 5 and the title of ref [2] call it the "Arcade Learning Environment".
- p. 4 Section 4, first sentence
all 57 games in the Atari Learning Environment [2]
- p. 5 Section 4, Atari paragraph
games of the Arcade Learning Environment
- p. 5 Results
Arcade Learning Environment
Judge's note: Page 4, Results, first paragraph: the image reads 'all 57 games in the Atari Learning Environment [2]'. Page 5 reads 'games of the Arcade Learning Environment'. The name is inconsistent within the paper.
Second check: none (judge only)
Report a problem with this finding
Typo Misspelling "occuring at the final step of the episode" (should be "occurring").
- p. 4 Section 3, paragraph on value targets
occuring at the final step of the episode
Judge's note: The page 4 image, Section 3, shows 'occuring'.
Second check: none (judge only)
Report a problem with this finding
Typo Section 3 and the Figure 1B caption write "An action a_{t+1} ∼ π_t is then selected", whereas Figure S2 (Search) writes "a_t ∼ π_t".
- p. 4 Section 3
An action at+1 ∼πt is then selected.
- p. 19 Figure S2, Search
at ∼πt
Judge's note: Page 4 has a_{t+1} ∼ π_t, and the Figure 1B caption on page 3 also has a_{t+1}. Figure S2 on page 19 has a_t ∼ π_t. The time index is inconsistent; either convention could be intended.
Second check: none (judge only)
Report a problem with this finding
Typo Reference [10] title prints "TreeQN and ATreec"; the method name is ATreeC (the lowercase c looks like BibTeX lower-casing).
- p. 8 Reference [10]
TreeQN and ATreec
Judge's note: Viewed p.8. The entry reads 'TreeQN and ATreec: Differentiable tree planning...'. This is a cosmetic capitalisation slip in the method name.
Second check: none (judge only)
Report a problem with this finding
Typo Appendix A writes "a hidden state s_{k−1} and an action a_k ... s_k = g(s_{k−1}, a_k)" with subscript k, whereas elsewhere hypothetical steps are superscripts (s^k, a^k) and subscripts denote real time-steps.
- p. 11 Appendix A, item 1
the search algorithm can transition to a new node sk = g(sk−1, ak)
Judge's note: Viewed p.11 (subscripts) against p.3 (Figure 1 caption, s^{k−1}, a^k) and p.12 (Appendix B, superscripts). This is a notational inconsistency only; the meaning is clear from context.
Second check: none (judge only)
Report a problem with this finding
Typo Eq. (3) writes "γ^τ r_{k+1+τ} + γ^{l−k} v^l" with a subscripted reward index, while the rest of Appendix B uses superscripts (r^k, r^l) for hypothetical steps.
- p. 12 Appendix B, Eq. (3)
γτrk+1+τ + γl−kvl
Judge's note: Viewed p.12. Eq. 3 has r with subscript k+1+τ; the surrounding text has r^k = R(s^{k−1}, a^k) and r^l. This is a notational inconsistency, and the meaning can be inferred.
Second check: none (judge only)
Report a problem with this finding
Typo Eq. (6) "p_α = N(α)^{1/T} / Σ_b N(b)^{1/T}" uses Greek α as an action index, while Appendix G uses α as the prioritized-replay exponent ("we set α = β = 1").
- p. 13 Appendix D, Eq. (6)
pα =
- p. 15 Appendix G
we set α = β = 1
Judge's note: Viewed p.13 (Eq. 6 shows a Greek α as the action index, with b in the sum) and p.15 (p_i^α in the priority, 'α = β = 1'). One symbol carries two meanings, and the action is written a everywhere else. This is cosmetic.
Second check: none (judge only)
Report a problem with this finding
Typo Appendix F uses "invertible transform h(x) = sign(x)(√(|x|+1) − 1 + εx)" while h (h_θ) denotes the representation function throughout the paper.
- p. 14 Appendix F, paragraph 2
invertible transform h(x) =
Judge's note: Viewed p.14 and p.3. The symbol h is overloaded. Context separates the two uses, and the notation follows ref [30], so this is only a minor clash.
Second check: none (judge only)
Report a problem with this finding
Typo Appendix I: "Elo ratings were computed from the results of a 800 simulations per move tournament" should read "an 800-simulations-per-move tournament".
- p. 15 Appendix I
a 800 simulations per move tournament
Judge's note: Viewed p.15. '800' is read 'eight hundred', so the article should be 'an'. Cosmetic.
Second check: none (judge only)