Asynchronous Methods for Deep Reinforcement Learning
arXiv (Cornell University), 2016 · paper · AI and machine learning
Proposes a lightweight framework that trains deep reinforcement learning controllers using asynchronous parallel actor-learners. The best variant, asynchronous actor-critic, surpassed the prior state of the art on Atari in half the training time on a multi-core CPU, and also worked on motor control and 3D mazes.
Checked: PDF from arxiv.org, fetched 4 Oct 2026. Page numbers are those of that PDF.
1 finding
Reference or label Figure S7 (PDF p16, not p12): in the second row, panels 2 and 4 are both titled "Pendulum" and look nearly identical. The Pointmass 2D panel, which Figure S8 (p17) shows in the same position, is absent.
- p. 12 Figure S7, second row, panels 2 and 4
Pendulum
- p. 12 Figure S8, second row, panel 4
Pointmass 2D
Only the authors can say what is right here.
Judge's note: I compared the Figure S7 page image (p16) with the Figure S8 page image (p17). Figure S8's second row is Gripper, Pendulum, Reacher, Pointmass 2D. Figure S7's second row is Gripper, Pendulum, Reacher, Pendulum, with scatter patterns that look nearly identical. Only the authors can say whether the fourth panel is a mislabelled Pointmass 2D panel or a duplicated Pendulum panel.
Second check: another model agreed
Report a problem with this finding
22 minor (typos and the like; most checked by the judge only)
Wording Related Work gives Gorila "a total of 130 machines" (100 actor-learners + 30 parameter servers), while Table 1 lists its resources as "4 days, 100 machines".
- p. 2 Section 2 Related Work
a total of 130 machines
- p. 5 Table 1, Gorila row
4 days, 100 machines
Only the authors can say what is right here.
Judge's note: p2 says 100 actor-learner processes plus 30 parameter server instances, a total of 130 machines. Table 1 on p5 says 100 machines. The table may be counting only the actor-learners, but the paper does not say so. Only the authors can say which count is intended.
Second check: another model agreed
Report a problem with this finding
Wording The Table 1 caption says "Mean and median human-normalized scores on 57 Atari games", but Table S3 gives DQN and Gorila scores for only 49 games.
- p. 5 Table 1 caption
Mean and median human-normalized scores on 57 Atari games
- p. 19 Table S3, DQN and Gorila columns (blank for Berzerk, Defender, Phoenix, Pit Fall, Skiing, Solaris, Surround, Yars Revenge)
Berzerk
Only the authors can say what is right here.
Judge's note: In Table S3 (p19) the DQN and Gorila columns are blank for 8 games: Berzerk, Defender, Phoenix, Pit Fall, Skiing, Solaris, Surround and Yars Revenge. Section 2 (p2) also says Gorila was evaluated on 49 games. The caption does not say which game set each row's aggregate covers.
Second check: another model agreed
Report a problem with this finding
Wording Sec 9: "the two outputs of the policy network are two real number vectors which we treat as the mean vector µ and scalar variance σ2" calls both outputs vectors, but one of them is a scalar.
- p. 12 Section 9, second paragraph
two real number vectors which we treat as the mean vector µ and scalar variance σ2
Judge's note: Read on p12. This is a minor internal inconsistency; the meaning (a spherical covariance) is recoverable.
Second check: another model agreed
Report a problem with this finding
Typo "many updates are required the propagate a reward" should read "required to propagate".
- p. 2 Section 3, last paragraph
dates are required the propagate a reward
- p. 2 Section 3, last paragraph
many up-dates are required the propagate a reward
- p. 2
updates are required the propagate a reward
Judge's note: The page 2 image, last paragraph of Section 3, shows the text as quoted.
Second check: none (judge only)
Report a problem with this finding
Typo Sec 5.4 prints "exploring random 3D maxes using only a visual input" (should be 'mazes'); Sec 3 also prints "many updates are required the propagate a reward" (should be 'to propagate').
- p. 6 Section 5.4, Labyrinth results
exploring random 3D maxes
- p. 6 Sec 5.4 Labyrinth
exploring random 3D maxes using only
- p. 7 Figure 2 caption
acheive good scores
- p. 2 Section 3
required the propagate a reward
Judge's note: Both are visible in the page images: p6 right column, Sec 5.4, and p2 right column, Sec 3. The same paragraph on p6 says "previously unseen mazes". The 'acheive' typo that reviewer 4 also lists is judged under C019.
Second check: none (judge only)
Report a problem with this finding
Typo The update interval is written I_AsyncUpdate in Algorithm 1 but I_Update in Section 8 ("I_Update = 5").
- p. 3 Algorithm 1
if t mod IAsyncUpdate == 0 or s is terminal
- p. 12 Section 8
IUpdate = 5
Judge's note: Algorithm 1 on p3 uses I_AsyncUpdate; Section 8 on p12 uses I_Update for the same hyperparameter. The notation is inconsistent but the meaning is clear.
Second check: none (judge only)
Report a problem with this finding
Typo Section 3 writes "A(a_t, s_t) = Q(a_t, s_t)−V(s_t)" and "Q^π(a_t, s_t)", reversing the (s, a) argument order used elsewhere, e.g. Q(s,a) and A(s_t, a_t; θ, θ_v).
- p. 3 Section 3, advantage definition
A(at, st) = Q(at, st)−V (st)
- p. 4 Section 4, A3C
A(st, at; θ, θv)
Judge's note: Checked the p3 image: there the action comes first. Q(s,a) on p2 and A(s_t,a_t;θ,θ_v) on p4 put the state first. The notation is inconsistent but the meaning is recoverable.
Second check: none (judge only)
Report a problem with this finding
Typo The Table 1 caption reads "Supplementary Table SS3"; the text and the supplement call it Table S3.
- p. 5 Table 1 caption
Table SS3 shows the raw scores for all games
- p. 5 Table 1 caption
Supplementary Table SS3 shows the raw scores
- p. 5 Sec 5.1 text
Supplementary Table S3 shows the scores on all games
- p. 5 Table 1 caption
Supplementary Table SS3 shows the raw scores for all games.
Judge's note: The p5 image shows "SS3" in the caption and "S3" in the body text.
Second check: none (judge only)
Report a problem with this finding
Typo "Mujoco and Labyrinth" and "Mujoco physics engine" alongside "MuJoCo" elsewhere.
- p. 5 Section 5
Mujoco and Labyrinth
- p. 6 Section 5.3
Mujoco physics engine
Judge's note: On p5 "Mujoco" and "MuJoCo" appear in the same sentence. "Mujoco" also appears on p6 and p12. The capitalisation is inconsistent.
Second check: none (judge only)
Report a problem with this finding
Typo "A3C significantly improves on state-of-the-art the average score" is missing a word ("in"), and "current state-of-the art" is missing a hyphen.
- p. 5 Section 5.1
A3C significantly improves on state-of-the-art the average
- p. 5 Section 5.1
current state-of-the art
Judge's note: Both are visible in the p5 crop of Section 5.1.
Second check: none (judge only)
Report a problem with this finding
Typo Three typos: "Asynchronous Advantage-Critic" (missing "Actor"), "random 3D maxes" (mazes) and "acheive good scores" (achieve).
- p. 6 Section 5.3
Asynchronous Advantage-Critic found good solutions
- p. 6 Sec 5.4
random 3D maxes
- p. 7 Figure 2 caption
acheive good scores
Judge's note: The first two are on p6 (Sections 5.3 and 5.4); "acheive" is in the Figure 2 caption on p7.
Second check: none (judge only)
Report a problem with this finding
Typo Table 2 caption: "we measured the time to re-quired reach a fixed reference score" has garbled word order (should read 'the time required to reach').
- p. 6 Table 2 caption
the time to re-quired reach a fixed reference score
- p. 6 Table 2 caption
using one thread divided the time required
- p. 6 Table 2 caption
time to re-quired reach a fixed reference score
Judge's note: Seen in the p6 Table 2 caption. The missing 'by' in 'divided the time' is judged separately under C018.
Second check: none (judge only)
Report a problem with this finding
Typo Table 2 caption: "the time required to reach a fixed reference score using one thread divided the time required to reach the reference score using n threads" is missing 'by' (should be 'divided by the time').
- p. 6 Table 2 caption
one thread divided the time required to reach the reference score
Judge's note: Seen in the p6 Table 2 caption. This is a different defect from the word-order error in C017.
Second check: none (judge only)
Report a problem with this finding
Typo Figure 2 caption: "for which all random initializations acheive good scores" (should be 'achieve').
- p. 7 Figure 2 caption
which all random initializations acheive good scores
- p. 7 Figure 2 caption
acheive good scores
- p. 7 Figure 2 caption
initializations acheive good scores
- p. 7
random initializations acheive good scores
Judge's note: Visible in the p7 Figure 2 caption.
Second check: none (judge only)
Report a problem with this finding
Typo Wymann et al. reference prints the author name as mojibake: "EspiÃl', E."
- p. 10 Reference list, Wymann et al.
EspiÃl’, E.
Judge's note: The garbled characters are rendered in the p10 page image, so this is not a text-extraction artefact. The intended spelling (Espié) is inferred from the obvious encoding garble.
Second check: none (judge only)
Report a problem with this finding
Typo Sec 8: "followed by a convolutional layer with with 32 filters of size 4 × 4" repeats the word 'with'.
- p. 12 Section 8, network description
a convolutional layer with with 32 filters
- p. 12 Section 8
convolutional layer with with 32 filters
- p. 12 Section 9
in the the output layer
- p. 13 Section 9
case the we used a cost
Judge's note: Visible on p12, Sec 8. The other duplicated words this cluster mentions ('the the' and 'the we') are judged under C026 and C029.
Second check: code check
Report a problem with this finding
Typo Sec 9: "The most important difference in the architecture is in the the output layer of the policy network" repeats 'the'.
- p. 12 Section 9, second paragraph
is in the the output layer of the policy network
Judge's note: Visible on p12, Sec 9, second paragraph.
Second check: none (judge only)
Report a problem with this finding
Typo Sec 8: "The model used by actor-critic agents had two set of outputs" (should be 'two sets').
- p. 12 Section 8
had two set of outputs
Judge's note: Visible on p12, Sec 8.
Second check: none (judge only)
Report a problem with this finding
Typo Sec 9 (p13): "In the continuous case the we used a cost on the differential entropy" contains a stray 'the'.
- p. 13 Section 9, top of page
case the we used a cost on the differential entropy
Judge's note: The sentence starts at the bottom of p12 and continues at the top of p13, where the image shows "case the we used a cost".
Second check: none (judge only)
Report a problem with this finding
Typo Algorithm S3 bootstrap line writes "V(st, θ′v)" with a comma, while the lines below it in the same algorithm write "V(si; θ′v)" with a semicolon.
- p. 14 Algorithm S3, bootstrap line
V (st, θ′v)
Judge's note: Seen in the p14 page image. The notation is inconsistent but the meaning is clear.
Second check: none (judge only)
Report a problem with this finding
Typo The Figure S9 caption reads "different numbers of actor-learners one-step Sarsa on five Atari games" (a word such as 'for' is missing), and the Figure S10 caption reads "actor-learners for all one-step Sarsa on five Atari games" (the words 'all' or 'for all' are stray).
- p. 17 Figure S9 caption
actor-learners one-step Sarsa on five Atari games
- p. 18 Figure S10 caption
actor-learners for all one-step Sarsa on five Atari games
Judge's note: I looked at the crops of the Figure S9 caption (p17) and the Figure S10 caption (p18). Both wordings appear as quoted. The Figure 3 caption has the parallel form "actor-learners for three asynchronous methods", which suggests the S9/S10 wording is a copy-edit slip. It is cosmetic only.
Second check: none (judge only)
Report a problem with this finding
Typo A row label in Table S3 is truncated: it reads "Chopper Comman" where it should read "Chopper Command".
- p. 19 Table S3, row label
Chopper Comman
Judge's note: I looked at the Table S3 crop (p19). The row label is printed as "Chopper Comman" in the image, so this is not an extraction artefact. The text layer has the same truncation at line 2820. The correct game name is the obvious one, and the error is cosmetic.
Second check: none (judge only)