Playing Atari with Deep Reinforcement Learning
arXiv (Cornell University), 2013 · paper · AI and machine learning
Presents a convolutional neural network trained with a variant of Q-learning to learn game-playing control directly from raw pixels. Applied to seven Atari 2600 games without changing architecture, it outperforms previous approaches on six and a human expert on three.
Checked: PDF from arxiv.org, fetched 4 Oct 2026. Page numbers are those of that PDF.
4 findings
Reference or label Reference [10] gives the venue as "Proc. International Conference on Computer Vision and Pattern Recognition (CVPR 2009), pages 2146–2153". The paper appeared at ICCV 2009.
- p. 9 Reference [10]
Proc. International Conference on Computer Vision and Pattern Recognition (CVPR 2009), pages 2146–2153
Only the authors can say what is right here.
Judge's note: Rests on outside knowledge: that Jarrett et al., 'What is the best multi-stage architecture for object recognition?', was published in ICCV 2009 proceedings, pp. 2146–2153. The page image of p9 shows the quoted CVPR 2009 venue text as stated.
Second check: another model agreed
Report a problem with this finding
Wording The Introduction ("all hyperparameters used for training were kept constant across the games"), the opening of Section 5 ("same ... hyperparameters settings across all seven games") and the Conclusion ("no adjustment of the architecture or hyperparameters") conflict with Section 5, which says k = 3 for Space Invaders was "the only difference in hyperparameter values between any of the games".
- p. 2 Introduction, para 2
all hyperparameters used for training were kept constant across the games
- p. 6 Section 5, para 1
We use the same network architecture, learning algorithm and hyperparameters settings across all seven games
- p. 6 Section 5, para 3
this change was the only difference in hyperparameter values between any of the games
- p. 8 Conclusion
with no adjustment of the architecture or hyperparameters
Judge's note: Checked on p.2, p.6 and p.8. The paper itself calls frame-skip k a hyperparameter whose value differed for one game, so the blanket claims of identical hyperparameters overstate. The abstract claims only no adjustment of architecture or learning algorithm, which is consistent. The exception is disclosed in Section 5, so this is a wording overstatement and not a wrong fact.
Second check: another model agreed
Report a problem with this finding
Wording The greedy action is written with max instead of argmax: "a = max_a Q(s, a; θ)" (p.3) and "a_t = max_a Q*(φ(s_t), a; θ)" (Algorithm 1, p.5).
- p. 3 Section 2, last paragraph
a = maxa Q(s, a; θ)
- p. 5 Algorithm 1, line 6
otherwise select at = maxa Q∗(φ(st), a; θ)
Judge's note: Both are visible in the images. The left-hand side is an action, so the operator should be argmax. Elsewhere, max_a' is used correctly for values (Eq. 1, 3 and y_j). The intended meaning is clear, so this is a notation slip.
Second check: another model agreed
Report a problem with this finding
Wording Algorithm 1 selects actions with "Q*(φ(s_t), a; θ)", although Q* is defined as the true optimal action-value function and the network is Q(s,a;θ) ≈ Q*(s,a).
- p. 5 Algorithm 1, action selection
Q∗(φ(st), a; θ)
Judge's note: The Algorithm 1 image shows Q* with θ on the action-selection line, while y_j and the loss use Q(·;θ). Section 2 (p.3) defines Q(s,a;θ) as the approximator of Q*. This is a notation inconsistency, separate from the max/argmax issue in C006.
Second check: another model agreed
Report a problem with this finding
14 minor (typos and the like; most checked by the judge only)
Reference or label Reference [8], "A neuro-evolution approach to general atari game playing. 2013.", gives no venue, report or URL, although the Table 1 HNeat baselines are taken from it.
- p. 9 Reference [8]
A neuro-evolution approach to general atari game playing. 2013.
Only the authors can say what is right here.
Judge's note: The p9 page image confirms that the entry lists only authors, title and year. A reader cannot locate the source of the HNeat baseline numbers from the entry alone. It may have been unpublished at the time, so only the authors can supply the source.
Second check: another model agreed
Report a problem with this finding
Wording The Introduction says DQN "outperformed all previous RL algorithms on six of the seven games", but against the RL baselines (Sarsa, Contingency) Table 1 and Section 5.3 ("outperforms the other learning methods by a substantial margin on all seven games") show wins on all seven. The one loss (Space Invaders) is to HNeat, which the paper calls an evolutionary policy search approach.
- p. 2 Introduction, para 2
outperformed all previous RL algorithms on six of the seven games
- p. 7 Section 5.3, para 2
outperforms the other learning methods by a substantial margin on all seven games
Judge's note: Table 1 (p.8), S. Invaders: DQN 581 vs Sarsa 271 and Contingency 268, but HNeat Best 1720. The abstract ('all previous approaches') and p.8 ('on all the games, except Space Invaders') are consistent with this. Only the Introduction's 'RL algorithms' label fits awkwardly, unless HyperNEAT counts as RL. Reviewer 2's claim that 'all seven' contradicts the data is wrong: Section 5.3 refers to the first five rows. This is a minor terminology inconsistency.
Second check: another model agreed
Report a problem with this finding
Wording Section 4 says the network "operates directly on RGB images" (and p.7 says the agents "only receive the raw RGB screenshots as input"), but Section 4.1 converts frames "to gray-scale", down-samples and crops them to 84×84×4 before input.
- p. 4 Section 4, para 1
a deep neural network which operates directly on RGB images
- p. 5 Section 4.1
first converting their RGB representation to gray-scale
Judge's note: Checked p.4, p.5 and p.6. 'RGB' probably means raw screens with no hand-crafted features, but taken literally the network input is gray-scale. Minor wording ambiguity.
Second check: another model agreed
Report a problem with this finding
Wording Training length is ambiguous: Section 5 says "We trained for a total of 10 million frames", while Figure 2 runs to 100 epochs, and its caption says "One epoch corresponds to 50000 minibatch weight updates". That is 5M updates. Algorithm 1 makes one update per agent step, and with frame skip k=4, 10M frames would be about 2.5M steps.
- p. 7 Figure 2 caption and x-axes (0–100 epochs)
One epoch corresponds to 50000 minibatch weight updates
- p. 6 Section 5, para 2
We trained for a total of 10 million frames
Only the authors can say what is right here.
Judge's note: I checked the Figure 2 caption and x-axes (0–100 epochs) on p7, Section 5 on p6 (10 million frames, k=4 frame skip) and Algorithm 1 on p5 (one minibatch gradient step per time step). The paper never says whether 'frames' means emulator frames or agent steps. Read as emulator frames, 10M frames give about 2.5M updates, fewer than the 5M plotted. Read as agent steps, the figures cover only half the training run. Either way, the relation between Section 5 and Figure 2 is left unclear. Only the authors can resolve this, so it is an ambiguity, not a proven error.
Second check: another model agreed
Report a problem with this finding
Wording Section 5.3 says "it achieves close to human performance on Beam Rider", but Table 1 gives DQN 4092 against Human 7456, about 55% of the human score.
- p. 8 Section 5.3, final paragraph
it achieves close to human performance on Beam Rider
- p. 8 Table 1, B. Rider column
4092
Judge's note: Table 1 on p8 confirms DQN 4092, Human 7456 and DQN Best 5184 for B. Rider. 'Close' is a relative judgement: next to Q*bert (about 10%), Seaquest (about 6%) and Space Invaders (about 16%), Beam Rider is clearly nearer to human level. Even so, the plain wording overstates a 45% gap. I treat this as a minor wording point.
Second check: another model agreed
Report a problem with this finding
Typo p.2: "were designed to be difficult for humans players" should read "human players".
- p. 2 Introduction, para 2
difficult for humans players
- p. 2 Section 1 Introduction, second paragraph on p.2
were designed to be difficult for humans players
Judge's note: Visible on the page 2 image.
Second check: none (judge only)
Report a problem with this finding
Typo The name is written inconsistently: "TD-gammon" in Section 3 body text, but "TD-Gammon" in footnote 1 and on p.4.
- p. 3 Section 3, paras 1–2
TD-gammon used a model-free reinforcement learning algorithm
- p. 3 Footnote 1
In fact TD-Gammon approximated
Judge's note: Both forms appear on page 3, and p.4 uses TD-Gammon. Cosmetic.
Second check: none (judge only)
Report a problem with this finding
Typo Algorithm 1: "preprocessed sequenced φ1 = φ(s1)" should read "preprocessed sequence".
- p. 5 Algorithm 1, line 4
preprocessed sequenced
- p. 5 Algorithm 1
preprocessed sequenced φ1 = φ(s1)
Judge's note: Visible in the Algorithm 1 image on p.5.
Second check: none (judge only)
Report a problem with this finding
Typo Algorithm 1 mixes spellings: "Initialize replay memory" / "Initialize action-value function" vs "Initialise sequence s1".
- p. 5 Algorithm 1, lines 1–4
Initialise sequence s1
Judge's note: Visible in the Algorithm 1 image. The paper also uses both 'behaviour' (p.3) and 'behavior' (p.5, p.6). Mixed British/US spelling; cosmetic.
Second check: none (judge only)
Report a problem with this finding
Typo p.6: "The input to the neural network consists is an 84 × 84 × 4 image" has a duplicated verb.
- p. 6 Section 4.1, first line of p.6
network consists is an 84 × 84 × 4 image
- p. 6 Section 4.1, architecture paragraph
The input to the neural network consists is an 84 × 84 × 4 image
Judge's note: Visible in the first lines of the page 6 image.
Second check: none (judge only)
Report a problem with this finding
Typo p.6: "hyperparameters settings" should read "hyperparameter settings".
- p. 6 Section 5, para 1
hyperparameters settings
Judge's note: Visible on the page 6 image, in the first paragraph of Section 5.
Second check: none (judge only)
Report a problem with this finding
Typo Section 5.1: "states the policy visits ." has a stray space before the full stop.
- p. 6 Section 5.1
states the policy visits .
Judge's note: The space is visible in the crop image, so it is not an extraction artefact.
Second check: none (judge only)
Report a problem with this finding
Typo In reference [10] the author name "MarcAurelio Ranzato" is missing its apostrophe (Marc'Aurelio).
- p. 9 Reference [10]
MarcAurelio Ranzato
Judge's note: The p9 page image and the crop show 'MarcAurelio' with no apostrophe. This is a cosmetic BibTeX escaping slip.
Second check: none (judge only)
Report a problem with this finding
Typo The same author is spelled two ways: "Csaba Szepesvari" in reference [14] and "Csaba Szepesvári" in [15]. Sutton is also given as "Rich Sutton" in [14] and "Richard S. Sutton" in [15].
- p. 9 Reference [14]
Csaba Szepesvari
- p. 9 Reference [15]
Csaba Szepesvári
Judge's note: The p9 page image shows that the accent is missing in [14] and present in [15], and that the Sutton name forms differ. The inconsistency is cosmetic.
Second check: none (judge only)