Human-level Control through Deep Reinforcement Learning
Nature, 2015 · paper · AI and machine learning
Introduces the deep Q-network, an agent that learns to play Atari 2600 games directly from screen pixels and score using end-to-end reinforcement learning. It surpassed prior algorithms and reached a level comparable to a professional human tester across 49 games with one set-up.
Checked: PDF from storage.googleapis.com, fetched 4 Oct 2026. Page numbers are those of that PDF.
3 findings
Substance 'Our DQN method outperforms the best existing reinforcement learning methods on 43 of the games': against the better of the two literature columns in Extended Data Table 2 (Best Linear Learner, ref. 12; Contingency (SARSA), ref. 15), DQN wins 42 of 49. The count of 43 holds only against the Best Linear Learner alone.
- p. 2 Main text, right column
Our DQN method outperforms the best existing reinforcement learning methods on 43 of the games
- p. 11 Extended Data Table 2
Kung-Fu Master 258.5 19544 29151 22736 23270 (±5955)
- p. 2 Main text, right column
outperforms the best existing reinforcement learning methods on 43 of the games
- p. 11 Extended Data Table 2, Kung-Fu Master row
29151
Only the authors can say what is right here.
Judge's note: I went through all 49 rows of ED Table 2 (p.11). Against max(Linear, SARSA), DQN is lower in Bowling (42.4<43.9), Centipede (8309<8803), Double Dunk (-18.1<-13.1), Gravitar (306.7<429), Kung-Fu Master (23270<29151 SARSA), Montezuma's Revenge (0<259) and Tennis (-2.5<0): 7 losses. Against Linear alone, Kung-Fu Master becomes a win (23270>19544): 6 losses. The preceding sentence defines the comparison set as the best methods 'where results were available12,15'. Fig. 3's grey bars also appear to include the SARSA value for Kung-Fu Master (grey bar longer than the 102% DQN bar). I found no statement restricting the count to the linear learner. Only the authors can say which comparison was meant.
Second check: another model agreed
Report a problem with this finding
Reference or label Text says '(Extended Data Table 2 and Fig. 3, denoted by 100% (human) and 0% (random) on y axis', but in Fig. 3 the percentage scale (0 … 600, 1,000, 4,500%) is on the horizontal (x) axis.
- p. 2 Main text, right column, comparison paragraph
denoted by 100% (human) and 0% (random) on y axis
- p. 3 Figure 3
4,500%
Judge's note: Page 3 image: Fig. 3 is a horizontal bar chart with game names on the vertical axis and the normalized percentage scale along the bottom (horizontal) axis. The text on p.2 names the wrong axis.
Second check: another model agreed
Report a problem with this finding
Reference or label Fig. 3 legend labels the grey bars 'Best linear learner', but for several games (Video Pinball, Kung-Fu Master, Gopher, James Bond) the grey bar lengths match the Contingency (SARSA) values of Extended Data Table 2, not the Best Linear Learner values.
- p. 3 Figure 3 legend
Best linear learner
- p. 11 Extended Data Table 2, Video Pinball row
Video Pinball 16257 16871 19761 17298
Only the authors can say what is right here.
Judge's note: I measured on the p.3 image (0 at x≈302, 100% ≈44 px). The Video Pinball grey bar ends near x≈450 (≈330%): SARSA gives 337%, linear gives 59%. The Kung-Fu Master grey bar extends beyond the 102% DQN bar to ≈127%: SARSA 128.5%, linear 85.8%. Gopher ≈102% (SARSA 102.3%, linear 49.9%). James Bond ≈86% (SARSA 86%, linear 46%). Games where linear is higher (Atlantis ≈308%, Krull ≈223%, Double Dunk ≈177%) match the linear values. The grey bars look like the better of the two literature methods, but the legend says only 'Best linear learner'.
Second check: another model agreed
Report a problem with this finding
7 minor (typos and the like; most checked by the judge only)
Reference or label Fig. 3 title 'Comparison of the DQN agent with the best reinforcement learning methods15 in the literature' cites only ref. 15, while its legend series 'Best linear learner' is credited to ref. 12 in the Extended Data Table 2 footnote.
- p. 3 Figure 3 caption title and legend
Comparison of the DQN agent with the best reinforcement learning methods15 in the literature
- p. 11 Extended Data Table 2 footnote
Best Linear Learner is the best result obtained by a linear function approximator on different types of hand designed features12
Only the authors can say what is right here.
Judge's note: ED Table 2 footnote (p.11): 'Best Linear Learner is the best result obtained by a linear function approximator on different types of hand designed features12. Contingency (SARSA) agent figures are the results obtained in ref. 15.' The Fig. 3 caption cites only ref. 15. Per C008, the grey bars appear to mix values from both sources, so the citation is incomplete rather than simply wrong. The reviewer's premise that the bars are linear-only does not hold. It should be cited as refs 12,15.
Second check: another model agreed
Report a problem with this finding
Reference or label Ref. 33 reads 'Kaelbling, L. P., Littman, M. L. & Cassandra, A. R. Planning and acting in partially observable stochastic domains. Artificial Intelligence 101, 99–134 (1994).'
- p. 7 Reference 33
Artificial Intelligence 101, 99–134 (1994)
Only the authors can say what is right here.
Judge's note: Rests on outside knowledge: that Artificial Intelligence vol. 101 and this article date from 1998, not 1994. The page 7 image shows '(1994)' as quoted. A 1994 technical-report version may explain the date, but the entry cites the journal volume and pages.
Second check: another model agreed
Report a problem with this finding
Wording Abstract and Discussion say the agent was 'receiving only the pixels and the game score as inputs', but the Methods list the minimal prior knowledge as images, score, number of actions 'and the life count', and say the number of lives 'is then used to mark the end of an episode during training'.
- p. 1 Abstract
receiving only the pixels and the game score as inputs
- p. 4 Discussion
receiving only the pixels and the game score as inputs
- p. 6 Methods, Training details / prior knowledge
number of lives left in the game, which is then used to mark the end of an episode during training
Only the authors can say what is right here.
Judge's note: Page 6 Methods, Training details: the emulator 'sends the number of lives left in the game, which is then used to mark the end of an episode during training'. The prior-knowledge paragraph explicitly includes 'the life count'. The life count is not an input to the network, but the unqualified 'only the pixels and the game score' does not match the Methods' own list of what the agent uses. This is a wording/disclosure inconsistency.
Second check: another model agreed
Report a problem with this finding
Wording Methods: 'clipping the squared error to be between −1 and 1 corresponds to using an absolute value loss function for errors outside of the (−1,1) interval', whereas the preceding sentence clips the error term r+γ max Q(s′,a′;θ⁻)−Q(s,a;θ).
- p. 7 Methods, Training algorithm, error clipping paragraph
clipping the squared error to be between −1 and 1
Judge's note: Page 7 image confirms the wording. A squared error is non-negative, so clipping it to [−1,1] is not the operation described. The stated equivalence to an absolute-value loss outside (−1,1) holds for clipping the error term, i.e. the derivative of the squared loss, as the preceding sentence says. This is loose wording with an evident fix ('clipping the error term').
Second check: another model agreed
Report a problem with this finding
Typo Main text: 'more than 75% of the human score on more than half of the games (29 games'; Fig. 3 caption: 'operationalized as a level of 75% or above'.
- p. 2 Main text
than 75% of the human score on more than half of the games (29 games
- p. 3 Figure 3 caption
operationalized as a level of 75% or above
Judge's note: The threshold is written as > on p.2 and as ≥ in the Fig. 3 caption (p.3). In ED Table 2 no game sits at exactly 75% (lowest above is H.E.R.O. 76.5%, highest below is Asterix 70.0%), so the count of 29 is the same either way. This is only a wording inconsistency.
Second check: none (judge only)
Report a problem with this finding
Typo Ref. 16 reads 'McClelland, J. L., Rumelhart, D. E. & Group, T. P. R.', which renders the corporate author 'the PDP Research Group' as a person 'Group, T. P. R.'.
- p. 5 Reference 16
McClelland, J. L., Rumelhart, D. E. & Group, T. P. R.
Judge's note: Page 5 image confirms the text. The initials T. P. R. come from 'The PDP Research' parsed as a personal name; this is a bibliographic formatting slip.
Second check: none (judge only)
Report a problem with this finding
Typo Ref. 20 reads 'Tsitsiklis, J. & Roy, B. V.', which renders the surname Van Roy as 'Roy' with initial 'V.'.
- p. 5 Reference 20
Tsitsiklis, J. & Roy, B. V.
Only the authors can say what is right here.
Judge's note: Rests on outside knowledge: that the co-author's surname is 'Van Roy' (Benjamin Van Roy). The page 5 image shows 'Tsitsiklis, J. & Roy, B. V.' as quoted.
Second check: none (judge only)