Continuous Control with Deep Reinforcement Learning
arXiv (Cornell University), 2015 · paper · AI and machine learning
Adapts deep Q-learning ideas to continuous actions with a model-free actor-critic algorithm based on the deterministic policy gradient. One configuration solved over 20 simulated physics tasks, matched a planner with full knowledge of the dynamics, and often learned from raw pixels.
Checked: PDF from arxiv.org, fetched 4 Oct 2026. Page numbers are those of that PDF.
3 findings
Substance Supplement section 7: the pixel-case final-layer initialisation is printed as "[3 × 10−4, 3 × 10−4]", a degenerate interval. By parallel with "[−3 × 10−3, 3 × 10−3]" it presumably should be [−3 × 10−4, 3 × 10−4].
- p. 11 Section 7 Experiment Details
[3 × 10−4, 3 × 10−4]
- p. 11 Sec. 7 Experiment Details
[3 × 10−4, 3 × 10−4] for the low dimensional and pixel cases respectively
- p. 11
[3 × 10−4, 3 × 10−4] for the
Judge's note: The p11 image shows the low-dimensional interval with a minus sign on its lower bound and the pixel interval with none. So this is not an extraction artefact. As printed, the range has identical bounds.
Second check: another model agreed
Report a problem with this finding
Wording p.4: batch normalization "normalizes each dimension across the samples in a minibatch to have unit mean and variance". It normalizes to zero mean and unit variance.
- p. 4 Section 3, batch normalization paragraph
across the samples in a minibatch to have unit mean and variance
Judge's note: Page 4 image confirms the wording. Later on the same page the paper itself says each layer receives 'whitened input', which means zero mean. This is a misdescription of the cited technique.
Second check: another model agreed
Report a problem with this finding
Wording Sec. 9.2 says "below is a summary of all the physics environments", but the description table (pp. 12–14) has no entries for cartpoleParallelTriple, hardCheetahNice, hyqKick or reacherDouble, which are listed in Table 2.
- p. 12 Section 9.2
below is a summary of all the physics environments
- p. 14 Environment description table
reacherObstacle
Only the authors can say what is right here.
Judge's note: I compared the Table 2 task list on p.12 with the description table on pp.12–14. The four named tasks are in Table 2 but have no description. A text search shows that these names appear only in Table 2.
Second check: another model agreed
Report a problem with this finding
25 minor (typos and the like; most checked by the judge only)
Reference or label The reference list has two "Heess ... 2015" entries ("Memory-based control with recurrent neural networks" and "Learning continuous control policies by stochastic value gradients") without a/b suffixes, so "Heess et al. (2015)" in the text is ambiguous.
- p. 8 Section 5
Heess et al. (2015) introduced SVG(0)
- p. 9 References
Memory-based control with recurrent neural networks
- p. 9 References
Heess, N., Hunt, J. J, Lillicrap, T. P, and Silver, D. Memory-based control with recurrent neural networks
Judge's note: P9 shows both entries without suffixes, while Koutník 2014a/b and Schulman 2015a/b are disambiguated. On p8 the text cites "Heess et al. (2015)" twice. From context these refer to the SVG paper, but the citation key does not tell the reader which entry is meant.
Second check: another model agreed
Report a problem with this finding
Wording p.1: "leads to an action space with dimensionality: 3^7 = 2187". 2187 is the number of discrete actions, not the dimensionality of the space.
- p. 1 Introduction, para 3
leads to an action space with dimensionality: 37 = 2187
Judge's note: Page 1 image: the preceding clause correctly speaks of 'the number of actions'. The arithmetic is correct. The loose use of 'dimensionality' could be read as the size of the DQN output layer, so this is a minor wording point.
Second check: another model agreed
Report a problem with this finding
Wording Table 1 is captioned "Performance after training across all environments" and the page-5 text says it covers "all of the environments". It lists 26 MuJoCo tasks, but Table 2 lists 32. cartpoleParallelTriple, hardCheetahNice, hyqKick, movingGripperRandom, reacherDouble and reacherObstacle have no results.
- p. 7 Table 1 caption
Performance after training across all environments for at most 2.5 million steps.
- p. 12 Table 2
cartpoleParallelTriple
- p. 2 Introduction, paragraph 3
attempted all tasks using both low-dimensional observations (e.g. joint angles) and directly from pixels
- p. 7 Table 1 caption
Performance after training across all environments for at most 2.5 million steps
Only the authors can say what is right here.
Judge's note: I counted the rows on the page images. Table 1 (p7) has 26 physics rows plus torcs. Table 2 (p12), described as "a summary of all the physics environments", has 32 rows. I found no statement explaining why six tasks are left out. Only the authors can say whether these tasks were not run or their results were omitted.
Second check: another model agreed
Report a problem with this finding
Wording p.4: µ′ first denotes the target actor network "µ′(s|θ^µ′)", then is reused in "We constructed an exploration policy µ′ by adding noise" (Eq. 7, µ′(s_t) = µ(s_t|θ^µ_t) + 𝒩).
- p. 4 Section 3, target networks paragraph and Eq. (7)
We constructed an exploration policy µ′ by adding noise
Judge's note: Page 4 image confirms both uses on the same page. Algorithm 1 uses µ′ only for the target network. Context makes each meaning recoverable, so this is a notation clash only.
Second check: another model agreed
Report a problem with this finding
Wording Page 5: "In all domains but cheetah the actions were torques applied to the actuated joints", but the supplement says the cartpole tasks are solved "by applying forces to the cart alone".
- p. 5 Section 4, first paragraph
In all domains but cheetah the actions were torques applied to the actuated joints.
- p. 13 Environment descriptions, cartpole
by applying forces to the cart alone
Judge's note: Page 5 has the general claim. The page-13 descriptions of cartpole and cartpoleBalance say forces are applied to the cart, and cart "move[s] a simple mass". 'Torques' may be meant loosely as generalized forces, and the sentence sits in the MuJoCo paragraph, so Torcs is arguably out of its scope. Still, the wording does not match the supplement for the cart tasks. This is a minor wording inconsistency.
Second check: another model agreed
Report a problem with this finding
Wording The Conclusion claims stable learning "without the need for any modifications between environments", but page 6 says that for Torcs they "altered the noise process for exploration because of the very different time scales involved".
- p. 8 Section 6 Conclusion
without the need for any modifications between environments
- p. 6 Section 4 Results, Torcs paragraph
but altered the noise process for exploration because of the very different time scales involved
Only the authors can say what is right here.
Judge's note: Both passages were checked on the page images (p8 Conclusion, p6 Torcs paragraph). The Conclusion makes no exception for Torcs, which is reported in Table 1, so the claim is broader than the methods described.
Second check: another model agreed
Report a problem with this finding
Typo Copy-edit slips: p.1 "domains is to to simply discretize" (duplicated 'to'); also Table 1 caption (p.7) "DDPG algorithn" and "For comparision", and Figure 1 caption (p.6) "a gasp and move task" (for 'grasp').
- p. 1 Introduction, para 3
domains is to to simply discretize the action space
- p. 1 Introduction para 3
is to to simply discretize the action space
- p. 7 Table 1 caption
DDPG algorithn
- p. 7 Table 1 caption
comparision
Judge's note: Page 1 image shows "is to to simply discretize". The crop of the Table 1 caption shows "algorithn" and "comparision", and the Figure 1 crop shows "gasp and move". The stray article "a finding" and the garbled "following the applying" are judged on their own in C002 and C007.
Second check: none (judge only)
Report a problem with this finding
Typo Stray article on p.1: "since it relies on a finding the action that maximizes the action-value function" should read 'relies on finding'.
- p. 1 Introduction, para 2
it relies on a finding the action that maximizes
- p. 1 Introduction, paragraph 2
since it relies on a finding the action that maximizes the
Judge's note: Confirmed in the page 1 image, Introduction paragraph 2.
Second check: none (judge only)
Report a problem with this finding
Typo p.3: "we can describe it as a function µ : S ← A". The arrow points left, but a deterministic policy maps states to actions (S → A), consistent with π: S → P(A) on p.2.
- p. 3 Text before Eq. (3)
µ : S ← A
Judge's note: Page 3 image confirms the left-pointing arrow before Eq. (3). The intended meaning is clear from context, so this is a typographic slip.
Second check: none (judge only)
Report a problem with this finding
Typo p.3, Section 3: "The actor is updated by following the applying the chain rule" is garbled (leftover from an edit).
- p. 3 Section 3, second paragraph
The actor is updated by following the applying the chain rule
- p. 3 Section 3
by following the applying the chain rule
Judge's note: Confirmed in the page 3 image.
Second check: none (judge only)
Report a problem with this finding
Typo Footnote 2, p.3: "as in commonly done in policy gradient implementations" should read 'as is commonly done'.
- p. 3 Footnote 2
as in commonly done in policy gradient implementations
Judge's note: Confirmed in the page 3 image.
Second check: none (judge only)
Report a problem with this finding
Typo p.4: "it is used to minimize covariance shift during training" should be 'covariate shift'.
- p. 4 Section 3, batch normalization paragraph
it is used to minimize covariance shift during training
Judge's note: Page 4 image confirms 'covariance shift'. The paper's own reference entry for Ioffe & Szegedy reads '...reducing internal covariate shift'.
Second check: none (judge only)
Report a problem with this finding
Typo Page 4: "An advantage of off-policies algorithms such as DDPG" should read "off-policy algorithms".
- p. 4 Section 3, exploration paragraph
An advantage of off-policies algorithms such as DDPG
- p. 4 exploration paragraph
off- policies algorithms such as DDPG
Judge's note: The page-4 crop shows "off- / policies algorithms such as DDPG" exactly as quoted. This is a grammatical slip.
Second check: none (judge only)
Report a problem with this finding
Typo Page 6: "DDPG is still able learn good policies" is missing "to".
- p. 6 Section 4, Q-estimates paragraph
DDPG is still able learn good policies
Judge's note: The page-6 image shows the text exactly as quoted.
Second check: none (judge only)
Report a problem with this finding
Typo Figure 1 caption: "a gasp and move task" should read "a grasp and move task".
- p. 6 Figure 1 caption
a gasp and move task
Judge's note: The Figure 1 caption on page 6 reads "a gasp and move task".
Second check: none (judge only)
Report a problem with this finding
Typo Table 1 caption: "the DDPG algorithn" should read "algorithm".
- p. 7 Table 1 caption
DDPG algorithn
- p. 7 Table 1 caption
DDPG algorithn in the low-dimensional
- p. 7 Table 1 caption
For comparision we also include
Judge's note: The page-7 crop shows "algorithn". The "comparision" misspelling is treated in C022.
Second check: none (judge only)
Report a problem with this finding
Typo Table 1 caption: "For comparision we also include" should read "comparison".
- p. 7 Table 1 caption
For comparision we also include
Judge's note: The page-7 image shows "comparision".
Second check: none (judge only)
Report a problem with this finding
Typo Conclusion: "our experimental results demonstrate that stable learning without the need for any modifications between environments." The clause has no verb (e.g. "is possible").
- p. 8 Section 6 Conclusion, first paragraph
our experimental results demonstrate that stable learning without the need for any modifications between environments.
- p. 8 Conclusion
our experimental results demonstrate that stable learning without the need for any modifications between environments
Judge's note: The page-8 image shows the sentence ending at "environments." with no predicate and no missing line.
Second check: none (judge only)
Report a problem with this finding
Typo Glorot et al. 2011 reference: "JMLR W&CP Volume, volume 15" repeats "Volume".
- p. 9 References, Glorot et al. 2011
JMLR W&CP Volume, volume 15
Judge's note: The p9 image shows the text as quoted.
Second check: none (judge only)
Report a problem with this finding
Typo Misspelling in Sec. 9.2: "simple threshholds on the height" should be "thresholds".
- p. 12 Section 9.2
simple threshholds on the height
Judge's note: The p.12 image reads "threshholds".
Second check: none (judge only)
Report a problem with this finding
Typo Misspelling in the canada2d description: "a ball initialzed to a random start location" should be "initialized".
- p. 13 canada2d description
a ball initialzed to a random start location
- p. 13 canada2d
initialzed to a random start location
- p. 12 Sec. 9.2
simple threshholds on the height
- p. 13 gripper description
manuver the object
Judge's note: The p.13 image reads "initialzed". The other misspellings grouped in this cluster (threshholds, manuver, stabalizing, quadroped) are handled under C033, C035, C036 and C037.
Second check: none (judge only)
Report a problem with this finding
Typo Misspelling in the gripper description: "manuver the object to a fixed target" should be "maneuver".
- p. 13 gripper description
manuver the object to a fixed target
- p. 13 gripper
manuver the object
Judge's note: The p.13 image reads "manuver".
Second check: none (judge only)
Report a problem with this finding
Typo Misspelling in the hardCheetah description: "the stabalizing joint stiffness" should be "stabilizing".
- p. 13 hardCheetah description
the stabalizing joint stiffness
- p. 13 hardCheetah
stabalizing joint stiffness
Judge's note: The p.13 image reads "stabalizing".
Second check: none (judge only)
Report a problem with this finding
Typo Misspelling in the hyq description: "keep a quadroped model based on the hyq robot" should be "quadruped".
- p. 13 hyq description
keep a quadroped model based on the hyq robot
- p. 13 hyq
quadroped model
Judge's note: The p.13 image reads "quadroped".
Second check: none (judge only)
Report a problem with this finding
Typo cartpoleBalance description: "The pole starts in the upright positions at the beginning of each episode" should read "upright position" (there is a single pole).
- p. 13 cartpoleBalance
upright positions at the beginning of each episode
Judge's note: The p.13 image shows the plural "positions" for a single pole. This is a grammatical slip only.
Second check: none (judge only)