Trust Region Policy Optimization
arXiv (Cornell University), 2015 · paper · AI and machine learning
Describes an iterative policy-optimization procedure with guaranteed monotonic improvement, and a practical approximation called Trust Region Policy Optimization. It performed robustly on simulated robot swimming, hopping and walking and on Atari games, tending to improve steadily with little hyperparameter tuning.
Checked: PDF from arxiv.org, fetched 4 Oct 2026. Page numbers are those of that PDF.
9 findings
Substance Section 8.1 gives "Walker. 18-dimensional state space", but Table 2 lists State space dim. for Walker as 20 (Swimmer 10 and Hopper 12 agree with the text).
- p. 7 Section 8.1, model list item 3
Walker. 18-dimensional state space.
- p. 15 Table 2, State space dim., Walker column
20
- p. 15 Table 2 (Appendix E), State space dim.
State space dim.
Only the authors can say what is right here.
Judge's note: Page 7 shows '18-dimensional state space' for the walker. The page 15 Table 2 'State space dim.' row reads 10/12/20. The text points to Table 2 for details of the same setup, and no different state representation is described anywhere. Only the authors can say which value is correct.
Second check: another model agreed
Report a problem with this finding
Substance Figure 4 caption: "for the hopper and walker, a score of −1 is achievable without any forward velocity", but the reward is v_x − 10^-5||u||^2 with "a bonus of +1 for being in a non-terminal state", so standing still scores about +1.
- p. 7 Figure 4 caption
a score of −1 is achievable without any forward velocity
- p. 7 Section 8.1, Hopper description
same reward as the swimmer, with a bonus of +1 for being in a non- terminal state
- p. 7 Section 8.1, Hopper description
with a bonus of +1 for being in a non-terminal state
Only the authors can say what is right here.
Judge's note: The C014-a crop shows a printed minus sign. The hopper and walker axes are labelled 'reward', and the non-walking curves (CEM, natural gradient) level off near +1, not −1. With v_x=0 and small u, the per-step reward is about +1.
Second check: another model agreed
Report a problem with this finding
Substance Lemma 3, Eq. (32): the intermediate bound '|E_π̃[Ā(s_t)] − E_π[Ā(s_t)]| ≤ 2α max_s Ā(s)' is never proved and is false as printed; the proof (Eqs. 36–40) yields only the outer bound 4α(1−(1−α)^t) max|A|.
- p. 11 Lemma 3, Eq. (32)
≤ 2α max s ¯A(s) ≤ 4α(1 −(1 −α)t) max s |Aπ(s, a)|
- p. 11 Lemma 3, Eq. (32)
≤ 2α max_s Ā(s) ≤ 4α(1 − (1 − α)^t) max_s|A_π(s, a)|
Only the authors can say what is right here.
Judge's note: Reworked on p11. The proof gives LHS = P(n_t>0)|…| ≤ (1−(1−α)^t)·4α max|A|. It never uses or derives 2α max_s Ā(s). That term lacks the absolute value and the (1−(1−α)^t) factor, so it can be 0 while LHS > 0. I checked the reviewer's counterexample. States s0 (start, r=0) and s1 (absorbing, r=1); at s0, a1→s1 and a2→s0; π picks a1; π̃ picks a2 w.p. α; the policies are identical at s1; P(a≠ã|s) ≤ α, so the pair is α-coupled. Then V(s1)=1/(1−γ), V(s0)=γ/(1−γ), A(s0,a1)=0, A(s0,a2)=γ²/(1−γ)−γ/(1−γ)=−γ, Ā(s0)=−αγ and Ā(s1)=0. So max_s Ā = 0. At t=1, E_π[Ā(s1)]=0 and E_π̃[Ā(s1)]=α·(−αγ), so LHS = α²γ > 0, which is a contradiction. The outer bound 4α²γ still holds, so Eq. (40) and Theorem 1 are unaffected. The fix is probably to delete the middle term; the authors should confirm.
Second check: another model agreed
Report a problem with this finding
Reference or label Figure 4 Swimmer panel y-axis labelled "cost (-velocity + ctrl)", yet the curves rise during training and the text calls TRPO (highest curves, ~0.15) the best; the plotted quantity behaves as reward (velocity − ctrl).
- p. 7 Figure 4, Swimmer panel y-axis
cost (-velocity + ctrl)
- p. 7 Section 8.1 results text
Single path and vine TRPO solved all of the problems, yielding the best solutions.
Judge's note: In the C013-a crop the y ticks run from -0.10 up to 0.15, all methods increase over iterations, and Vine/Single Path end highest. The text says single path and vine 'solved all of the problems, yielding the best solutions'. If the axis were a cost, higher would be worse. The swimmer reward is defined as v_x − 10^-5||u||^2.
Second check: another model agreed
Report a problem with this finding
Reference or label In the proof of Lemma 3: 'Where the second inequality follows from Lemma 3.' The bound used (|Ā(s)| ≤ 2α max|A|) is Lemma 2, Eq. (28).
- p. 11 Proof of Lemma 3, after Eq. (39)
Where the second inequality follows from Lemma 3.
Judge's note: Checked on p11. Step (38)→(39) applies Eq. (28) of Lemma 2 to each term: 2α max|A| + 2α max|A| = 4α max|A|. Citing Lemma 3 inside its own proof is a mis-numbered pointer. The step itself is valid with Lemma 2.
Second check: another model agreed
Report a problem with this finding
Wording Introduction says "we first prove that minimizing a certain surrogate objective function guarantees policy improvement", but the surrogate M_i is maximized throughout (Algorithm 1 arg max; "by maximizing M_i at each iteration").
- p. 1 Introduction, paragraph 3
we first prove that minimizing a certain sur- rogate objective function guarantees policy improvement
- p. 3 Section 3, after Eq. (10)
Thus, by maximizing Mi at each iteration, we guarantee
- p. 3 Section 3, after Eq. (10)
Thus, by maximizing Mi at each iteration
Judge's note: The page 1 crop confirms 'minimizing'. On page 3, Algorithm 1 uses arg max over [L - C D_KL^max], and the text after Eq. (10) says 'by maximizing M_i'. The MM framing (the surrogate minorizes eta) needs maximization. No negated loss is defined anywhere.
Second check: another model agreed
Report a problem with this finding
Wording Eq. (41) prints "|η(π̃) − Lπ(π̃)| = Σ_t γ^t |E_{τ∼π̃}[Ā(s_t)] − E_{τ∼π}[Ā(s_t)]|" as an equality, but subtracting (26) and (27) gives only ≤ (triangle inequality).
- p. 12 Appendix A, Eq. (41)
|η(π̃) − L_π(π̃)| = Σ γ^t |E_{τ∼π̃}[Ā(s_t)] − E_{τ∼π}[Ā(s_t)]|
Only the authors can say what is right here.
Judge's note: Page 12 image: Eq. (41) has '=' with the absolute value inside the sum. Writing d_t for the per-timestep difference, η − L = Σγ^t d_t, so |η − L| = |Σγ^t d_t| ≤ Σγ^t|d_t|. Equality holds only when all the d_t have the same sign. The reviewer's counterexample checks out: d1 = α·Ā(sB) > 0 and d2 = α(1−α)·Ā(sX) < 0, and the two sides come to 0.15625 and 0.21875. Line (42) is already an inequality, so writing ≤ in (41) repairs the chain and Theorem 1 still holds.
Second check: another model agreed
Report a problem with this finding
Wording In Eq. (50), lines 2–4 carry the opposite sign to lines 1 and 5: line 1 has (π̃(a|s) − π(a|s)), but line 2 has "(πθ(a|s) − πθ̃(a|s))", line 3 "(π(a|s) − π̃(a|s))" and line 4 "(pπ(s′|s) − pπ̃(s′|s))", while the final line is γvΔρπ with Δ = Pπ̃ − Pπ.
- p. 13 Appendix B, Eq. (50), lines 1, 3–5
(pπ(s′|s) −p˜π(s′|s))γv(s′)
Only the authors can say what is right here.
Judge's note: Page 13 image checked. Section 4 defines Lθ(θ̃) := Lπθ(πθ̃), so πθ is the old policy π and line 2's factor is (π − π̃), the reverse of line 1. Lines 2→3→4 are consistent with each other (r(s) and v(s) drop out because Σ_a(π−π̃) = 0). Line 4, however, equals γv(Pπ − Pπ̃)ρπ = −γvΔρπ, and line 5 silently restores the sign. Line 1 equal to line 5 is correct, so the stated identity and Theorem 1 still hold; restoring (π̃ − π) in lines 2–4 repairs the chain.
Second check: another model agreed
Report a problem with this finding
Wording In Eq. (54) the second line "≤ γ‖vΔ‖∞‖GΔG̃ρ‖1" drops a factor γ (line 1 is γ‖γrGΔ‖∞ = γ‖γvΔ‖∞). The next step then uses 2αε, but Eq. (51) bounds only |(γvΔ)_s|, not ‖vΔ‖∞.
- p. 13 Appendix B, Eq. (54), second and third lines
≤γ∥v∆∥∞∥G∆˜Gρ∥1
- p. 13 Appendix B, Eq. (54), lines 1–3
≤ γ‖vΔ‖∞‖GΔG̃ρ‖1
Only the authors can say what is right here.
Judge's note: Page 13 image checked. Line 1 ≤ line 2 holds (γ² ≤ γ), but line 2 ≤ line 3 fails. My own counterexample: state s0 has two actions; a1 is a self-loop and a2 moves to an absorbing state z. r(s0) = 1, r(z) = 0, γ = 0.4. π̃ always takes a1; π takes a2 with probability α = 0.1. ρ = e_s0. Then v(s0) = 1/(1 − 0.36) = 1.5625, A(a1) = 0.0625, A(a2) = −0.5625, ε = 0.5625, ‖vΔ‖∞ = 0.15625 (> 2αε = 0.1125), and ‖GΔG̃ρ‖1 = 0.5208. Line 2 = 0.0326 > line 3 = 0.025. Going straight from line 1 to line 3 via (51) is valid, so (54) and Theorem 1 still hold; writing γ‖γvΔ‖∞ in line 2 fixes the step.
Second check: another model agreed
Report a problem with this finding
26 minor (typos and the like; most checked by the judge only)
Wording Algorithm 1 says "Solve the constrained optimization problem" for π_{i+1} = arg max_π [L_{π_i}(π) − C D_KL^max(π_i, π)], which is an unconstrained penalty problem; the text then contrasts TRPO as using "a constraint on the KL divergence rather than a penalty".
- p. 3 Algorithm 1
Solve the constrained optimization problem
- p. 3 Section 3, last paragraph
which uses a constraint on the KL divergence rather than a penalty
Judge's note: Page 3 image: the problem in Algorithm 1 has no constraint; C is a penalty coefficient. The paragraph below says TRPO differs from Algorithm 1 by using a constraint 'rather than a penalty', so the word 'constrained' in Algorithm 1 is inconsistent with that. One could argue the optimization is over valid policies, so this is minor.
Second check: another model agreed
Report a problem with this finding
Wording Section 5.2: "In large or continuous state spaces, we can construct an estimator of the surrogate objective using importance sampling" follows the case of "small, finite action spaces"; the contrast evidently concerns action spaces.
- p. 5 Section 5.2, after Eq. (15)
In large or continuous state spaces, we can construct an estima- tor of the surrogate objective using importance sampling.
Only the authors can say what is right here.
Judge's note: Page 5 image: Eq. (15) enumerates all K actions when the action space is finite. Eq. (16) replaces that enumeration with self-normalized importance sampling over the sampled actions. The size of the state space has no bearing on this. 'State' is most likely a slip for 'action'; only the authors can confirm.
Second check: another model agreed
Report a problem with this finding
Wording Figure 3 (top), described as the network for "the locomotion task", labels the hidden layer "30 units", while Table 2 gives hidden layer size 30 (Swimmer) but 50 (Hopper, Walker).
- p. 6 Figure 3 (top), locomotion network
30 units
- p. 15 Table 2, Hidden layer size
50
Judge's note: Page 6: Figure 3 shows '30 units'. Page 7: the text says the policies use 'the architecture shown in Figure 3'. Page 15: Table 2's hidden layer size row reads 30/50/50. The figure is plausibly a schematic, but the unit count it prints fits only the swimmer.
Second check: another model agreed
Report a problem with this finding
Wording Figure 4 legends in all four panels include "RWR", but the list "The following algorithms were considered in the comparison" (single path, vine, CEM, CMA, natural gradient, empirical FIM, max KL) never mentions, defines or cites RWR.
- p. 7 Figure 4 legends (all four panels)
RWR
- p. 7 Section 8.1, list of algorithms
The following algorithms were considered in the compari- son: single path TRPO; vine TRPO;
- p. 7 Section 8.1, list of algorithms
The following algorithms were considered in the comparison
Only the authors can say what is right here.
Judge's note: The page 7 image shows RWR in every legend. A grep of extract/main.txt finds 'RWR' only in the figure legend text (4 hits) and no 'reward-weighted' anywhere, so the method is undefined in the paper.
Second check: another model agreed
Report a problem with this finding
Wording Figure 4 Swimmer panel x-axis 'number of policy iterations' runs 0–50, while Table 2 lists 'Policy iter.' = 200 for Swimmer (Hopper/Walker panels do run to 200).
- p. 7 Figure 4, Swimmer panel x-axis
number of policy iterations
- p. 15 Table 2, Policy iter., Swimmer
200
Only the authors can say what is right here.
Judge's note: Viewed Fig. 4 on p7: the Swimmer axis ends at 50, and the Hopper and Walker axes end at 200. Table 2 on p15 gives 200 policy iterations for all three tasks. Neither the caption nor the text says the Swimmer curve is truncated. Only the authors can say whether the plot is truncated or the table entry is wrong.
Second check: another model agreed
Report a problem with this finding
Wording Appendix B states "‖G‖1 = ‖G̃‖1 = 1/(1 − γ) and ‖Δ‖1 = 2α". In general only ‖Δ‖1 ≤ 2α holds.
- p. 13 Appendix B, text after Eq. (52)
‖G‖1 = ‖G̃‖1 = 1/(1 − γ) and ‖Δ‖1 = 2α
Only the authors can say what is right here.
Judge's note: Page 13 image checked. ‖Δ‖1 = max_s Σ_s′ |Σ_a (π̃ − π)(a|s) p(s′|s,a)| ≤ max_s Σ_a |π̃ − π| ≤ 2α. If all actions have the same transition distribution, then Δ = 0 while α > 0, so the equality is false as printed. Eq. (53) uses only the upper bound, so the result stands. The statement ‖G‖1 = 1/(1 − γ) is correct for stochastic P.
Second check: another model agreed
Report a problem with this finding
Wording Table 2: for Walker, "SP: num. path 10000" × "SP: path len. 1000" = 10M, but "Sim. steps per iter." is 1M. For Swimmer (50×1000 = 50K) and Hopper (1000×1000 = 1M) the product matches.
- p. 15 Table 2, SP: num. path / SP: path len. / Sim. steps per iter., Walker
10000
- p. 15 Table 2
SP: num. path 50 1000 10000
- p. 15 Table 2
Sim. steps per iter. 50K 1M 1M
Only the authors can say what is right here.
Judge's note: Page 15 Table 2 image checked. Path length may be a maximum (episodes can end early), which the table does not say. Even so, the computation times (Walker/Hopper ≈ 2.9 for both vine and SP: 40/14 and 100/35) suggest similar sample counts, which fits 1000 paths better than 10000. Only the authors can say which entry is right.
Second check: another model agreed
Report a problem with this finding
Typo First-page footnote reads "Proceedings of the 31st International Conference on Machine Learning, Lille, France, 2015"; the 2015 Lille meeting appears to be the 32nd ICML.
- p. 1 First-page footnote
Proceedings of the 31 st International Conference on Machine Learning, Lille, France, 2015
- p. 9 Reference list, Pirotta et al.
Proceedings of The 30th International Conference on Machine Learning, pp. 307– 315, 2013
Only the authors can say what is right here.
Judge's note: Rests on outside knowledge: ICML is held once a year, and the 2015 Lille meeting (W&CP vol. 37) was the 32nd. The page 1 image shows '31st ... Lille, France, 2015. JMLR: W&CP volume 37'. The paper's own reference list (Pirotta et al.) gives the 30th ICML as 2013, which also points to the 32nd for 2015. This looks like a leftover from a template.
Second check: none (judge only)
Report a problem with this finding
Typo Inconsistent method name: 'covariance matrix adaption (CMA)' (p7) vs 'covariance matrix adaptation (CMA)' (p1).
- p. 7 Section 8.1, algorithm list
covariance matrix adaption (CMA)
- p. 1 Introduction
covari- ance matrix adaptation (CMA)
Judge's note: Both forms appear on the page images. 'Adaption' is a recognised variant spelling, but the paper uses both forms, and the method's standard name uses 'adaptation'.
Second check: none (judge only)
Report a problem with this finding
Typo After Eq. (4): "a sufficiently small step π_θ0 → π̃ that improves L_{π_θold} will also improve η" mixes θ0 (from Eq. 4) with θold, which is not introduced until Section 4.
- p. 2 Paragraph after Eq. (4)
a sufficiently small step πθ0 →˜π that improves Lπθold will also improve η
Judge's note: The page 2 crop shows Eq. (4) written at θ0 and the next sentence using π_θ0 and then L_{π_θold}. θold is first defined on page 3, Section 4. This is a notational inconsistency only.
Second check: none (judge only)
Report a problem with this finding
Typo Eq. (10) reads "η(π_{i+1}) − η(π_i) ≥ M_i(π_{i+1}) − M(π_i)"; the last term should be M_i(π_i).
- p. 3 Eq. (10)
η(πi+1) −η(πi) ≥Mi(πi+1) −M(πi).
- p. 3 Eq. (10)
Mi(πi+1) −M(πi)
- p. 3 Section 3, Eq. (10)
η(π_{i+1}) − η(π_i) ≥ M_i(π_{i+1}) − M(π_i)
Judge's note: The page 3 image shows M(π_i) with no subscript. M is defined only as M_i, and the line above uses η(π_i) = M_i(π_i).
Second check: none (judge only)
Report a problem with this finding
Typo Section 3: "rather than just mixture polices" should read "mixture policies".
- p. 3 Section 3, first paragraph
rather than just mixture polices
Judge's note: The page 3 image, left column, first paragraph, shows 'mixture polices'.
Second check: none (judge only)
Report a problem with this finding
Typo Section 6: "Here we present two practical policy optimization algorithm" should read "algorithms".
- p. 5 Section 6, first sentence
two practical policy optimization algo- rithm based on the ideas above
- p. 5 Section 6, first sentence
Here we present two practical policy optimization algo-rithm based on the ideas above
Judge's note: The page 5 image shows 'algo-rithm' with no 's'.
Second check: none (judge only)
Report a problem with this finding
Typo "The benefit of the vine method over the single path method that is our local estimate of the objective has much lower variance" is missing a verb; it should read "... method is that our local estimate ...".
- p. 5 Section 5.2, last paragraph of left column
over the single path method that is our local estimate of the objective
- p. 5 Section 5.2, last paragraph
The benefit of the vine method over the single path method that is our local estimate
Judge's note: The page 5 image, left column, last paragraph of Section 5.2, confirms the wording.
Second check: none (judge only)
Report a problem with this finding
Typo Misspelling: 'the theorecally justified maximum KL divergence' should read 'theoretically'.
- p. 7 Section 8.1, right column
theorecally justified maximum KL divergence
Judge's note: The misspelling is visible on the p7 page image, right column, end of the Section 8.1 paragraph.
Second check: none (judge only)
Report a problem with this finding
Typo Acknowledgements: 'for pointing our errors in a previous version of the manuscript' should read 'pointing out errors'.
- p. 8 Acknowledgements
pointing our errors in a previous version
- p. 8 Acknowledgements
for pointing our errors in a previous version of the manuscript
- p. 8 Acknowledgements
for pointing our errors in a previous version
Judge's note: The text is visible on the p8 page image, so it is not an extraction artefact.
Second check: none (judge only)
Report a problem with this finding
Typo 'apply the same policy search method to methods as diverse as robotic locomotion and image-based game playing': the second 'methods' should be 'tasks'/'domains'.
- p. 8 Section 8.2, end
apply the same policy search method to methods as di- verse as robotic locomotion
- p. 8 Section 8.2
apply the same policy search method to methods as di-
Judge's note: Viewed on p8, left column, end of Section 8.2. The wrong word is used, but the meaning can be recovered.
Second check: none (judge only)
Report a problem with this finding
Typo Missing sentence-ending period: '(Bellemare et al., 2013) The images were preprocessed'; the same sentence also has 'Mnih et al (2013)' without a period after 'al'.
- p. 8 Section 8.2 continued, left column
(Bellemare et al., 2013) The images were preprocessed
- p. 8 Section 8.2
Mnih et al (2013)
- p. 8 Section 8.2
(Bellemare et al., 2013) The images
Judge's note: Both slips are visible on the p8 page image, left column.
Second check: none (judge only)
Report a problem with this finding
Typo Lemma 1: 'τ := (s0, a0, s1, a0, . . . )' repeats a0 where a1 is meant.
- p. 10 Lemma 1, text after Eq. (19)
τ := (s0, a0, s1, a0, . . . )
Judge's note: Checked on the p10 page image, in the text after Eq. (19). The fourth element is a0.
Second check: none (judge only)
Report a problem with this finding
Typo Appendix A overview: 'we jointly define the policies π and π′' while the rest of the overview and proof use π̃.
- p. 10 Appendix A, first paragraph
we jointly define the policies π and π′
Judge's note: The p10 page image shows π′ in the first paragraph; the next sentence and all later text use π̃. This is a notation slip.
Second check: none (judge only)
Report a problem with this finding
Typo Duplicated word: 'Surrogate loss Lπ(π̃) accounts for the the advantage of π̃'.
- p. 10 Appendix A, first paragraph
accounts for the the advantage
- p. 10
gate loss Lπ(˜π) accounts for the the advantage of ˜π the
Judge's note: The repeated 'the the' is visible on the p10 page image.
Second check: code check
Report a problem with this finding
Typo Ungrammatical: 'The expectation decomposes similarly for actions are sampled using π:' (should be 'for actions sampled' / 'when actions are sampled').
- p. 11 Proof of Lemma 3, before Eq. (34)
The expectation decomposes similarly for actions are sampled using π:
Judge's note: Visible on the p11 page image, before Eq. (34).
Second check: none (judge only)
Report a problem with this finding
Typo Eq. (32) right-hand side 'max_s |A_π(s, a)|' leaves a unbound; Eqs. (28), (39) and (40) use max_{s,a}.
- p. 11 Lemma 3, Eq. (32)
4α(1 − (1 − α)^t) max_s|A_π(s, a)|
Judge's note: The p11 page image shows the subscript 's' alone under max in Eq. (32), while Eq. (40), the proved form, has 's,a'. This is a notation slip, separate from the false middle term in C028.
Second check: none (judge only)
Report a problem with this finding
Typo Appendix B uses the undefined symbol c for the reward dual vector r: "η(π̃) = cG̃ρ0" and "Thus we can write γcGΔGρ0 = γvΔρπ".
- p. 12 Appendix B proof, paragraph before Eq. (46) and after Eq. (49)
η(˜π) = c ˜Gρ0
- p. 12 Appendix B, proof text
η(π̃) = cG̃ρ0
- p. 12 Appendix B, last paragraph
γcGΔGρ0 = γvΔρπ
- p. 12 Appendix B
γcG∆Gρ0
Judge's note: Page 12 image: r is defined as the reward dual vector, η(π) = rGρ0 is written immediately before, and Eq. (49) and the following sentence ("leading term γrGΔGρ0") use r. The symbol c appears twice and is never defined. It should be r.
Second check: none (judge only)
Report a problem with this finding
Typo Appendix C writes the search-direction system first as "approximately solving the equation Ax = g" and two sentences later as "approximately solve the equation Ax = b"; b is never defined.
- p. 14 Appendix C, second paragraph
Ax = b without forming this full matrix
Judge's note: Page 14 image checked. Ax = b is the generic conjugate-gradient form, so this is a minor notational inconsistency; the next paragraph uses s ≈ A⁻¹g.
Second check: none (judge only)
Report a problem with this finding
Typo Appendix D: "The components of µ are computed by taking applying a neural network to the input s" contains two verbs in a row.
- p. 15 Appendix D, last paragraph
computed by taking applying a neural network
Judge's note: Page 15 image confirms the text; "taking" should be deleted.
Second check: none (judge only)