Proximal Policy Optimization Algorithms
2017 · paper · AI and machine learning
Proposes proximal policy optimization, a family of policy gradient methods using a clipped surrogate objective that allows several epochs of minibatch updates per sample. On robotic locomotion and Atari tests, it outperforms other online policy gradient methods while being simpler than trust region methods.
Checked: PDF from arxiv.org, fetched 4 Oct 2026. Page numbers are those of that PDF.
3 findings
Substance Eq. (10) prints "γ^{T−t+1} r_{T−1} + γ^{T−t} V(s_T)"; the exponent on r_{T−1} should be T−t−1.
- p. 5 Section 5, Equation (10)
γT−t+1rT−1 + γT−tV (sT )
Judge's note: Checked on the p.5 image. In the series r_t + γ r_{t+1} + ..., the term r_{t+k} carries γ^k, so r_{T−1} (k = T−1−t) carries γ^{T−t−1}. That is consistent with the following term V(s_T) carrying γ^{T−t}. The printed T−t+1 is two too large. Arithmetic example with T=10, t=0: the correct exponent is 9, the printed one is 11.
Second check: another model agreed
Report a problem with this finding
Substance Eq. (11) prints "(γλ)^{T−t+1} δ_{T−1}"; the exponent should be T−t−1.
- p. 5 Section 5, Equation (11)
(γλ)T−t+1δT−1
Judge's note: Checked on the p.5 image. The term δ_{t+k} carries weight (γλ)^k, so δ_{T−1} carries (γλ)^{T−t−1}. Only with that exponent does Eq. (11) reduce to the corrected Eq. (10) at λ=1, as the text claims. This is the same off-by-two error as in C010 but in a separate equation. Arithmetic example with T=10, t=0: correct exponent 9, printed 11.
Second check: another model agreed
Report a problem with this finding
Wording Captions and text name only PPO and A2C, yet ACER is also plotted and tabulated. Figure 6 caption: "Comparison of PPO and A2C on all 49 ATARI games". Appendix B: "a comparison of PPO against A2C on a larger collection of 49 Atari games". Table 6 caption: "Mean final scores ... of PPO and A2C".
- p. 11 Figure 6 caption
Comparison of PPO and A2C on all 49 ATARI games
- p. 10 Appendix B text
Here we include a comparison of PPO against A2C on a larger collection of 49 Atari games.
- p. 12 Table 6 caption
Mean final scores (last 100 episodes) of PPO and A2C on Atari games
- p. 10 Appendix B
comparison of PPO against A2C on a larger collection of 49 Atari games
Judge's note: On page 11, the Figure 6 legend lists A2C, ACER and PPO, and green ACER curves appear in the panels. On page 12, Table 6 has an ACER column. On page 10, the Appendix B text mentions only A2C. The captions and the text leave out a method that is shown.
Second check: another model agreed
Report a problem with this finding
15 minor (typos and the like; most checked by the judge only)
Wording Introduction states without qualification "On continuous control tasks, it performs better than the algorithms we compare against", while Section 6.2 says "almost all the continuous control environments" and the Figure 3 Swimmer-v1 panel shows TRPO ending above PPO (Clip).
- p. 1 Introduction, last paragraph
On continuous control tasks, it performs better than the algorithms we compare against.
- p. 7 Section 6.2 text
PPO outperforms the previous methods on almost all the continuous control environments.
- p. 7 Section 6.2 text
We see that PPO outperforms the previous methods on almost all the continuous control environments.
- p. 7 Figure 3, Swimmer-v1 panel
Swimmer-v1
Only the authors can say what is right here.
Judge's note: Checked p.1 and p.7. In the Figure 3 Swimmer-v1 panel the cyan TRPO curve ends near 120 and the purple PPO (Clip) curve near 105-110. Section 6.2 itself says 'almost all'. The Introduction claim is broader than both the results text and the figure. This is a wording overstatement and the data are not affected.
Second check: another model agreed
Report a problem with this finding
Wording Abstract says "we show that PPO ... overall strikes a favorable balance between sample complexity, simplicity, and wall-time", but the paper reports no wall-clock timing results.
- p. 1 Abstract, last sentence
overall strikes a favorable balance between sample complexity, simplicity, and wall-time.
Only the authors can say what is right here.
Judge's note: A grep for wall/time/computation finds 'wall-time' only in the abstract. Figures 3 and 4 plot results against timesteps, and the other results also contain no runtime measurement. The wall-time part of the claim is therefore unsupported by data in the paper.
Second check: another model agreed
Report a problem with this finding
Wording p.3: "r is clipped at 1 − ϵ or 1 + ϵ depending on whether the advantage is positive or negative" lists the bounds in the reverse order to Figure 1, where A>0 clips at 1+ϵ and A<0 clips at 1−ϵ.
- p. 3 Section 3, paragraph before Figure 1
r is clipped at 1 − ϵ or 1 + ϵ depending on whether the advantage is positive or negative
Judge's note: In Figure 1 on p.3, the left panel (A>0) has its kink at 1+ϵ and the right panel (A<0) at 1−ϵ. The sentence has no 'respectively', so it may be meant loosely. Read in order, though, it pairs the bounds the wrong way round. Swapping the order would remove the ambiguity.
Second check: another model agreed
Report a problem with this finding
Wording Algorithm 1 computes "advantage estimates Â1, . . . , ÂT", while Eq. (10) uses r_t…r_{T−1} and V(s_T) with "t specifies the time index in [0, T]", which implies t = 0..T−1.
- p. 5 Algorithm 1
Compute advantage estimates Â1, . . . , ÂT
- p. 5 Text after Eq. (10)
where t specifies the time index in [0, T]
Judge's note: Checked on the p.5 image. With 0-based time, Eq. (10) covers t = 0..T−1, so Â_T would need rewards beyond the segment, while the pseudocode uses 1..T. The bracket [0, T] is itself loose. This is a minor indexing inconsistency between informal pseudocode and the equation.
Second check: another model agreed
Report a problem with this finding
Typo Misspelling on p.1: "have poor data effiency and robustness" should read "efficiency".
- p. 1 Introduction, paragraph 1
have poor data effiency and robustness
- p. 1 Introduction
poor data effiency and robustness
- p. 1
vanilla policy gradient methods have poor data effiency and robustness
Judge's note: The page 1 image shows 'effiency' at the start of the line in Introduction paragraph 1.
Second check: none (judge only)
Report a problem with this finding
Typo Duplicated word on p.2, Section 2.2: "where the the characteristics change over the course of learning".
- p. 2 Section 2.2, last paragraph
where the the characteristics change
- p. 2 Section 2.2
where the the characteristics
- p. 2
where the the characteristics change over the course of learning
- p. 2
ithin a single problem, where the the characteristics cha
Judge's note: The crop of page 2 shows 'the the' as typeset.
Second check: code check
Report a problem with this finding
Typo Stray article on p.4, Section 4: "The initial value of β is a another hyperparameter".
- p. 4 Section 4, last paragraph
The initial value of β is a another hyperparameter
- p. 4 Section 4
is a another hyperparameter
Judge's note: The page 4 image shows 'is a another hyperparameter' as typeset.
Second check: none (judge only)
Report a problem with this finding
Typo Misspelling on p.4, Section 5: "automatic dif-ferentation" should read "differentiation".
- p. 4 Section 5, paragraph 1
automatic dif- ferentation
Judge's note: The page 4 image shows 'dif-' at a line end followed by 'ferentation'. The letters 'ti' are missing.
Second check: none (judge only)
Report a problem with this finding
Typo Missing word on p.4, Section 5: "make use a learned state-value function" should read "make use of a".
- p. 4 Section 5, paragraph 2
make use a learned state-value function
Judge's note: The page 4 image shows 'make use a' at the line end and 'learned state-value function' on the next line.
Second check: none (judge only)
Report a problem with this finding
Typo Eq. (11) contains a doubled ellipsis: "δt + (γλ)δt+1 + · · · + · · · + (γλ)^{T−t+1}δT−1".
- p. 5 Equation (11)
δt + (γλ)δt+1 + · · · + · · · + (γλ)T−t+1δT−1
- p. 5 Eq. (11)
+ · · · + · · · +
Judge's note: The rendered equation on the p.5 image shows '+ ··· + ··· +'. One ellipsis is redundant.
Second check: none (judge only)
Report a problem with this finding
Typo Section 6.1 'Clipping' prints "min(rt(θ)Ât, clip(rt(θ)), 1 − ϵ, 1 + ϵ)Ât" with a misplaced parenthesis; Eq. (7) gives min(rt(θ)Ât, clip(rt(θ), 1−ϵ, 1+ϵ)Ât).
- p. 5 Section 6.1, list of surrogate variants, 'Clipping'
min(rt(θ)Ât, clip(rt(θ)), 1 − ϵ, 1 + ϵ)Ât
- p. 5 Section 6.1 Clipping
clip(rt(θ)), 1 −ϵ, 1 + ϵ)
Judge's note: The p.5 image shows ')' immediately after clip(rt(θ). As printed, 1−ϵ and 1+ϵ become arguments of min and Ât multiplies the whole min. Eq. (7) on p.3 gives the correct form, so the intended formula is clear. This is a typesetting error.
Second check: none (judge only)
Report a problem with this finding
Typo Duplicated word on p.6, Section 6.1: "the other hyperparameters are provided in in Table 3".
- p. 6 Section 6.1, paragraph 2
provided in in
- p. 6 Section 6.1
provided in in Table 3
- p. 6
are provided in in Table 3
- p. 6
hyperparameters are provided in in Table 3. To represe
Judge's note: The crop of page 6 shows 'in in' at the line end before 'Table 3'.
Second check: code check
Report a problem with this finding
Typo Section 6.4 misspells 'architecture': "used the same policy network architechture as used in [Mni+16]".
- p. 8 Section 6.4, paragraph 1
policy network architechture
- p. 8
policy network architechture as used in [Mni+16]
Judge's note: I checked page 8, Section 6.4, paragraph 1, on the page image. The text reads 'architechture'. It should be 'architecture'.
Second check: none (judge only)
Report a problem with this finding
Typo The Conclusion is missing an article: "requiring only few lines of code change" should read "requiring only a few lines of code change".
- p. 8 Section 7 Conclusion
requiring only few lines of code change
Judge's note: I checked the Section 7 Conclusion on page 8 of the page image. The phrase is printed as quoted. The intended sense is plainly 'only a few'.
Second check: none (judge only)
Report a problem with this finding
Typo The Table 3 caption spells the engine "Mujoco" ("used for the Mujoco 1 million timestep benchmark"), while the body text, the Figure 3 caption and [TET12] use "MuJoCo".
- p. 10 Table 3 caption
used for the Mujoco 1 million timestep benchmark
Judge's note: I checked the page 10 image and searched the text layer. 'MuJoCo' appears at line 301, in the Figure 3 caption and in the reference. 'Mujoco' appears only in the Table 3 caption. This is a cosmetic inconsistency.
Second check: none (judge only)