Deep Reinforcement Learning from Human Preferences

• p. 6 Section 3.1.1, last paragraph

  Figure 2 shows the results of training our agent with 700 queries to a human rater

• p. 7 Figure 2 legend (pendulum panel)

  750 human queries

• p. 1 Abstract

  providing feedback on less than 1% of our agent’s interactions with the environment

• p. 14 Appendix A.1

  which varies from 15 to 60 timesteps depending on the task

Only the authors can say what is right here.

Judge's note: On the page 7 image the Figure 2 legend reads "750 human queries"; the page 6 text says 700. The count may be approximate, but the two do not match. The separate point about the abstract's "less than 1%" is not confirmed. The abstract names Atari and locomotion, pendulum is not locomotion, and A.1 gives only the range 15-60 timesteps per segment, not the pendulum value.

Second check: another model agreed

• p. 2 Introduction, paragraph beside Figure 1

  while being equally useful for learning human preferences

• p. 10 Section 3.3, comparisons vs targets paragraph

  For continuous control tasks we found that predicting comparisons worked much better than predicting scores.

Judge's note: Both passages are confirmed on pages 2 and 10. Section 3.3 reports that comparisons were much better on continuous control and gave significantly different performance on Atari, with neither consistently better. "Equally useful" therefore misdescribes the ablation finding, although it does not change which method comes out ahead. This is a wording point.

Second check: another model agreed

• p. 7 Section 3.1.1 text below Figure 2

  Surprisingly, by 1400 labels our algorithm performs slightly better than if it had simply been given the true reward

• p. 7 Figure 2, pendulum and reacher panels

  1400 synthetic queries

Judge's note: On the page 7 image, Figure 2 shows the orange real-reward curve at about -2, above the dark-blue 1400 curve at about -2.5, on reacher. The 1400 curve is above real reward on walker, cheetah and hopper. The claim holds for some tasks and not others, so the issue is overgeneral wording.

Second check: another model agreed

• p. 8 Figure 3 legend

  5.6k synthetic labels

• p. 10 Figure 6 caption

  using 5,500 synthetic labels

Judge's note: The page 8 legend reads 5.6k; the page 10 caption reads 5,500. No text describes a 5,600-label condition; A.2 mentions only 5,000 vs 5,500 for the ablations. This may be a label slip or a genuinely different budget; only the authors can say.

Second check: another model agreed

• p. 15 Appendix A.2, paragraph 1

  life loss ending an episode (but not resetting the environment)

• p. 14 Appendix A, second bullet

  In Atari games, we do not send life loss or episode end signals to the agent

• p. 14 Appendix A

  we do not send life loss or episode end signals to the agent

Judge's note: Read both passages in the p14 and p15 crops. A consistent reading exists: the wrapper marks episode ends at life loss, and those ends are withheld from the agent or replaced with a penalty, as the A bullet says for synthetic feedback. The text does not state this link, so the wording is ambiguous rather than a definite contradiction.

Second check: another model agreed

• p. 15 Appendix A.2, paragraph 3

  policy gradient using Adam with α = 0.99 and ϵ = 10−5

Only the authors can say what is right here.

Judge's note: Rests on outside knowledge: in Adam (Kingma & Ba) α denotes the step size, and α = 0.99, ε = 1e-5 are the RMSProp decay and epsilon of the A3C setup. The p15 crop shows the text as quoted. In the same section, α is also used for the leaky-ReLU slope (0.01) and the dropout value (0.5). Only the authors can say which optimizer was used.

Second check: another model agreed

• p. 2 Related Work, first paragraph

  Other lines of research considers the general problem

Judge's note: Confirmed on the page 2 image, Related Work paragraph 1.

Second check: none (judge only)

• p. 3 Section 1.1, paragraph 4

  Our work could also be seen of a specific instance

Judge's note: Confirmed on the page 3 image.

Second check: none (judge only)

• p. 3 Related Work, first paragraph on page 3

  the complexity of our environments force us

Judge's note: Confirmed on the page 3 image, line 2.

Second check: none (judge only)

• p. 4 Footnote 1

  Here we assume here that the reward is a function

Judge's note: Confirmed on the page 4 image, footnote 1.

Second check: none (judge only)

• p. 4 Section 2.1, Qualitative

  how well the agent satisfies to the human’s preferences

• p. 4 Qualitative bullet

  how well the agent satisfies to the human’s

Judge's note: Confirmed on the page 4 image, in the Qualitative item.

Second check: none (judge only)

• p. 5 Section 2.2.3

  Luce-Shephard choice rule (Luce, 2005; Shepard, 1957)

Judge's note: Confirmed on the page 5 image; the citation in the same parenthesis spells the name Shepard.

Second check: none (judge only)

• p. 5 Section 2.2.3, third bullet

  as the difference in reward difference becomes extreme

Judge's note: Confirmed on the page 5 image, third bullet.

Second check: none (judge only)

• p. 9 Figure 5 caption

  described in Section Section 3.3

• p. 9 Figure 5 caption

  as described in Section Section 3.3

Judge's note: The repeated word is visible in the page-9 crop of the Figure 5 caption, so it is not an extraction artefact. The Figure 6 caption reads "described in Section 3.3" correctly.

Second check: none (judge only)

• p. 11 References

  Riad Akrour, Marc Schoenauer, and Michele Sebag.

Judge's note: Checked the reference crop on p11. The first Akrour entry has no accent and the next two have it. This is a cosmetic inconsistency.

Second check: none (judge only)

• p. 14 Appendix A.1, last paragraph

  our rate of labeling after T frames 2 ∗ 106/(T + 2 ∗ 106)

Judge's note: Checked the crop on p14. The clause has no verb. The parallel sentence in A.2 reads "is decreased ... to be roughly proportional to 5 ∗ 10^6/(T + 5 ∗ 10^6)", which suggests the intended wording, likely "is proportional to".

Second check: none (judge only)

• p. 15 Appendix A.2, paragraph 4

  with the adapative scheme described in Section 2.2.3

• p. 15 A.2

  adapative scheme

Judge's note: Visible in the p15 image (the C022 crop shows the same paragraph).

Second check: none (judge only)

• p. 16 Appendix B.2

  play the game yourself for a five minutes

Judge's note: Visible in the p16 crop. The heading above reads "for 5 minutes" with no article.

Second check: none (judge only)