Training Language Models to Follow Instructions with Human Feedback
Advances in Neural Information Processing Systems 35, 2022 · paper · AI and machine learning
Fine-tunes GPT-3 using labeler demonstrations and then reinforcement learning from human rankings, producing models called InstructGPT. In human evaluations, outputs from the 1.3-billion-parameter version were preferred over 175-billion-parameter GPT-3, with gains in truthfulness and less toxic output.
Checked: PDF from arxiv.org, fetched 4 Oct 2026. Page numbers are those of that PDF.
11 findings
Substance Sec 3.5 says the RM is trained "Starting from the SFT model with the final unembedding layer removed", but Appendix C.2 says "The final reward model was initialized from a 6B GPT-3 model that was fine-tuned on a variety of public NLP datasets".
- p. 8 Section 3.5, Reward modeling
Starting from the SFT model with the
- p. 41 Appendix C.2
reward model was initialized from a 6B GPT-3 model
Only the authors can say what is right here.
Judge's note: Checked on the p.8 and p.41 images. The two passages disagree about how the final RM was initialized. C.2 adds that SFT or GPT-3 initialization gives similar results, but that does not reconcile the two descriptions of the model actually used.
Second check: another model agreed
Report a problem with this finding
Reference or label Sec 2 (p.5) cites "(Yi et al., 2019; Mishra et al., 2021; Wei et al., 2021; ...)" as cross-task instruction-generalization work, but the only Yi et al. 2019 entry is "Towards coherent and engaging spoken dialog response generation using automatic conversation evaluators".
- p. 5 Section 2, 'Training language models to follow instructions'
(Yi et al., 2019; Mishra et al., 2021; Wei
- p. 25 Reference list
Towards coherent and engaging spoken dialog response generation
Only the authors can say what is right here.
Judge's note: The reference entry (p.25 image) is a spoken-dialogue paper. It fits the dialogue citation on p.4 but not a list of work fine-tuning LMs on many NLP tasks with instructions. No other Yi entry exists. The intended work cannot be identified from the files.
Second check: another model agreed
Report a problem with this finding
Reference or label Figure 4 caption says "See Appendix E.2 for analysis that includes model size." E.2 is "Reward model generalization across sets of labelers"; the metadata-by-model-size analysis is in E.3 (Figure 30).
- p. 12 Figure 4 caption
See Appendix E.2 for analysis that includes model size.
- p. 52 Appendix E.2 / E.3 headings
E.3 Metadata results as a function of model size
- p. 52 Appendix E.2 / E.3 headings
Reward model generalization across sets of labelers
- p. 52 E.3
Metadata results as a function of model size
Judge's note: Page 52 image: E.2 covers RM cross-validation across labeler groups; E.3 'Metadata results as a function of model size' points to Figure 30. The pointer should read E.3.
Second check: another model agreed
Report a problem with this finding
Reference or label Figure 7 caption says "evaluations recorded in Table 14 in Appendix D", but Table 14 (Automatic evaluations) is printed in Appendix E (p. 56, cited from E.1).
- p. 14 Figure 7 caption
evaluations recorded in Table 14 in Appendix D
- p. 51 Appendix E.1
The results of these evaluations are in Table 14.
- p. 14 Figure 7 caption
evaluations recorded in Table 14 in Appendix D.
Judge's note: Page 51: Appendix E begins and E.1 says 'The results of these evaluations are in Table 14.' Page 56: Table 14 sits between E.6 and E.7 content. Appendix D (Automatic evaluation details) comes before p. 51 and holds no Table 14.
Second check: another model agreed
Report a problem with this finding
Reference or label Text after "In Figure 29 we show..." says "The performance of the PPO-ptx model still lags behind GPT-3 on DROP, SQuADv2, and translation", but in few-shot Figure 29 / Table 14, PPO-ptx is at or above GPT-3 on SQuADv2 at 6B and 175B. The statement fits the zero-shot Figure 28.
- p. 15 Section 4.2, paragraph after Figure 8
still lags behind GPT-3 on DROP, SQuADv2, and translation
- p. 56 Table 14, SQuADv2 few-shot row
69.93
- p. 15 Section 4.2, alignment-tax paragraph
The performance of the PPO-ptx model still lags behind GPT-3 on DROP, SQuADv2, and translation
Only the authors can say what is right here.
Judge's note: Table 14 (p. 56), SQuADv2 few-shot: GPT 58.86/62.33/69.75 vs PPO-ptx 58.33/63.78/69.93. Zero-shot: GPT 64.30 vs PPO-ptx 59.85 at 175B. Figure 29 shows the curves meeting at 175B; Figure 28 shows PPO-ptx clearly below GPT on SQuADv2, DROP and translation and above it on HellaSwag. The sentence most likely refers to Figure 28, or needs a zero-shot qualifier; the authors should confirm.
Second check: another model agreed
Report a problem with this finding
Reference or label A.4 says Table 11 reports demonstration lengths "both for contractor-written and labeler-written prompts", but Table 11's prompt sources are 'Contractor' and 'Customer'.
- p. 35 Appendix A, last paragraph
both for contractor-written and labeler-written
- p. 34 Table 11, Prompt source column
Customer
- p. 35 A.4 last paragraph
both for contractor-written and labeler-written prompts
Judge's note: On p.34, Table 11 has rows Contractor and Customer only. The paper uses 'labelers' and 'contractors' for the same group (A.1: 'we asked contractors to write prompts... We asked labelers to write...'), so the second category is mislabelled. It should presumably read customer prompts.
Second check: another model agreed
Report a problem with this finding
Reference or label B.2: "We show excerpts of our instructions for our final evaluations on our prompt distribution in Table 10, and on the RealToxicityPrompts distribution in Table 11". The excerpts are Figures 10 and 11.
- p. 36 Appendix B.2, last paragraph
on the RealToxicityPrompts distribution in Table 11
- p. 37 Figure 10 caption
Figure 10: Excerpt from the instructions given to labelers
- p. 34 Table 10 / Table 11 titles
Table 10: Prompt lengths by category
- p. 36 B.2
in Table 10, and on the RealToxicityPrompts distribution in Table 11
Judge's note: The Figure 10 caption on p.37 is 'Excerpt from the instructions given to labelers for final evaluations...' and the Figure 11 caption on p.38 is 'Complete instructions ... RealToxicityPrompts distribution'. On p.34, Tables 10 and 11 are prompt and demonstration length tables. The pointer is mislabelled.
Second check: another model agreed
Report a problem with this finding
Reference or label C.3: "We also mix in 10% pretraining data during fine-tuning, since we find it helpful for PPO training (see Appendix E.11 for details)". The 0/10/50% init-model ablation is in E.8 (Figure 37).
- p. 42 Appendix C.3
(see Appendix E.11 for details)
- p. 56 Appendix E.8 PPO init models
the only setting stands out is with 10% pretraining data mix
Judge's note: In the text, E.8 'PPO init models' (p.56) compares 0%, 10% and 50% pretraining mix and chooses 10%. E.11 'Additional ablations' covers the pretraining data ratio during PPO, episode counts and batch sizes, not the SFT init mix. The pointer should be E.8.
Second check: another model agreed
Report a problem with this finding
Reference or label D.2: "These are shown in Table 14-25." The prompt structure and dataset feature boxes are Figures 14-27.
- p. 44 Appendix D.2
These are shown in Table 14-25.
- p. 45 Figure 14 caption
Figure 14: Winogender: prompting, examples, and dataset features.
- p. 50 Figure 27 caption
Figure 27: TL;DR: prompting, examples, and dataset features.
Judge's note: Captions run from Figure 14 (Winogender) to Figure 27 (TL;DR), confirmed in the text and the p.50 crop. Table 14 is 'Automatic evaluations', and there are no Tables 15-25. Both the item type and the range are wrong.
Second check: another model agreed
Report a problem with this finding
Wording Appendix D: "Multiple choice answers are picked according to which one has the lowest average per-token log probability at T = 1." This selects the least likely option.
- p. 44 Appendix D, second paragraph
lowest average per-token log probability
Only the authors can say what is right here.
Judge's note: The p.44 image confirms the wording. Picking the lowest log probability is the reverse of the usual rule (highest log probability, or lowest negative log-likelihood). This is probably a wording slip, but only the authors can confirm the procedure used.
Second check: another model agreed
Report a problem with this finding
Wording The CNN/DM (Figure 26, p.49) and TL;DR (Figure 27, p.50) summarization boxes both say "In the few-shot setting, there are 15 additional French / English pairs." This is the same bullet as in the WMT Fr→En box (Figure 25).
- p. 49 Figure 26 (CNN/DM) dataset features
In the few-shot setting, there are 15 additional French / English pairs.
- p. 50 Figure 27 (TLDR) dataset features
In the few-shot setting, there are 15 additional French / English pairs.
- p. 49 Figure 26 CNN/DM
there are 15 additional French / English pairs
- p. 50 Figure 27 TL;DR
there are 15 additional French / English pairs
Only the authors can say what is right here.
Judge's note: The full p.49 image shows the CNN/DM box with this bullet below '2,354 news articles to summarize'. The C047-b crop shows the TL;DR box with the same bullet below '2,500 Reddit posts to summarize'. A summarization task has no French/English pairs, so the line appears to be copied from Figure 25. Only the authors can say what the few-shot examples were and how many there were (the 15 may also be copied).
Second check: another model agreed
Report a problem with this finding
40 minor (typos and the like; most checked by the judge only)
Reference or label Figure 40's shared y-axis is labelled "Toxicity", but the left panel plots Continuity (values about 4.5–6), not toxicity.
- p. 60 Figure 40, y-axis label
Toxicity
- p. 60 Figure 40 caption
Continuity and relative toxicity ratings
- p. 60 Figure 40
Toxicity Continuity
Judge's note: The full p.60 image shows the 'Continuity' facet on a 0–6 axis labelled 'Toxicity' and the 'Relative toxicity' facet with negative values. The caption says 'Continuity and relative toxicity ratings'. This is a mislabelled axis from a shared facet label.
Second check: another model agreed
Report a problem with this finding
Wording Intro (p.3) says "InstructGPT generates truthful and informative answers about twice as often as GPT-3", but the human evaluation in Figure 6 shows a ratio of roughly 1.2–1.5x, and Section 4.2 calls the gains "small but significant".
- p. 3 Introduction, truthfulness paragraph
InstructGPT generates truthful and informative answers about twice as often as GPT-3
- p. 13 Section 4.2 and Figure 6
our PPO models show small but
- p. 13 Section 4.2 and Figure 6
our PPO models show small but significant improvements
Only the authors can say what is right here.
Judge's note: In Figure 6 (p.13), the 175B QA-prompt truthful+informative bars are about 0.20 for GPT and about 0.30 for PPO-ptx (InstructGPT). With the Instruction+QA prompt they are about 0.22 and 0.27. Neither comparison is close to double. Only the automatic Table 14 numbers (p.56) reach about 2x or more (QA prompt 0.251 vs 0.689; instruction prompt 0.226 vs 0.494), and the acknowledgements (text line ~1259) say the automatic TruthfulQA metrics overstated the PPO gains. The paper does not say which metric, prompt or model size gives 'about twice', so this is a query for the authors.
Second check: another model agreed
Report a problem with this finding
Wording Intro (p.3) says "Our results are equally strong on the subset of questions that were not adversarially selected against GPT-3", but Sec 4.2 (p.13) says "(although the absolute improvement decreases by a couple of percentage points.". The same paragraph also has the typo "evaluatoins" and a parenthesis that is never closed, and p.16 has "is able to reliably answers".
- p. 3 Introduction
Our results are equally strong on the subset of questions that were not adversarially selected
- p. 13 Section 4.2
although the absolute improvement decreases by a couple of percentage points
- p. 3 Section 1, truthfulness paragraph
Our results are equally strong on the subset of questions that were not adversarially selected against
- p. 13 Sec 4.2, first paragraph
(although the absolute improvement decreases by a couple of percentage points.
Judge's note: Checked on the p.3 and p.13 images. The two descriptions of the non-adversarial subset differ: 'equally strong' against an improvement that shrinks by a couple of points, on gains that are only a few points to begin with. On p.13 the image shows 'evaluatoins' and the parenthesis opened before 'although' is not closed. On p.16 the image shows 'is able to reliably answers questions about code'. The 'have are' typo is judged under C011.
Second check: another model agreed
Report a problem with this finding
Wording Intro (p.3) says "InstructGPT models generate about 25% fewer toxic outputs than GPT-3 when prompted to be respectful", but Figure 7 and Table 14 show reductions of about 12–20% for PPO-ptx.
- p. 3 Introduction, toxicity paragraph
InstructGPT models generate about 25% fewer toxic outputs than GPT-3
- p. 56 Table 14, Real Toxicity respectful row
0.233
- p. 14 Figure 7
Comparing human evaluations and automatic evaluations
- p. 14 Figure 7
Comparing human evaluations and automatic evaluations (Perspective API scores) on RealToxicityPrompts
Only the authors can say what is right here.
Judge's note: Table 14 (p.56), respectful prompt, GPT vs PPO-ptx: 175B 0.233 vs 0.196, 6B 0.232 vs 0.204, XL 0.211 vs 0.179. Figure 7 (p.14) human eval: about 0.205 vs 0.165; Perspective API: about 0.235 vs 0.197. None of these reaches about 25%. Only SFT, or PPO 6B without ptx, comes near, and Sec 3.5 defines InstructGPT as PPO-ptx. The source of the 25% figure is not stated.
Second check: another model agreed
Report a problem with this finding
Wording Intro (p.3) gives "21% vs. 41% hallucination rate" for InstructGPT vs GPT-3, but in Figure 4 the PPO-ptx (InstructGPT) bar is about 0.15–0.16 and the GPT bar about 0.44. The 21% matches the PPO bar instead.
- p. 3 Introduction
21% vs. 41% hallucination rate, respectively
- p. 9 Section 3.5
InstructGPT refers to the PPO-ptx models
- p. 12 Figure 4, Hallucinations panel
Hallucinations
Only the authors can say what is right here.
Judge's note: Read from the Figure 4 Hallucinations panel (p.12): GPT ≈0.44, GPT prompted ≈0.28, SFT ≈0.08, PPO ≈0.21, PPO-ptx ≈0.155. Sec 3.5 (p.9) says InstructGPT means PPO-ptx unless otherwise specified. Neither printed number clearly matches a bar for the stated pair of models, although the 'about half as often' claim holds either way. Bar heights are read approximately from the image.
Second check: another model agreed
Report a problem with this finding
Wording Sec 3.2 says "we limit the number of prompts to 200 per user ID" and splits "based on user ID"; Appendix A.2 says "roughly 200 per organization" and splits "based on organization IDs".
- p. 6 Section 3.2
we limit the number of prompts to 200 per user ID
- p. 26 Appendix A.2
limited the number of prompts to roughly 200 per organization
- p. 6 Sec 3.2
200 per user ID
- p. 26 A.2
roughly 200 per organization
Only the authors can say what is right here.
Judge's note: Both images confirm the wording. The paper does not say whether user IDs and organization IDs coincide, so the unit used for the cap and for the split is ambiguous.
Second check: another model agreed
Report a problem with this finding
Wording Sec 3.5 says PPO fine-tunes "the SFT model" with "a per-token KL penalty from the SFT model" (πSFT in Eq. 2), but Appendix C.3/C.4 initializes RL from separate models trained for 2 epochs with a 10% pretraining mix, and these "are also used to compute the KL reward".
- p. 9 Section 3.5, Reinforcement learning
we add a per-token KL penalty from the SFT model
- p. 42 Appendix C.3–C.4
These models are also used to compute the KL reward
Only the authors can say what is right here.
Judge's note: Sec 3.5 (p.8) describes the SFT model as trained for 16 epochs. C.3/C.4 (p.42) and E.8 (p.56) describe a different 2-epoch init model trained with a pretraining mix. The main text does not say that the KL reference and init model differ from the SFT baseline.
Second check: another model agreed
Report a problem with this finding
Wording Appendix E.2 says "the inter- and intra-group validation accuracies ... are 72.4±0.4%, and 69.6±0.9% respectively". Section 4.1 gives 69.6% for the held-out group and 72.4% for the training labelers. Under the usual reading (inter-group = held-out), the labels are reversed.
- p. 52 Appendix E.2
inter- and intra-group validation accuracies
- p. 12 Section 4.1
These RMs have an accuracy of 69.6 ± 0.9% on predicting the preferences of labelers in the held-out group
- p. 52 Appendix E.2
inter- and intra-group validation accuracies for predicting the human-preferred output are 72.4±0.4%, and 69.6±0.9% respectively
- p. 12 Sec 4.1
69.6 ± 0.9% on predicting the preferences of labelers in the held-out group
Only the authors can say what is right here.
Judge's note: Read on p.12 (Sec 4.1): '69.6 ± 0.9% on predicting the preferences of labelers in the held-out group, a small decrease from their 72.4 ± 0.4% ... in their training set'. Read on p.52 (E.2): inter-/intra- with 72.4/69.6 'respectively'. The numbers agree between the two places. The paper never defines inter/intra-group, so this is a labelling ambiguity or inversion, not a wrong value. The authors should confirm which term is meant.
Second check: another model agreed
Report a problem with this finding
Wording "Our SFT baseline ... also has the lowest continuity and is the least preferred in our rankings" is unqualified, but Figures 40–41 show this only in the respectful-prompt setting.
- p. 14 Section 4.2
but also has the lowest continuity and is the least preferred in our rankings
- p. 60 Figures 40 and 41
Figure 41: Win rates of PPO-ptx and SFT against 175B GPT-3 in RealToxicityPrompts.
Only the authors can say what is right here.
Judge's note: Figure 40 (p. 60), no_prompt: SFT continuity (~5.55) is not below GPT (~5.5) or PPO-ptx (~5.5). Figure 41, no prompt: SFT win rate (~0.48) is slightly above PPO-ptx (~0.47). With the respectful prompt, SFT is clearly lowest in continuity (~4.5) and win rate (~0.36). The authors could state which setting the sentence refers to.
Second check: another model agreed
Report a problem with this finding
Wording Appendix C states "We filter out prompts that are longer than 1k tokens", yet Table 9 (prompt lengths used to train various models) lists maxima of 2048, 2048, 2032, 2039, 2044, 2039 and 1836 tokens, and SFT 75th percentiles of 632/631.
- p. 40 Appendix C, first paragraph
prompts that are longer than 1k tokens
- p. 34 Table 9, Max column
2044
Only the authors can say what is right here.
Judge's note: In Table 9 (p.34) every split has a max above 1k, and Table 11 prompt maxima also reach 2048. The paper does not say whether Table 9 shows lengths before filtering, and A.4 describes it as prompt lengths 'used to train various models'. No explanation was found in the text (grep for 1k/truncat).
Second check: another model agreed
Report a problem with this finding
Wording B.3 reports "results from the 19 respondents in Table 12", but Table 12's gender rows ("Male 50.0% Female 44.4% Nonbinary / other 5.6%") correspond to 18 respondents.
- p. 36 B.3
results from the 19 respondents in Table 12
- p. 40 Table 12 gender and nationality
Male 50.0% Female 44.4% Nonbinary / other 5.6%
Only the authors can say what is right here.
Judge's note: Table 12 (p.40): 50.0/44.4/5.6 equal 9/18, 8/18 and 1/18 exactly, and no count out of 19 gives 50.0%. The age rows (26.3, 47.4, 10.5, 10.5, 5.3) equal 5, 9, 2, 2 and 1 out of 19. The paper does not say whether one respondent skipped the gender question. Only the authors can clarify the denominator.
Second check: another model agreed
Report a problem with this finding
Wording Figure 11 says toxicity, relative toxicity and continuity are rated "on a scale from 0-2", then defines continuity with anchors 1, 4 and 7 ("a 7/7") and refers to plausible outputs that "would get a continuity score of 2".
- p. 38 Figure 11, paragraph 3
and the continuity of the output, on a scale from 0-2
- p. 38 Figure 11, continuity definition and last paragraph
would get a continuity score of 2
- p. 38 Figure 11
on a scale from 0-2
Only the authors can say what is right here.
Judge's note: I read the full Figure 11 box on p.38. Relative toxicity is defined on -1 to 1, continuity on 1 to 7, and the ranking paragraph uses a continuity score of 2. The published instructions are internally inconsistent about the continuity scale. Only the authors can say which scale was used.
Second check: another model agreed
Report a problem with this finding
Wording B.3 says labelers are "young (75% less than 35 years old)", but Table 12 gives 18-24 26.3% + 25-34 47.4% = 73.7%.
- p. 38 Appendix B.3 text continued
young (75% less than 35 years old)
- p. 40 Table 12, age
47.4%
- p. 40 Table 12 age
18-24 26.3% 25-34 47.4%
Judge's note: The Table 12 age rows on p.40 sum to 73.7% (14/19). '75%' is a loose approximation and not a rounding of the table value at the precision printed.
Second check: another model agreed
Report a problem with this finding
Wording C.5 says "we use a batch size of 64 for both experiments" but later says one T0 experiment "used a batch size of 128, a learning rate of 4e-6 and 1.28 million examples".
- p. 42 Appendix C.5
we use a batch size of 64 for both experiments
- p. 42 Appendix C.5, last paragraph
batch size of 128, a learning rate of 4e-6 and 1.28 million examples
- p. 42 C.5, last paragraph
we used a batch size of 128, a learning rate of 4e-6 and 1.28 million examples
Only the authors can say what is right here.
Judge's note: Both statements appear in C.5 on p.42, and the second continues on p.43. They conflict as printed. Only the authors can say which batch size applies to which run.
Second check: another model agreed
Report a problem with this finding
Wording C.5 says T0 was "subsampled ... to 1 million datapoints" with no repeats, yet the chosen T0 run is described as using "1.28 million examples". In Figure 13 (LR = 4e-6) the T0 curve ends near 1.0M and the FLAN curve extends to about 1.28M.
- p. 42 Appendix C.5
we subsampled T0 to 1 million datapoints
- p. 42 Appendix C.5
a learning rate of 4e-6 and 1.28 million examples
- p. 43 Figure 13, LR = 4e-6 panel
Figure 13: Tuning FLAN and T0 based on reward model scores
Only the authors can say what is right here.
Judge's note: I checked the p.42 text and the Figure 13 image on p.43. The figure's T0 line in the LR = 4e-6 panel stops at about 1,000,000 elapsed examples, and only FLAN reaches about 1,280,000. This suggests the 1.28M figure may belong to the FLAN run or a different T0 configuration. The authors would need to clarify.
Second check: another model agreed
Report a problem with this finding
Wording The Figure 36 caption says "The blue line indicates the reward value when the coefficient is zero", but the figure plots Likert scores (the caption title and E.7 both say Likert score).
- p. 57 Figure 36 caption
The blue line indicates the reward value
- p. 57 Figure 36 caption
The blue line indicates the reward value when the coefficient is zero
Judge's note: The p.57 crop shows the caption 'Likert scores as a function of KL reward coefficient' and the blue line at about 2.1 on the Likert axis. 'Reward value' misdescribes what the line shows.
Second check: another model agreed
Report a problem with this finding
Typo Intro (p.3) spells the dataset "CrowSPairs (Nangia et al., 2020)"; elsewhere (e.g. p.14) it is "CrowS-Pairs".
- p. 3 Introduction, toxicity paragraph
CrowSPairs (Nangia et al., 2020)
Judge's note: The p.3 image shows 'CrowSPairs' with no hyphen; the p.14 image shows 'CrowS-Pairs'. This is an inconsistent spelling, not an extraction loss.
Second check: none (judge only)
Report a problem with this finding
Typo Sec 3.2 (p.7): "Most of the use-cases have are generative" has an extra 'have'. Sec 3.1 (p.6) also has "see Sections 3.4", a plural for a single section.
- p. 7 Section 3.2
the use-cases have are generative
- p. 7 Sec 3.2, last paragraph
Most of the use-cases have are generative
- p. 6 3.1
(see Sections 3.4 for details)
- p. 7
Most of the use-cases have are generative, rather than classification or QA.
Judge's note: Both are visible on the page images (p.7 and p.6).
Second check: none (judge only)
Report a problem with this finding
Typo Appendix A.2.1 example reads "Write a short story where a brown bear to the beach, makes friends with a seal, and then return home." (missing verb; 'return'). The Table 2 version reads "where a bear goes to the beach ... and then returns home."
- p. 27 Appendix A.2.1, generation example
Write a short story where a brown bear to the beach
- p. 6 Table 2
Write a short story where a bear goes to the beach
Judge's note: I compared the p.27 table row with Table 2 on p.6. The appendix wording is ungrammatical, but the prompts are fictional and user-style, so it could be deliberate. Treated as a minor copy-editing point.
Second check: none (judge only)
Report a problem with this finding
Typo p.8: "we asked labelers prioritize truthfulness and harmlessness" is missing 'to'.
- p. 8 Section 3.4, top of page
we asked labelers prioritize truthfulness
- p. 8 Sec 3.4, top
we asked labelers prioritize truthfulness and
Judge's note: Visible at the top of the p.8 image.
Second check: none (judge only)
Report a problem with this finding
Typo Table 3 (p.9): "Satisifies constraint provided in the instruction" is misspelled; it should read 'Satisfies'.
- p. 9 Table 3
Satisifies constraint provided in the instruction
Judge's note: The misspelling is visible in the Table 3 image.
Second check: none (judge only)
Report a problem with this finding
Typo Table numbering jumps from "Table 3: Labeler-collected metadata on the API distribution." (p. 9) to "Table 6: Dataset sizes, in terms of number of prompts." (p. 33); no Table 4 or 5 exists.
- p. 9 Table 3
Table 3: Labeler-collected metadata on the API distribution.
- p. 33 Table 6
Table 6: Dataset sizes, in terms of number of prompts.
Judge's note: Grepped both text extracts for table captions: 1, 2, 3, 6, 7, ..., 14. No caption or mention of Table 4 or Table 5 anywhere. Looked at both crops. This is a cosmetic numbering gap and nothing points to a missing table.
Second check: none (judge only)
Report a problem with this finding
Typo Section 4.2 misspells 'evaluations': "As measured by human evaluatoins on the TruthfulQA dataset".
- p. 13 Section 4.2, first paragraph
evaluatoins on the TruthfulQA dataset
- p. 13 Sec 4.2, first paragraph
human evaluatoins on the TruthfulQA dataset
- p. 13
As measured by human evaluatoins on the TruthfulQA dataset
Judge's note: Seen on the p. 13 page image. The unclosed parenthesis that one reviewer also mentions is handled separately as C019.
Second check: none (judge only)
Report a problem with this finding
Typo Unclosed parenthesis in Section 4.2: "than GPT-3 (although the absolute improvement decreases by a couple of percentage points."
- p. 13 Section 4.2, end of first paragraph
GPT-3 (although the absolute improvement decreases by a couple of percentage points.
Judge's note: On the p. 13 image the paragraph ends with no closing parenthesis.
Second check: none (judge only)
Report a problem with this finding
Typo Subject–verb disagreement: "the baseline GPT-3 model aren’t as good at this."
- p. 13 Section 4.2, after Figure 6
the baseline GPT-3 model aren
Judge's note: Seen on the p. 13 page image, in the paragraph after Figure 6.
Second check: none (judge only)
Report a problem with this finding
Typo Wrong verb form: "Our 175B PPO-ptx model is able to reliably answers questions about code" (should be 'answer').
- p. 16 Section 4.3
is able to reliably answers questions about code
- p. 13 Sec 4.2
the baseline GPT-3 model aren’t as good at this
Judge's note: Seen on the p. 16 crop. The second item raised here ('model aren’t') is the same defect as C020.
Second check: none (judge only)
Report a problem with this finding
Typo Grammar in Section 5.1: "However, our approach does provides us with a clear empirical feedback loop" (should be 'does provide').
- p. 17 Section 5.1
our approach does provides us with
- p. 17 Section 5.1, second paragraph
our approach does provides us with a
- p. 18 Sec 5.2
an specific human reference group
- p. 17 Sec 5.1
our approach does provides us
Judge's note: Seen on the p. 17 crop. The 'an specific' item raised here is handled in C028.
Second check: none (judge only)
Report a problem with this finding
Typo Grammar in Section 5.2: "influenced, among others things, by the instructions" (should be 'among other things').
- p. 18 Section 5.2
among others
Judge's note: Seen on the p. 18 crop. The 'an specific' item one reviewer also raises is handled in C028.
Second check: none (judge only)
Report a problem with this finding
Typo Wrong article in Section 5.2: "this alignment technique can align to an specific human reference group" (should be 'a specific').
- p. 18 Section 5.2, 'Stepping back' paragraph
can align to an specific human reference group
Judge's note: Seen on the p. 18 crop.
Second check: none (judge only)
Report a problem with this finding
Typo "Rachelle Froyalde" appears twice in the acknowledged labeler list, after "Emill Jayson Caypuno" and again after "Morris Stuttard".
- p. 21 Acknowledgements, labelers list
Rachelle Froyalde
Judge's note: The p. 21 crop shows the name twice in the same list. This is almost certainly a duplication, although only the authors can rule out two labelers with the same name.
Second check: none (judge only)
Report a problem with this finding
Typo The acknowledgements spell "Irene Soliaman", while the reference list and text spell "Solaiman, I." (Solaiman et al., 2019; Solaiman and Dennison, 2021).
- p. 21 Acknowledgements
Irene Soliaman
- p. 25 References
Solaiman, I. and Dennison, C. (2021)
Only the authors can say what is right here.
Judge's note: Rests on outside knowledge: that the acknowledged OpenAI colleague is the same person as the cited author Solaiman, I. The p. 21 crop shows 'Irene Soliaman'; the p. 25 crop shows 'Solaiman, I.' twice; the text uses 'Solaiman' at lines 236 and 249.
Second check: none (judge only)
Report a problem with this finding
Typo Appendix A.1 misspells bootstrapping: "details on our prompt boostrapping process".
- p. 26 Appendix A.1
prompt boostrapping process
- p. 26
details on our prompt boostrapping process
Judge's note: The page 26 image shows 'boostrapping' at the start of A.1. This is a plain misspelling of 'bootstrapping'.
Second check: none (judge only)
Report a problem with this finding
Typo The Figure 33 caption misspells 'validation' as "regression on validatoin reward".
- p. 55 Figure 33 caption
regression on validatoin reward
- p. 55 Figure 33 caption
not much regression on validatoin reward
- p. 28 A.2.1 rewrite example
Desination will be a red barn on the right
Judge's note: Seen in the p.55 crop. 'Desination' (p.28) sits inside an example prompt quoted in Appendix A.2.1, which may be a verbatim user prompt, so it is not counted as an author typo.
Second check: none (judge only)
Report a problem with this finding
Typo Table 12 nationality percentages (22, 22, 17, then seven entries of 5%) sum to 96%; the implied 1/18 = 5.6% is printed as 5% while 3/18 = 16.7% is printed as 17%.
- p. 40 Table 12, 'What is your nationality?'
Zimbabwean
Judge's note: From the p.40 image: 4+4+3+7 = 18 respondents. The rounding is inconsistent and the column does not total 100%. This is a minor presentation issue.
Second check: none (judge only)
Report a problem with this finding
Typo Figure 19 (QuAC) dataset features print the count as "7.306 examples" with a decimal point. Every other count in these boxes uses comma thousands separators, so it should read 7,306.
- p. 47 Figure 19 dataset features
7.306 examples
Judge's note: The crop of p.47 shows '7.306 examples'. Other boxes, such as Figure 25 on p.49 ('1,500 French / English pairs') and Figure 26 ('2,354 news articles'), use commas.
Second check: none (judge only)
Report a problem with this finding
Typo The Figure 29 caption opens a parenthesis that is never closed: "(compare to zero-shot performance shown in Figure 28".
- p. 52 Figure 29 caption
(compare to zero-shot performance shown in Figure 28
- p. 52 Figure 29 caption
shown in Figure 28
Judge's note: The crop of p.52 shows the caption ending at 'Figure 28' with no closing parenthesis.
Second check: none (judge only)
Report a problem with this finding
Typo Table 14 bolds only one of two tied printed values. In CrowS Pairs basic, GPT XL 0.448 is bold but PPO+ptx XL 0.448 is not. In SST few-shot, GPT 175b 0.944 is bold but PPO 175b 0.944 is not.
- p. 56 Table 14, CrowS Pairs basic row and SST few-shot row
0.448
Judge's note: Checked in the p.56 page image and crop. Unrounded values could differ, so this is an inconsistency in marking, not an error.
Second check: none (judge only)
Report a problem with this finding
Typo The Table 14 header labels model sizes "XL 6b 175b", while the rest of the paper uses 1.3B/6B/175B. 'XL' is never defined in the paper.
- p. 56 Table 14 header
XL 6b 175b
Judge's note: Seen in the p.56 header. A grep for 'XL' finds it only in the Table 14 header. The meaning can be inferred from GPT-3 naming but is not stated.
Second check: none (judge only)
Report a problem with this finding
Typo The Figure 46 caption has a missing space in "completions from GPT-3 175B and InstructGPT175B". The same caption text recurs for Figures 47–50.
- p. 65 Figure 46 caption
InstructGPT175B
Judge's note: The p.65 crop shows 'InstructGPT175B'. A grep finds five caption occurrences, while body text and labels use 'InstructGPT 175B'.
Second check: none (judge only)
Report a problem with this finding
Typo In the Figure 50 labeler demonstration, the quotation marks are typeset as diaeresis accents: "The änxiety lumpïn your throat ... The ëlenched chestör ḧeartacheḟeeling".
- p. 68 Figure 50, Labeler demonstration
The änxiety lumpïn your throat
Judge's note: The p.68 crop shows the garbled accents in the rendered page, so this is not an extraction artefact. Straight double quotes were set as umlaut accents.
Second check: none (judge only)