Language Models are Few-Shot Learners

Brown, Tom B., Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan A, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss et al.

arXiv (Cornell University), 2020 · paper · AI and machine learning

Trains GPT-3, a 175-billion-parameter language model, and tests it on tasks given only through text prompts with a few examples and no fine-tuning. It finds scaling improves few-shot performance, though some datasets remain difficult.

Checked: PDF from arxiv.org, fetched 4 Oct 2026. Page numbers are those of that PDF.

24 findings

Substance Sec 2.4 says "we submit only the 200B few-shot results" and Appendix E lists "13.0B, and 200B (GPT-3)", but the largest model is 175.0B (Table 2.1).

p. 10 Section 2.4, last paragraph
we submit only the 200B few-shot results
p. 46 Appendix E, Procedure and design
13.0B, and 200B
p. 8 Table 2.1
175.0B

Judge's note: Both crops show '200B'. Table 2.1 and the rest of the paper give GPT-3 as 175B, and no 200B model exists in the paper.

Second check: another model agreed

Report a problem with this finding

Substance Winograd fine-tuned SOTA is "90.1" in Table 3.5 (WSC273, cited [SBBC19]) but "93.8" in the Table H.1 Winograd (test) row.

p. 16 Table 3.5, Fine-tuned SOTA Winograd
90.1
p. 63 Table H.1, Winograd row Fine-tune SOTA
93.8
p. 16 Table 3.5, Fine-tuned SOTA row, Winograd column
Fine-tuned SOTA 90.1a
p. 63 Table H.1, Winograd row, Fine-tune SOTA column
Winograd acc test 93.8

Only the authors can say what is right here.

Judge's note: Table 3.5 prints 90.1^a. The Table H.1 Winograd row prints Fine-tune SOTA 93.8. H.1 has a separate WSC (dev) row that is also 93.8, matching the SuperGLUE WSC SOTA in Table 3.8, so the Winograd cell may have been copied from WSC. Nothing in the paper explains the difference. The authors need to say which value is right.

Second check: another model agreed

Report a problem with this finding

Substance Table 3.6 gives PIQA GPT-3 Zero-Shot as "80.5*" (identical to One-Shot), while the text says "GPT-3 achieves 81.0% accuracy zero-shot" and Table H.1 gives 81.0.

p. 17 Table 3.6, GPT-3 Zero-Shot PIQA
80.5*
p. 17 Section 3.5 text
GPT-3 achieves 81.0% accuracy zero-shot, 80.5% accuracy one-shot
p. 63 Table H.1, PIQA zero-shot 175B
78.5 81.0
p. 17 Table 3.6, PIQA column, GPT-3 Zero-Shot row
GPT-3 Zero-Shot 80.5*

Judge's note: The page image of Table 3.6 shows 80.5* bold for both Zero-Shot and One-Shot. The text on the same page gives 81.0 for zero-shot and 80.5 for one-shot. The Table H.1 175B zero-shot value is 81.0. In Fig 3.6 the zero-shot point at 175B sits above the one-shot point. The table cell appears to have been copied from the one-shot row.

Second check: another model agreed

Report a problem with this finding

Substance QuAC GPT-3 One-Shot F1 is "43.3" in Table 3.7 but "43.4" in Table H.1 (175B one-shot).

p. 18 Table 3.7, GPT-3 One-Shot QuAC
43.3
p. 63 Table H.1, Quac one-shot 175B
40.6 43.4
p. 18 Table 3.7, One-Shot row, QuAC
GPT-3 One-Shot 84.0 34.3 43.3
p. 63 Table H.1, Quac one-shot 175B
21.1 26.9 31.9 32.3 37.4 39.0 40.6 43.4

Only the authors can say what is right here.

Judge's note: Table 3.7 shows One-Shot QuAC 43.3. The Table H.1 Quac one-shot 175B value is 43.4. The zero-shot (41.5) and few-shot (44.3) values agree between the two tables. The difference is small but it is the same quantity, and it is unclear which value is right.

Second check: another model agreed

Report a problem with this finding

Substance Text says few-shot GPT-3 achieves "80.2% at 3 digit addition", but Table 3.9 (Few-shot 3D+) and Table H.1 give 80.4.

p. 22 Section 3.9.1 text
80.2% at 3 digit addition
p. 23 Table 3.9, Few-shot 3D+
80.4

Judge's note: The p22 text gives 80.2%. The Table 3.9 image shows Few-shot 3D+ 80.4, and the Table H.1 3D+ 175B few-shot value is 80.4. The other values in the same sentence (100.0, 98.9, 94.2) match the table.

Second check: another model agreed

Report a problem with this finding

Substance Sec 3.9.2 reports few-shot GPT-3 at "66.9% on removing random insertions, 38.6% on cycling letters, 40.2% on the easier anagram task", but Table 3.10 (Few-shot row "37.9 15.1 39.7 67.2") and Figure 3.11 give RI 67.2, CL 37.9, A2 39.7.

p. 23 Section 3.9.2 text
the full GPT-3 model achieving 66.9% on removing
p. 24 Section 3.9.2 text
38.6% on cycling letters, 40.2% on the easier anagram task
p. 23 Table 3.10, Few-shot row
37.9 15.1 39.7 67.2
p. 24 text continuing
38.6% on cycling letters, 40.2% on the easier anagram task, and 15.1%

Judge's note: Checked the page 23 table image and the page 24 text and figure. The text introduces these numbers as 'The few-shot results are shown in Figure 3.11'; that figure plots RI at about 67 for 175B, matching Table 3.10. The text values instead match the Symbol Insertion score of 66.9 in Table C.1. That table's caption says its scores come from a different seed and 'will therefore differ slightly from the scores elsewhere in the paper'. Section 3.9.2 does not mention this, so the explanation does not cover the main text. A1 (15.1) agrees.

Second check: another model agreed

Report a problem with this finding

Substance Sec 3.9.4 says completions came "from four language models ranging in size from 125M to 175B", but Table 3.11 and Table E.1 list eight GPT-3 models (Small, Medium, Large, XL, 2.7B, 6.7B, 13B, 175B).

p. 25 Section 3.9.4
from four language models ranging in size from 125M to 175B (GPT-3) parameters
p. 26 Table 3.11
GPT-3 Small
p. 25 Section 3.9.4
a 160M parameter model with no context and increased output randomness
p. 26 Table 3.11 caption
an unconditional GPT-3 Small model with increased output randomness

Judge's note: Checked the p.25 text and the p.26 and p.47 tables. Eight model rows plus a control appear in both tables. The Table 3.11 caption also says 'five different models', which is a further inconsistency. The 160M-vs-Small point raised in this cluster is handled under C043.

Second check: another model agreed

Report a problem with this finding

Substance The control model is called "a 160M parameter model with no context" (Sec 3.9.4; "the 160M control model", Appendix E), but Table 3.11's caption calls it "an unconditional GPT-3 Small model", and GPT-3 Small is 125M in Table 2.1.

p. 25 Section 3.9.4
a 160M parameter model with no context and increased output randomness
p. 26 Table 3.11 caption
an unconditional GPT-3 Small model with increased output randomness
p. 48 Appendix E
the 160M control model

Only the authors can say what is right here.

Judge's note: Checked p.25, the p.26 caption and the p.48 crop. Table 2.1 lists GPT-3 Small at 125M, and the Figure 3.13 and Table 3.12 captions also say GPT-3 Small. Nothing in the paper reconciles the two sizes, so only the authors can say which is right.

Second check: another model agreed

Report a problem with this finding

Substance Table 3.11 caption says it "compares mean accuracy between ﬁve different models" and Section 3.9.4 says completions came "from four language models ranging in size from 125M to 175B", but Table 3.11 (and Table E.1) list a control plus eight GPT-3 sizes.

p. 26 Table 3.11 caption
This table compares mean accuracy between ﬁve different models
p. 26 Table 3.11 caption
compares mean accuracy between ﬁve different models
p. 25 Section 3.9.4
four language models ranging in size from 125M to 175B
p. 25 Section 3.9.4
from four language models ranging in size from 125M to 175B (GPT-3) parameters

Judge's note: On page 26 Table 3.11 has 9 rows: the control plus GPT-3 Small, Medium, Large, XL, 2.7B, 6.7B, 13B and 175B. The caption says five models. The page 25 text says four. Appendix E (p.46) lists outputs from eight model sizes, and Table E.1 has 9 rows. The paper gives no subset definition that would account for either 'four' or 'five'.

Second check: another model agreed

Report a problem with this finding

Substance Appendix E lists the models as "125M, 350M, 760M, 1.3B, 2.7B, 6.7B, 13.0B, and 200B (GPT-3)"; GPT-3 is 175B everywhere else, including Table E.1 ("GPT-3 175B") and Table 3.11.

p. 46 Appendix E, Procedure and design
13.0B, and 200B (GPT-3) parameter language models
p. 47 Table E.1
GPT-3 175B
p. 25 Sec 3.9.4
from four language models ranging in size from 125M to 175B (GPT-3) parameters
p. 26 Table 3.11 caption
This table compares mean accuracy between five different models

Judge's note: Page 46 image shows '200B (GPT-3)'. Tables E.1, D.1 and 3.11 give 175B. Related wording issue on the same study: Sec 3.9.4 (p25) says "four language models ranging in size from 125M to 175B" and the Table 3.11 caption says "five different models", while Table 3.11 and Table E.1 list eight GPT-3 sizes plus a control.

Second check: another model agreed

Report a problem with this finding

Substance Section 3.9.6 says "We give GPT-3 one human-generated correction and then ask it to correct 5 more", but Figure 3.17 shows three human-written input/output pairs and eight bold GPT-3 outputs.

p. 29 Section 3.9.6
We give GPT-3 one human-generated correction and then ask it to correct 5 more
p. 30 Figure 3.17
In the ﬁrst few examples example both the prompt and the completion are provided by a human
p. 30 Figure 3.17 caption
In the ﬁrst few examples example both the prompt

Only the authors can say what is right here.

Judge's note: In Figure 3.17 on page 30, three plain-text pairs (berries, designer, requested changes) are human-provided, and eight 'Good English output' lines are in bold. The caption itself says 'the first few examples' were human-provided. That fits the figure and contradicts 'one ... 5 more'. The duplicated-word point in note 2 is handled under C050.

Second check: another model agreed

Report a problem with this finding

Substance Section 4 says for Winograd that "The overlap analysis ﬂagged 45% of examples", but Table C.1 gives 164 dirty of 273 (60% dirty; 'Clean Percentage' 40%).

p. 32 Section 4, Winograd bullet
The overlap analysis ﬂagged 45% of examples
p. 45 Table C.1, Winograd row
273 90.2 164 86.2 109 40%
p. 45 Table C.1, Winograd row
Winograd test acc 9 88.6 273 90.2 164 86.2 109 40% -3%
p. 32 Section 4, Winograd bullet
The overlap analysis ﬂagged 45% of examples, and found a 2.6% decrease

Only the authors can say what is right here.

Judge's note: Page 32 text says 45%. Table C.1 on page 45 gives Winograd Total 273, Dirty 164, Clean 109, 40%. The Table C.1 caption says only the scores come from a different seed, so dirty counts are unaffected. The PIQA bullet (29% flagged) matches Table C.1 (71% clean). Nothing in the paper explains the Winograd difference.

Second check: another model agreed

Report a problem with this finding

Substance The Winogender method in Section 6.2.1 says the model "found the option with the lowest probability between the two possible options", which reads as the opposite of a likelihood-based choice.

p. 36 Section 6.2.1
found the option with the lowest probability between the two possible options

Only the authors can say what is right here.

Judge's note: The page 36 image reads 'lowest probability'. The method is described as measuring the ability to 'correctly assign a pronoun', and accuracies are reported (64.17%, 81.7% vs 76.7%). Under that framing the model's choice would be the higher-probability option. The paper does not explain choosing the lowest-probability option, so the authors need to confirm which rule was used.

Second check: another model agreed

Report a problem with this finding

Substance Table C.1 'Relative Difference Clean vs All' prints "20%" for Quac (clean 54.1 vs all 44.3) and "-21%" for DROP (clean 29.5 vs all 36.5). The printed scores give about +22% and -19%.

p. 45 Table C.1, Quac row
44.3 7353 44.3 7315 54.1 38 1% 20%
p. 45 Table C.1 Quac row
Quac dev f1 13 44.3 7353 44.3 7315 54.1 38 1% 20%
p. 45 Table C.1 DROP row
DROP dev f1 13 36.5 9536 37.0 8898 29.5 638 7% -21%
p. 45 Table C.1, DROP row
36.5 9536 37.0 8898 29.5 638 7% -21%

Only the authors can say what is right here.

Judge's note: The caption defines the column as the percent change between clean and all. Other rows reproduce: PIQA -4%, De→En -5%, Anagrams 2 -7%, CB -2%, WSC 3%. Rounding bounds: Quac is at least 54.05/44.35-1 = +21.9%; DROP is at most -19.4% in magnitude (29.45/36.55-1). The seed note in the caption applies to both columns equally, so it does not explain the gap. Measuring against the dirty score (DROP -20.3%) and dividing by the clean score also fail to reproduce the printed values. Only the authors can say which number is correct.

Second check: another model agreed

Report a problem with this finding

Substance Appendix E says "We recruited 718 unique participants to take part in 6 experiments", but Table E.1 lists 9 experiments (control plus 8 models) whose recruited counts sum to 713; 713-97 = 616, not the stated 621.

p. 46 Appendix E, Participants
We recruited 718 unique participants to take part in 6 experiments. 97 participants were excluded
p. 47 Table E.1, Participants Recruited column
76 80 80 81 79 80 76 81 80
p. 46 Appendix E Participants
We recruited 718 unique participants to take part in 6 experiments.
p. 47 Table E.1 Participants Recruited column
Control 76

Only the authors can say what is right here.

Judge's note: Read the Table E.1 columns from the page 47 image. Recruited: 76, 80, 80, 81, 79, 80, 76, 81, 80. Excluded: 7, 7, 7, 24, 14, 11, 5, 13, 9, which sum to 97 and match the text. Table E.2 (the ~500-word study) is described separately as 160 participants in 2 experiments, so it does not account for the gap. Nothing in the text explains the extra 5 participants or the count of 6 experiments.

Second check: another model agreed

Report a problem with this finding

Substance Table E.1 prints the same gender split "46:28:2" (=76) for GPT-3 Medium, Large, 6.7B and 13.0B, whose recruited-minus-excluded counts are 73, 57, 71 and 68; the column sums (367 m, 274 f, 10 other) also differ from the text's "343 male, 271 female, and 7 other".

p. 47 Table E.1, Medium/Large/6.7B/13.0B rows
81 24 46:28:2
p. 47 Table E.1, GPT-3 Medium / Large / 6.7B / 13.0B rows
46:28:2
p. 46 Appendix E Participants
leaving a total of 621 participants: 343 male, 271 female, and 7 other
p. 46 Appendix E Participants
343 male, 271 female, and 7 other

Only the authors can say what is right here.

Judge's note: The other five rows match recruited minus excluded exactly: Control 69, Small 73, XL 65, 2.7B 69, 175B 71. So the split counts retained participants, and the four repeated rows break that pattern. The Table E.2 control row is judged separately under C070.

Second check: another model agreed

Report a problem with this finding

Substance Table E.2 Control row "79 17 32:37:0 39" has a gender split summing to 69, but 79-17 = 62 participants remain; the split and mean age are identical to the Table E.1 control row (76-7 = 69).

p. 48 Table E.2, Control row
79 17 32:37:0
p. 48 Table E.2, Control row
Control 79 17 32:37:0 39 569:464
p. 48 Table E.2 Control row
79 17 32:37:0 39 569:464

Only the authors can say what is right here.

Judge's note: The page 48 image confirms the row. The GPT-3 175B row in Table E.2 is consistent: 81-19 = 62 = 32+30. Nothing explains the control row, which looks copied from Table E.1.

Second check: another model agreed

Report a problem with this finding

Reference or label One-Shot bullet (p.6) says "as shown in Figure 1"; no Figure 1 exists, and the illustration is Figure 2.1.

p. 6 Section 2, One-Shot bullet
in addition to a natural language description of the task, as shown in Figure 1.

Judge's note: The crop confirms the text "Figure 1". Figures are numbered 1.1, 1.2, 2.1 and so on, and the neighbouring text cites Figure 2.1 for the same illustration.

Second check: another model agreed

Report a problem with this finding

Reference or label Ro–En overall SOTA is cited as "[LHCG19b]" (Liu et al., 'Multi-task deep neural networks for natural language understanding'), whereas Table 3.4 attributes the Ro→En SOTA 39.9 to e[LGG+20] (mBART, 'Multilingual denoising pre-training for neural machine translation').

p. 16 Section 3.3
supervised ﬁnetuning on 608K labeled examples, and backtranslation [LHCG19b]
p. 15 Table 3.4 footnote e
e[LGG+20]
p. 71 Reference list
[LHCG19b] Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. Multi-task deep neural networks for natural language understanding.

Judge's note: Table 3.4 on p15 marks the Ro→En 39.9 with footnote e = [LGG+20]. The reference list on p71 shows that [LHCG19b] is the MT-DNN NLU paper. The text on p16 cites [LHCG19b] for the same SOTA, so the citation is inconsistent with the table and looks mis-keyed.

Second check: another model agreed

Report a problem with this finding

Reference or label Figure 3.9 caption says ANLI R3 "Results are on the dev-set, which has only 1500 examples", while Table C.1 lists ANLI R3 as split 'test' with 1200 examples, and Table H.1 also labels it 'test' with the same 40.2 few-shot result.

p. 21 Figure 3.9 caption
Results are on the dev-set, which has only 1500 examples
p. 45 Table C.1, ANLI R3 row
ANLI R3 test acc 13 40.2 1200

Only the authors can say what is right here.

Judge's note: Checked the Figure 3.9 caption on p21 and Table C.1 on p45 (ANLI R3 test acc 13 40.2 1200). The Table H.1 ANLI R3 row shows split test and 175B few-shot 40.2. Both the split and the example count conflict, and nothing in the paper reconciles them.

Second check: another model agreed

Report a problem with this finding

Reference or label Section 3.9.5 says "Table 3.16 shows the 6 examples we generated", but the examples are in Figure 3.16.

p. 29 Section 3.9.5
Table 3.16 shows the 6 examples we generated

Judge's note: On page 29 the boxed novel-word examples carry the caption 'Figure 3.16'. No Table 3.16 exists.

Second check: another model agreed

Report a problem with this finding

Reference or label Figure H.11 ("All results for all Translation tasks."): all twelve panels are titled '(SacreBLEU)' or '(Multi-BLEU)' but each y-axis is labelled 'Accuracy'.

p. 67 Figure H.11, all panels' y-axis
Figure H.11: All results for all Translation tasks.

Judge's note: I looked at the crop and at the full page 67 image, because this page's text layer is OCR. Every translation panel (De/Fr/Ro↔En, SacreBLEU and Multi-BLEU) has the y-axis label 'Accuracy', yet the panel titles name a BLEU metric and the values (e.g. De→En few-shot about 40) are on a BLEU scale. The plotted data are fine; the axis label names the wrong metric.

Second check: another model agreed

Report a problem with this finding

Wording Sec 3.9.1 defines 1DC as having "parentheses around the last two" (e.g. "6+(4*8)"), but the Appendix G example (Figure G.42) reads "Q: What is (2 * 4) * 6?", with parentheses around the first two.

p. 22 Section 3.9.1, 1DC bullet
with parentheses around the last two. For example, “Q: What is 6+(4*8)? A: 38”
p. 61 Figure G.42
Q: What is (2 * 4) * 6?

Only the authors can say what is right here.

Judge's note: Checked both page images. Page 22 defines 1DC with parentheses around the last two operands and gives 6+(4*8). The results text on the same page uses 9*(7+5), which matches that definition. Figure G.42 on p.61 shows (2 * 4) * 6. Either the dataset contains both forms or the definition or example is wrong; only the authors can say which.

Second check: another model agreed

Report a problem with this finding

Wording Sec 3.9.2 says one-shot performance is "significantly weaker (dropping by half or more)", but Table 3.10 shows one-shot at 57–68% of few-shot on CL, A1, A2 and RI, and RW rises slightly.

p. 24 Section 3.9.2
In the one-shot setting, performance is signiﬁcantly weaker (dropping by half or more)
p. 23 Table 3.10
21.7 8.62 25.9 45.4
p. 24 Section 3.9.2
performance is signiﬁcantly weaker (dropping by half or more)
p. 23 Table 3.10
GPT-3 One-shot 21.7 8.62 25.9 45.4 0.48

Only the authors can say what is right here.

Judge's note: Read Table 3.10 on the p.23 image. Few-shot to one-shot: CL 37.9 to 21.7, A1 15.1 to 8.62, A2 39.7 to 25.9, RI 67.2 to 45.4, RW 0.44 to 0.48. None of these is a drop of half or more. The zero-shot setting is covered by the next clause, so it cannot be what this clause refers to.

Second check: another model agreed

Report a problem with this finding

39 minor (typos and the like; most checked by the judge only)

Substance Table 2.1 GPT-3 XL row: "1.3B 24 2048 24 128". n_heads × d_head = 3072, not d_model = 2048, whereas the product equals d_model in every other row.

p. 8 Table 2.1, GPT-3 XL row
GPT-3 XL 1.3B 24 2048 24 128

Only the authors can say what is right here.

Judge's note: Read on the table image. The paper never states that d_head = d_model/n_heads, so the row is not provably wrong. However, every other row satisfies that relation (12*64=768, 16*64=1024, 16*96=1536, 32*80=2560, 32*128=4096, 96*128=12288), so n_heads or d_head may be misprinted. Only the authors can say which.

Second check: another model agreed

Report a problem with this finding

Substance Table 2.1 GPT-3 13B row prints d_model "5140" with 40 heads × 128 (= 5120); 5140 is not even divisible by 40.

p. 8 Table 2.1, GPT-3 13B row
5140

Only the authors can say what is right here.

Judge's note: Confirmed on the table image. 40*128 = 5120, and 5140/40 = 128.5 is not an integer number of dimensions per head. This suggests a misprint for 5120, but only the authors can confirm it.

Second check: another model agreed

Report a problem with this finding

Reference or label Section 6.3 cites Figure 2.2 for "tens of petaﬂop/s-days for a 1.5B parameter GPT-2 model", but Figure 2.2 contains no GPT-2 bar.

p. 39 Section 6.3 Energy Usage
compared to tens of petaﬂop/s-days for a 1.5B parameter GPT-2 model (Figure 2.2)
p. 9 Figure 2.2
Total compute used during training

Judge's note: Figure 2.2 on page 9 shows BERT, RoBERTa, T5 and GPT-3 Small through 175B only. The 175B 'several thousand' part of the sentence is supported by the figure. The GPT-2 1.5B figure is not.

Second check: another model agreed

Report a problem with this finding

Reference or label Figure 3.7 (CoQA) y-axis is labelled "Accuracy", but the plotted CoQA metric is F1 (caption: "achieves 85 F1"; Table 3.7: "All scores are F1 except ... RACE").

p. 19 Figure 3.7 y-axis
Accuracy
p. 18 Table 3.7 caption
All scores are F1 except results for RACE which report accuracy.

Judge's note: The page 19 image shows the y-axis label 'Accuracy'. The plotted values (85.0 few-shot at 175B) match the F1 scores in Table 3.7.

Second check: another model agreed

Report a problem with this finding

Reference or label Figure G.35 caption reads "Formatted dataset example for WebQA", whereas the dataset is called WebQuestions/WebQs elsewhere (Table C.1, Table H.1, Figure H.7).

p. 60 Figure G.35 caption
Formatted dataset example for WebQA

Judge's note: Page 60 image confirms 'WebQA'. The example is a WebQuestions-style item, and the rest of the paper uses WebQs/WebQS, so the label names the dataset inconsistently.

Second check: another model agreed

Report a problem with this finding

Wording Abstract says fine-tuning datasets are "of thousands or tens of thousands of examples", but the Introduction (p.3) and Section 2 (p.6) say "thousands to hundreds of thousands".

p. 1 Abstract
thousands or tens of thousands of examples
p. 3 Introduction, para 1
a dataset of thousands to hundreds of thousands
p. 6 Section 2, Fine-Tuning bullet
Typically thousands to hundreds of thousands of labeled examples are used

Judge's note: Checked all three crops. The wording on the page is as quoted, so the upper end of the range differs between the abstract and the body. Both are loose descriptions of typical dataset sizes, so this is a minor consistency point.

Second check: another model agreed

Report a problem with this finding

Wording Sec 2.2 says "the other datasets are sampled 2-3 times", but Table 2.2 gives Wikipedia 3.4 and Books1 1.9 epochs (the caption says "up to 3.4 times").

p. 8 Section 2.2, last paragraph
the other datasets are sampled 2-3 times
p. 9 Table 2.2 caption
some datasets are seen up to 3.4 times

Judge's note: Checked the p.8 text and the p.9 table. Two of the three 'other' datasets fall outside the stated 2-3 range. The text is a loose approximation of the table values.

Second check: another model agreed

Report a problem with this finding

Wording Figure 2.2 caption: "although GPT-3 3B is almost 10x larger than RoBERTa-Large (355M params)". No GPT-3 3B model exists (the figure and Table 2.1 have GPT-3 2.7B), and 2,650M/355M is about 7.5x.

p. 9 Figure 2.2 caption
although GPT-3 3B is almost 10x larger than RoBERTa-Large (355M params)
p. 9 Figure 2.2 caption
GPT-3 3B is almost 10x larger than RoBERTa-Large (355M params)

Only the authors can say what is right here.

Judge's note: Viewed the p.9 caption and the figure x-axis labels (GPT-3 2.7B). Table D.1 lists GPT-3 2.7B at 2,650M params and 55.2 PF-days, and RoBERTa-Large at 355M and 49.3 PF-days, so the caption evidently refers to the 2.7B model. The name used in the caption and the 'almost 10x' ratio do not match the tables.

Second check: another model agreed

Report a problem with this finding

Wording Text says "across all language pairs and across all three settings (zero-, one-, and few-shot), there is a smooth trend of improvement with model capacity", but the Table H.1 zero-/one-shot BLEU values are non-monotonic.

p. 16 Section 3.3, last paragraph
across all language pairs and across all three settings (zero-, one-, and few-shot), there is a smooth trend of improvement with model capacity
p. 63 Table H.1, Ro→En zero-shot
3.15 16.3 8.34 20.2 19.9

Only the authors can say what is right here.

Judge's note: In Table H.1, Ro→En BLEU-mb zero-shot runs 3.15, 16.3 (2.7B), 8.34 (6.7B), 20.2, 19.9. En→Fr zero-shot runs 15.1 then 8.82. De→En one-shot falls from 33.0 (13B) to 30.4 (175B). 'Smooth trend' overstates the zero- and one-shot data. The few-shot results (Fig 3.4) do look smooth.

Second check: another model agreed

Report a problem with this finding

Wording The mean length of the human-written newser articles is "215 words" in Sec 3.9.4 but "216" in every row of Table E.1 (human:model word count).

p. 25 Section 3.9.4
(mean length: 215 words)
p. 47 Table E.1, Average Word Count (human:model)
216:216
p. 25 Sec 3.9.4
newser.com (mean length: 215 words)

Judge's note: Checked the p.25 text and the p.47 Table E.1 image. It is the same 25 human articles, given as 215 and 216. The gap could come from rounding or from different counting methods, and neither is described.

Second check: another model agreed

Report a problem with this finding

Wording Table 3.11: for GPT-3 Medium, "10.3 (7e-21)" and the declared two-sample t-test do not correspond, given that "10.4 (5e-19)" and "10.7 (1e-19)" are printed for groups of similar size.

p. 26 Table 3.11, GPT-3 Medium row
10.3 (7e-21)
p. 26 Table 3.11, GPT-3 XL / 2.7B rows
10.7 (1e-19)

Only the authors can say what is right here.

Judge's note: Footnote 5 declares a two-sample Student's t-test. Table E.1 gives post-exclusion group sizes: Control 76-7=69, Medium 80-7=73 (df=140), 2.7B 80-11=69 (df=136), XL 79-14=65 (df=132), 6.7B 76-5=71 (df=138, t=11.2, p=3e-21). With these similar df the p value falls as t rises, so t=10.3 should give a p value above 5e-19, not 7e-21. Under the declared test, the t and p printed for Medium do not correspond. Only the authors can say which of the two is right.

Second check: another model agreed

Report a problem with this finding

Wording Section 6.2.2 says 'Asian' "ranked 1st in 3 out of 7 models", but in Figure 6.1 the Asian line appears highest at four sizes (760M, 1.3B, 6.7B, 13B).

p. 37 Section 6.2.2, last paragraph
it ranked 1st in 3 out of 7 models
p. 38 Figure 6.1
Racial Sentiment Across Models

Only the authors can say what is right here.

Judge's note: Reading Figure 6.1 on page 38: Indian is highest at 350M and 175B, Latinx at 2.7B, and Asian at 760M, 1.3B, 6.7B and 13B. The 'Black lowest in 5 of 7' claim matches the plot. The plot is low-resolution and the gaps at 6.7B and 13B are a few points, so this is put to the authors as a query.

Second check: another model agreed

Report a problem with this finding

Wording Appendix C says DROP has "94% of the examples are dirty" and "the 6% of examples that remain after filtering", but Table C.1 gives 8898 dirty and 638 clean of 9536 (93.3% / 6.7%), with Clean Percentage printed as 7%.

p. 44 Appendix C, Overlap results
DROP, a reading comprehension task in which 94% of the examples are dirty
p. 45 Table C.1, DROP row
9536 37.0 8898 29.5 638 7%
p. 45 Table C.1, Quac row
Quac dev f1 13 44.3 7353 44.3 7315 54.1 38 1% 20%
p. 45 Table C.1, DROP row
DROP dev f1 13 36.5 9536 37.0 8898 29.5 638 7% -21%

Judge's note: Checked the page 44 text and the Table C.1 DROP row on page 45. 6.7% truncated to 6% would give 94% dirty, so this looks like rounding down. Even so, the text does not match the table's own 7%. Nothing on the page explains the difference. The relative-difference part of this cluster is judged under C063.

Second check: another model agreed

Report a problem with this finding

Wording Table D.1 T5 rows print "Fwd-pass flops per active param per token" = 1, but the caption says each token takes "a single addition and a single multiply for each active parameter" (2 flops); with frac 0.5 and mult 3 the columns give 1.5, not the printed 3 flops per param per token.

p. 46 Table D.1, T5-Small..T5-11B rows
T5-Small 2.08E+00 1.80E+20 60 1,000 3 3 1 0.5
p. 46 Table D.1 caption
each token is involved in a single addition and a single multiply for each active parameter in the forward pass

Only the authors can say what is right here.

Judge's note: The printed totals are correct: params × tokens × 3 gives 60e6*1e12*3 = 1.8e20 and 11e9*1e12*3 = 3.3e22, matching the table, so the reviewer's claim that the totals are half is wrong (it applies the 0.5 fraction twice). The real inconsistency is the T5 forward-pass cell of 1. Under the caption's definition it should be 2, since 3×2×0.5 = 3. The GPT-3 rows are consistent (3×2×1 = 6).

Second check: another model agreed

Report a problem with this finding

Wording Table H.1 MultiRC acc ("4.72 9.65 12.3 13.6 14.3 18.4 24.2 27.6") and f1a ("57.0 59.7 60.4 59.9 60.0 64.5 71.4 72.9") are identical in the Zero-Shot and One-Shot columns for all eight models.

p. 63 Table H.1, MultiRC acc and f1a rows
4.72 9.65 12.3 13.6 14.3 18.4 24.2 27.6

Only the authors can say what is right here.

Judge's note: The crop image confirms all 16 values are repeated, while every other SuperGLUE task differs between the two settings. The printed SuperGLUE averages are consistent with these values (zero-shot Small gives 40.6, one-shot Small 54.4, one-shot 175B 68.9), so the authors' averages used them. Whether the repetition is real or a duplicated column can only be settled by the authors.

Second check: another model agreed

Report a problem with this finding

Typo Footnote 1 prints "the term “in context-learning” to refer to the inner loop"; it should read "in-context learning".

p. 4 Footnote 1
the term “in context-learning” to refer to the inner

Judge's note: The page-4 crop shows the hyphen misplaced in the footnote. The body text on the same page uses "in-context learning".

Second check: none (judge only)

Report a problem with this finding

Typo Duplicated word on p.5: "and in the the few-shot setting is sometimes competitive".

p. 5 Introduction, paragraph 'Broadly, on NLP tasks'
and in the the few-shot setting
p. 5 paragraph after Fig 1.3
and in the the few-shot
p. 5
and one-shot settings, and in the the few-shot setting is

Judge's note: The page image shows "the the" printed on a single line, so this is not an extraction artefact.

Second check: code check

Report a problem with this finding

Typo Figure 2.1 few-shot panel prints "plush girafe => girafe peluche", while the fine-tuning panel prints "plush giraffe => girafe peluche"; the English word is misspelled in the few-shot panel.

p. 7 Figure 2.1, few-shot panel line 4
plush girafe => girafe peluche

Judge's note: Viewed the full p.7 image. The few-shot panel line 4 reads "plush girafe" and the fine-tuning panel reads "plush giraffe". "girafe" is correct only on the French side.

Second check: none (judge only)

Report a problem with this finding

Typo Duplicated word on p.10: "the 175 billion parameter parameter GPT-3".

p. 10 Section 3, second paragraph
the 175 billion parameter parameter GPT-3
p. 10 Section 3 intro
175 billion parameter parameter GPT-3
p. 10
in Section 2 (the 175 billion parameter parameter GPT-3 and 7 smaller

Judge's note: The p.10 crop shows the repeated word on a single line.

Second check: code check

Report a problem with this finding

Typo Sec 3.2 misspells "auxilliary information" (should be "auxiliary").

p. 13 Section 3.2
without conditioning on auxilliary information
p. 13 Sec 3.2
auxilliary information

Judge's note: The p.13 crop shows "auxilliary".

Second check: none (judge only)

Report a problem with this finding

Typo Stray word: "Zero-shot GPT-3, which only receives on a natural language description of the task" ('on' is extraneous).

p. 14 Section 3.3, last paragraph
which only receives on a natural language description of the task

Judge's note: The page image shows 'only receives on a natural language description'. The word 'on' is superfluous.

Second check: none (judge only)

Report a problem with this finding

Typo Model name spelled "RoBERTA-large" (Fig 3.5 caption) and "RoBERTA model" (Sec 3.4) where the rest of the paper has "RoBERTa".

p. 16 Figure 3.5 caption
ﬁne-tuned RoBERTA-large
p. 17 Sec 3.4
RoBERTA model achieves 79%

Judge's note: Both instances are visible in the images. Elsewhere (p17 lower text, Sec 3.7, references) the paper uses RoBERTa.

Second check: none (judge only)

Report a problem with this finding

Typo SuperGLUE citation list begins with a duplicated citation: "[WPN+19] [WPN+19] [CLC+19]".

p. 18 Section 3.7
[WPN+19] [WPN+19] [CLC+19]

Judge's note: The image shows the repeated citation, and the reference list has only one [WPN+19] entry (SuperGLUE).

Second check: none (judge only)

Report a problem with this finding

Typo Figure 3.10 caption reads "(GPT-3 175), with the latter being able to reliably accurate 2 digit arithmetic": the model name lacks 'B' and the clause lacks a verb.

p. 22 Figure 3.10 caption
(GPT-3 175), with the latter being able to reliably accurate 2 digit arithmetic

Judge's note: Seen in the page 22 image as quoted. The likely fix is 'GPT-3 175B' and 'able to reliably perform accurate 2 digit arithmetic'.

Second check: none (judge only)

Report a problem with this finding

Typo Table 3.10 caption: "A1 is anagrams of but the first and last letters" is missing 'all'.

p. 23 Table 3.10 caption
A1 is anagrams of but the ﬁrst and last letters
p. 23 Sec 3.9.2 bullet
Anagrams of all but ﬁrst and last characters (A1)

Judge's note: The page 23 image shows the missing word. The bullet on the same page reads 'Anagrams of all but first and last characters (A1)'.

Second check: none (judge only)

Report a problem with this finding

Typo Sec 3.9.3: "with the the full 175 billion model" has a duplicated 'the'.

p. 24 Section 3.9.3, last sentence
with the the full 175 billion model
p. 24
ults improve with scale, with the the full 175 billion mo

Judge's note: Seen on the last line of p.24 in the page image. There is no line break between the two words.

Second check: code check

Report a problem with this finding

Typo Figure 3.12 caption: "Zero-, one-,and few-shot" is missing a space after the comma.

p. 25 Figure 3.12 caption
Zero-, one-,and few-shot

Judge's note: Seen in the p.25 caption image.

Second check: none (judge only)

Report a problem with this finding

Typo Figure 3.17 caption has a stray extra word: "In the ﬁrst few examples example both the prompt".

p. 30 Figure 3.17 caption
In the ﬁrst few examples example both the prompt

Judge's note: The page 30 image shows 'examples example' in the caption.

Second check: none (judge only)

Report a problem with this finding

Typo Section 5 writes "(WIC and ANLI respectively)", whereas the dataset is written 'WiC' elsewhere in the paper.

p. 33 Section 5 Limitations
(WIC and ANLI respectively)

Judge's note: The page 33 image shows 'WIC'. This is a cosmetic inconsistency in capitalisation.

Second check: none (judge only)

Report a problem with this finding

Typo Section 5 has an extra determiner: "than a human sees in the their lifetime".

p. 34 Section 5, sample efficiency paragraph
than a human sees in the their lifetime
p. 34 Sec 5, sample efficiency
sees in the their lifetime

Judge's note: Visible on the page 34 image.

Second check: none (judge only)

Report a problem with this finding

Typo Section 5 has a grammar slip: "at the scale of hundred of billions parameters" (should be 'hundreds of billions of parameters').

p. 34 Section 5, distillation paragraph
at the scale of hundred of billions parameters

Judge's note: Visible on the page 34 image.

Second check: none (judge only)

Report a problem with this finding

Typo The occupation-bias formula has an unbalanced parenthesis: "log(P(female|Context) / P(male|Context)) )".

p. 36 Section 6.2.1, average occupation bias formula
P (male|Context)) ) - was −1.11

Judge's note: In the rendered formula on page 36 the denominator shows 'P(male|Context))' followed by the closing ')' of log(. That makes three opening and four closing parentheses.

Second check: none (judge only)

Report a problem with this finding

Typo Appendix A item 1 misspells Wikipedia: "such as WebText, Wikiedia, and our web books corpus".

p. 43 Appendix A, item 1
such as WebText, Wikiedia, and our web books
p. 43 Appendix A item 1
WebText, Wikiedia, and our web books
p. 43 Appendix A item 1
Wikiedia
p. 60 Figure G.36 caption
other langauge tasks

Judge's note: Page 43 image shows 'Wikiedia'. The 'langauge' and 'WebQA' items in this cluster are judged separately under C072 and C073.

Second check: none (judge only)

Report a problem with this finding

Typo Appendix E sentence has no main verb: "The same output procedure for each model with the exception of the removal of the intentionally bad control model, as described in the main text."

p. 46 Appendix E, Procedure and design
The same output procedure for each model with the exception of the removal of the intentionally bad control model

Judge's note: Page 46 image confirms the sentence as printed. It is missing a verb such as 'was used'.

Second check: none (judge only)

Report a problem with this finding

Typo Figure G.24 target completion prints a brace instead of an en dash: "(July 6, 1931 { November 19, 2017)".

p. 56 Figure G.24 target completion
(July 6, 1931 { November 19, 2017)

Judge's note: The crop image shows a literal '{' in the typewriter font, so this is not an extraction artefact.

Second check: none (judge only)

Report a problem with this finding

Typo Figure G.36 caption misspells language: "for this and other langauge tasks".

p. 60 Figure G.36 caption
other langauge tasks
p. 60
for this and other langauge tasks, the format for zero-shot learning

Judge's note: Page 60 image shows 'langauge'.

Second check: none (judge only)

Report a problem with this finding

Typo Figure H.7 NaturalQuestions panel x-axis uses tick labels "12.8B" and "174.6B" while the TriviaQA and WebQS panels (and other figures) use "13B" and "175B".

p. 66 Figure H.7 NaturalQuestions panel x-axis
12.8B 174.6B

Judge's note: The page 66 image shows the inconsistent ticks. The values match the Table D.1 parameter counts (12,850M; 174,600M), so this is only a labelling inconsistency.

Second check: none (judge only)

Report a problem with this finding

Typo Reference [LPP+20] ends its author list with "Sebastian Riedel, and Kiela Douwe", reversing the given name and surname; [NWD+19] in the same list gives "Douwe Kiela".

p. 71 Reference [LPP+20]
Sebastian Riedel, and Kiela Douwe

Judge's note: The crop of page 71 shows 'and Kiela Douwe.' in [LPP+20]. Within the same reference list, [NWD+19] spells the name 'Douwe Kiela', so the paper's own list shows the correct order.

Second check: none (judge only)

Report a problem with this finding

Typo The reference list has two entries for the same work: [ZSW+19a] (p.74) and [ZSW+19b] (p.75), both "Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences", 2019.

p. 74 Reference list
[ZSW+19a] Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences, 2019.
p. 75 Reference list
[ZSW+19b]
p. 74 Reference list
[ZSW+19a] Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu
p. 75 Reference list
[ZSW+19b] Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu

Judge's note: The crops of pages 74 and 75 show the same authors, title and year in both entries; only 19b adds 'ArXiv, abs/1909.08593'. The text cites [ZSW+19a] once (Section 5 text) and [ZSW+19b] once (Section 7 text), so the two keys read as if they were different works. The two entries should be merged.

Second check: none (judge only)

Report a problem with this finding

Something wrong with this page, or a finding we missed?