Language Models are Unsupervised Multitask Learners
2019 · paper · AI and machine learning
Shows that language models trained on millions of webpages begin to perform tasks like question answering and translation without explicit supervision. The 1.5-billion-parameter GPT-2 achieves state-of-the-art results on 7 of 8 language modeling datasets in a zero-shot setting.
Checked: PDF from cdn.openai.com, fetched 4 Oct 2026. Page numbers are those of that PDF.
2 findings
Reference or label Table 15, first block: the cell headed "French reference" contains "Un homme expliquait que le fonctionnement de la hernia fonctionnelle qu'il avait reconnaît avant de faire, le fonctionnement de la hernia fonctionnelle que j'ai réussi, j'ai réussi.", which is the GPT-2 French translation of the second block, not the French reference for the "The Next Day Extra" sentence.
- p. 22 Table 15, first block, 'French reference' cell
Un homme expliquait que le fonctionnement de la hernia fonctionnelle qu’il avait reconnaît avant de faire
- p. 22 Table 15, second block, 'GPT-2 French translation' cell
Un homme expliquait que le fonctionnement de la hernia
- p. 22 Table 15, first block, 'GPT-2 English translation' cell
This release, originally released on three discs: the origi-
Only the authors can say what is right here.
Judge's note: Viewed the page 22 image. Block 1: English reference is about the re-release "The Next Day Extra" on three disks, and the GPT-2 English translation beside the French reference cell ("This release, originally released on three discs: the original album, sessions studio inédites and remixes, a DVD...") is clearly a translation of a French sentence about that album. The "French reference" cell instead holds the hernia text, word for word identical to the "GPT-2 French translation" cell of block 2. Blocks 2 and 3 each pair an English and a French reference of the same sentence, so the layout is not meant to mix examples. The correct French reference for block 1 is not printed anywhere in the paper, so only the authors can supply it.
Second check: another model agreed
Report a problem with this finding
Wording Equation (1) is printed as p(x) = prod_{i=1}^{n} p(s_n | s_1, ..., s_{n-1}): the product runs over i but the factor is indexed by n; it should be p(s_i | s_1, ..., s_{i-1}).
- p. 2 Section 2, Equation (1)
p(sn|s1, ..., sn−1)
Judge's note: Read on the crop: the product has i=1 below and n above, and the factor carries subscripts n and n-1, so i never appears and the expression as printed is one term multiplied n times, not the chain-rule factorisation the sentence describes. The intended meaning is clear from the prose, so this is a notational slip (S3), with an evident fix. The preceding sentence also uses n for both the number of examples (x_1..x_n) and the sequence length (s_1..s_n). The later conditional p(s_{n-k},...,s_n | s_1,...,s_{n-k-1}) is fine as written.
Second check: another model agreed
Report a problem with this finding
16 minor (typos and the like; most checked by the judge only)
Wording "As seen in Figure 2 performance steadily improves as model size is increased", but Table 3 CBT-CN accuracy is 93.45 for the 762M model and 93.30 for the 1542M model; the quoted "new state of the art results of 93.3% on common nouns" is below the paper's own 762M figure.
- p. 5 Section 3.2
performance steadily improves as model size is increased
- p. 5 Table 3, CBT-CN column, rows 762M and 1542M
93.45
- p. 5 Section 3.2
GPT-2 achieves new state of the art results of 93.3% on common nouns
- p. 5 Table 3, CBT-CN column, row 1542M
93.30
Judge's note: Table 3 on the page 5 image: CBT-CN 87.65, 92.35, 93.45, 93.30; CBT-NE 83.4, 87.1, 88.0, 89.05. The Common Nouns panel of Figure 2 shows the same slight dip at the last point. The statement holds for named entities and for three of four steps on common nouns; the drop is 0.15 points. No explanation of the dip found in the text. Wording overstatement.
Second check: another model agreed
Report a problem with this finding
Wording Figure 2 plots the "Bajgar et al. (2016)" reference lines at about 83.7 (common nouns) and 78.4 (named entities), while Table 3 gives the CBT SOTA attributed to Bajgar et al. (2016) as 85.7 and 82.3; the caption also reads "Human performance are from".
- p. 5 Figure 2 caption
Human performance are from Bajgar et al. (2016)
- p. 5 Figure 2, dashed line 'Bajgar et al. (2016)' in both panels
Bajgar et al. (2016)
- p. 5 Table 3, SOTA row, CBT-CN and CBT-NE columns (caption: CBT results are from Bajgar et al., 2016)
85.7
Only the authors can say what is right here.
Judge's note: Measured on the crop: in the Common Nouns panel the 85 and 80 ticks are 60 px apart and the dashed line is 16 px below 85, about 83.7; in the Named Entities panel the line is 19 px below 80, about 78.4. Table 3 SOTA row prints 85.7 and 82.3 and its caption says "CBT results are from (Bajgar et al., 2016)". The paper gives no statement that the figure plots a different system or split from that source (for example validation versus test, since Section 3.2 says results are reported on the validation set, or single model versus ensemble); only the authors can say which. Figure values are read off the plot. The grammar slip "performance are" is confirmed on the same crop and is S4.
Second check: another model agreed
Report a problem with this finding
Typo "This suggests that multitask training many need just as many effective training pairs": "many need" should be "may need".
- p. 1 Introduction, right column, paragraph 4
multitask training many need just as many effective training
Judge's note: Read on the crop and the page 1 image; the wording is as quoted and the sentence has no verb unless "many" is read as "may".
Second check: none (judge only)
Report a problem with this finding
Typo The same summarization baseline is labelled "Lead-3" in Figure 1 and "Lede-3" in Table 4.
- p. 2 Figure 1, Summarization panel, top reference line
Lead-3
- p. 6 Table 4, second row
Lede-3
Judge's note: Figure 1 Summarization panel (page 2 image) labels the top reference line "Lead-3" at about 31.5; Table 4 (page 6 image) has the row "Lede-3" with R-AVG 31.55. Same baseline, two spellings; both spellings are in use, so cosmetic.
Second check: none (judge only)
Report a problem with this finding
Typo Duplicated word: "Since the supervised objective is the the same as the unsupervised objective".
- p. 3 Section 2, left column, paragraph 2
Since the supervised objective is the the same as the
- p. 3 Section 2
the supervised objective is the the same as the
- p. 3 Sec. 2, paragraph 3
is the the same as the
- p. 3
e the supervised objective is the the same as the unsuper
Judge's note: Read on the crop: both words "the the" are on one line, so this is not a line-break artefact.
Second check: code check
Report a problem with this finding
Typo "how WebText LM's perform" (apostrophe in a plural; "LMs" elsewhere) and "a similar phenomena could be happening" (should be "phenomenon").
- p. 4 Section 3.1, first sentence
interested in understanding how WebText LM’s perform
- p. 8 Section 4, first paragraph
issue becomes increasingly likely which suggests a similar
- p. 4 Section 3.1
how WebText LM’s perform
Judge's note: Both read on the images: Section 3.1 first sentence on page 4 (the same crop shows "word-level LMs" and "our LMs" without an apostrophe) and Section 4 first paragraph on page 8.
Second check: none (judge only)
Report a problem with this finding
Typo Table 3, 1BW column, 762M row: "44.575" is printed with three decimals; every other perplexity in the table has at most two (75.20, 55.72, 42.16 in the same column).
- p. 5 Table 3, 1BW column, row 762M
44.575
- p. 5 Table 3, 1BW column neighbours
75.20
Judge's note: Read on the page 5 image. The value sits plausibly between 55.72 and 42.16, so it looks like an unrounded value rather than a wrong one; formatting inconsistency only.
Second check: none (judge only)
Report a problem with this finding
Typo Dataset names are spelled inconsistently: Table 3 header "WikiText2" and "WikiText103", Table 3 caption "WikiText-2", Table 6 header "WikiText-2" and "Wikitext-103", Section 4 text "WikiText-103".
- p. 5 Table 3 header
WikiText103
- p. 8 Table 6 header
Wikitext-103
- p. 8 Section 4, paragraph 4
test set of WikiText-103
- p. 5 Table 3 header
WikiText2
Judge's note: All forms read on the page 5 and page 8 images. Cosmetic; the datasets are unambiguous.
Second check: none (judge only)
Report a problem with this finding
Typo Subject-verb disagreement: "the generated summaries only begin to approach the performance of classic neural baselines and just barely outperforms selecting 3 random sentences" (should be "outperform").
- p. 6 Section 3.6 Summarization
the generated summaries only begin to approach the performance of classic neural baselines and just barely outperforms selecting 3 random sentences
Judge's note: Read on the page 6 image, Section 3.6; plural subject "summaries" takes "begin" and then "outperforms".
Second check: none (judge only)
Report a problem with this finding
Typo Stray comma before "worse": "The performance of GPT-2 is still much, much, worse than the 30 to 50% range".
- p. 7 Section 3.8
is still much, much, worse than the 30 to 50%
Judge's note: Read on the page 7 image, right column, section 3.8. The second comma separates the adverb from the adjective it modifies; "much, much worse" is meant.
Second check: none (judge only)
Report a problem with this finding
Typo Superfluous word: "with an average of overlap of 3.2%" (should read "an average overlap of 3.2%").
- p. 8 Section 4, paragraph 3
with an average of overlap of 3.2%
Judge's note: Read on the page 8 image, left column, third paragraph of Section 4. The next sentence has the correct construction, "with an average of 5.9% overlap". Checked the number too: the WebText train row of Table 6 averages to 3.155%, which rounds to 3.2%, so only the wording is at fault.
Second check: none (judge only)
Report a problem with this finding
Typo Hyphenation differs: Table 6 caption has "test set 8 grams overlapping with training sets" while the text below has "Bloom filters containing 8-grams".
- p. 8 Table 6 caption
test set 8 grams overlapping
- p. 8 Section 4 text
Bloom filters containing 8-grams
Judge's note: Both forms read on the page 8 image. The body text uses "8-grams" and "8-gram" throughout (also in the Figure 5 axis label and caption on page 13); only the Table 6 caption drops the hyphen. House-style inconsistency.
Second check: none (judge only)
Report a problem with this finding
Typo "without the need for supervised adaption or modification": "adaption" where "adaptation" is the usual form.
- p. 9 6. Discussion
supervised adaption or modification
Judge's note: Read on the page 9 image, right column, first paragraph of Section 6. "Adaption" is a recognised though uncommon variant of "adaptation", so either form could be legitimate; noted as a consistency point only.
Second check: none (judge only)
Report a problem with this finding
Typo Stray period in the Jelinek & Mercer reference: "North-Holland, May., 1980."
- p. 11 Reference list, Jelinek & Mercer
North-Holland, May., 1980.
Judge's note: Read on the page 11 image, left column. "May" is not an abbreviation, so the period before the comma is spurious punctuation (likely carried over from the BibTeX month field).
Second check: none (judge only)
Report a problem with this finding
Typo "To complement the reported perplexity gains of bigger LMs on WebText show in Figure 4" should read "shown in Figure 4".
- p. 13 Appendix 8.1
WebText show in Figure 4
- p. 13 8.2 Text Memorization
less often then the baseline rate
- p. 13 8.1 Model capacity
bigger LMs on WebText show in Figure 4
Judge's note: Read on the page 13 image, section 8.1. Missing participle ending. Figure 4 (page 9) does show perplexity against model size, so the pointer itself is right. The "less often then" wording that one reviewer note also mentions is handled in C024.
Second check: none (judge only)
Report a problem with this finding
Typo "GPT-2 repeats text from the training set less often then the baseline rate of held-out articles": "then" should be "than".
- p. 13 Appendix 8.2, second paragraph
from the training set less often then the baseline rate of held-out
- p. 13 Appendix 8.2
repeats text from the training set less often then the baseline rate
- p. 13 Sec. 8.2
less often then the baseline rate
- p. 13 8.2 Text Memorization
less often then the baseline rate of held-out
Judge's note: Read on the page 13 image, section 8.2, second paragraph. A comparative requires "than".
Second check: none (judge only)