Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
2019 · paper · AI and machine learning
Studies transfer learning for language tasks by recasting every problem as text-to-text, comparing pre-training objectives, architectures, data and other factors. Combining these findings with scale and a new web-text corpus, it reports state-of-the-art results on many benchmarks.
Checked: PDF from arxiv.org, fetched 4 Oct 2026. Page numbers are those of that PDF.
6 findings
Substance Table 10 row "Adapter layers, d = 2048" (81.51 16.62 79.47 63.03 19.83 27.50 22.63) duplicates the d = 128 row; Table 16 gives d = 2048 as 82.62, 18.30, 79.40, 68.61, 25.64, 36.92, 26.93.
- p. 30 Table 10, rows d = 128 and d = 2048
Adapter layers, d = 2048
- p. 57 Table 16, row 10 'Adapter layers, d = 2048'
82.62
- p. 30 Table 10
Adapter layers, d = 2048 81.51 16.62 79.47 63.03 19.83 27.50 22.63
- p. 57 Table 16, Table 10 rows, 'Adapter layers, d = 2048'
Adapter layers, d = 2048 82.62 ... 18.30 ... 79.40 ... 68.61 ... 25.64 36.92 26.93
Judge's note: The p30 image shows the d=128 and d=2048 rows identical in all seven columns. On the p57 image, Table 16 (Table 10 block) gives distinct d=2048 values. The d=128 row in Table 16 matches Table 10 (81.51, 16.62, 79.47, 63.03, 19.83, 27.50, 22.63). The Table 16 d=2048 values fit the text's statement that higher-resource tasks need large d.
Second check: another model agreed
Report a problem with this finding
Substance EnRo BLEU for "4× ensembled" is 28.57 (bold) in Table 13 but 28.09 in Table 16; the other six columns of the row agree.
- p. 35 Table 13, row '4× ensembled', EnRo
28.57
- p. 57 Table 16, row 13 '4× ensembled', EnRo BLEU
28.09
- p. 57 Table 16, row '4× ensembled', EnRo BLEU (last column; the 4× ensembled, fine-tune only row ends with the same value)
28.05 40.53 28.09
- p. 35 Table 13, row '4× ensembled', EnRo
4× ensembled 84.77 20.10 83.09 71.74 28.05 40.53 28.57
Only the authors can say what is right here.
Judge's note: The p35 image shows Table 13 "4× ensembled ... 28.05 40.53 28.57". The p57 image shows Table 16 "28.05 40.53 28.09", which equals the fine-tune-only row's EnRo. The text on p35 says ensembling significantly outperformed every other approach on WMT EnRo. That holds with 28.57 but not with 28.09, since "2× size, 2× training steps" scores 28.19. Only the authors can say which value is correct.
Second check: another model agreed
Report a problem with this finding
Reference or label "For reference, these details are described in Section 2." The AdaFactor optimizer, inverse square root schedule, fine-tuning learning rate and vocabulary are described in Section 3.1 (3.1.2–3.1.3), not Section 2.
- p. 38 Section 3.7, paragraph after 'Test set'
For reference, these details are described in Section 2.
Judge's note: Text search shows AdaFactor and the inverse square root schedule only in 3.1.2 (Training), and SentencePiece/vocabulary in 3.1.3. Section 2 covers model, C4, tasks and format. Checked on the p38 image.
Second check: another model agreed
Report a problem with this finding
Wording Table 3 deshuffling input 'party me for your to . last fun you inviting week Thank' contains 'fun', which is not in the source sentence 'Thank you for inviting me to your party last week .'
- p. 21 Table 3, Deshuffling row, Inputs
party me for your to . last fun you inviting week Thank
Judge's note: The p.21 crop shows 12 tokens including 'fun'; the source has 11. Section 3.3.1 describes deshuffling as pure shuffling with the original text as target. Unlike the grey 'apple' in the BERT-style row (an intentional random replacement), 'fun' is not marked.
Second check: another model agreed
Report a problem with this finding
Wording "our largest (11 billion parameter) model performed best among our model size variants across all tasks" is contradicted by Table 14 WMT EnRo, where T5-3B scores 28.2 and T5-11B 28.1.
- p. 38 Section 3.7 text
model performed best among our model size variants across all tasks
- p. 39 Table 14, WMT EnRo BLEU column
28.2
- p. 38 Section 3.7, results paragraph
our largest (11 billion parameter) model performed best among our model size variants across all tasks
- p. 39 Table 14, WMT EnRo BLEU column
T5-3B 28.2 / T5-11B 28.1
Judge's note: The p39 image shows the EnRo column: Large 28.1, 3B 28.2, 11B 28.1. I checked the other columns and found 11B best or tied everywhere else. The claim of 'all tasks' fails on EnRo only.
Second check: another model agreed
Report a problem with this finding
Wording Appendix B labels "The city councilmen refused the demonstrators a permit because they feared violence." as the hypothesis and "The demonstrators feared violence." as the premise, but the next steps find "they" in the premise and remove "feared violence" from the hypothesis, so the labels are swapped.
- p. 45 Appendix B
the hypothesis would be “The city councilmen refused the demonstrators a permit because they feared violence.” with the premise “The demonstrators feared violence.”
Judge's note: The p45 image confirms the wording. The sentence labelled premise contains no pronoun, while the procedure says the pronoun 'they' is located in the premise. The two cannot both hold.
Second check: another model agreed
Report a problem with this finding
20 minor (typos and the like; most checked by the judge only)
Wording The schedule 1/sqrt(max(n,k)) is described as one that 'then exponentially decays the learning rate'; for n>k it decays as n^(-1/2), a power law.
- p. 12 Section 3.1.2
then exponentially decays the learning rate until pre-training is over
Judge's note: The page 12 crop shows the formula 1/√max(n,k) with k=10^4, followed by 'exponentially decays'. For n>k the rate is n^(-1/2), which is not of the form a·b^n. The wording misdescribes the formula printed beside it.
Second check: another model agreed
Report a problem with this finding
Wording Table 8: the Wikipedia + TBC GLUE score '83.65' is not boldfaced, although it is within two standard deviations of the maximum 84.03 under the stated boldface rule.
- p. 27 Table 8, GLUE column, Wikipedia + TBC row
83.65
- p. 14 Section 3.1.5 boldface rule
boldface any score that is within two standard deviations of the maximum
Judge's note: The bold rule is on p.14. GLUE standard deviation is 0.235 (Table 16, p.57). The threshold is 84.03-2*0.235=83.56. RealNews-like 83.83 is bold; Wikipedia + TBC 83.65 is in normal weight on the p.27 image.
Second check: another model agreed
Report a problem with this finding
Wording Text says 'using a denoising objective always results in better downstream task performance compared to a language modeling objective', but Table 2 has same-architecture cells where LM scores higher.
- p. 20 Section 3.2.4 text
using a denoising objective always results in better downstream task performance compared to a language modeling objective
- p. 20 Table 2, Language model rows
56.51
- p. 20 Section 3.2.4, text below Table 2
objective always results in better downstream task performance compared to a language modeling objective
- p. 20 Table 2, EnFr column
Enc-dec, shared Denoising ... 39.03 ... Enc-dec, shared LM ... 39.17
Judge's note: The p.20 image (Table 2) shows: Language model SGLUE 55.02 (denoising) vs 56.51 (LM); Language model EnDe 25.09 vs 25.23; Enc-dec shared EnFr 39.03 vs 39.17; Enc-dec 6 layers EnFr 38.40 vs 38.42. 'Always' overstates the table.
Second check: another model agreed
Report a problem with this finding
Wording Text names the variants '“replace corrupted spans” and “drop corrupted spans”', but Table 5 labels the second 'Drop corrupted tokens'.
- p. 23 Section 3.3.2 text
(“replace corrupted spans” and “drop corrupted spans”)
- p. 22 Table 5 row label
Drop corrupted tokens
Judge's note: The page 23 crop shows 'drop corrupted spans'. The page 22 Table 5 crop shows 'Drop corrupted tokens', and the surrounding text refers to dropping tokens. Naming inconsistency.
Second check: another model agreed
Report a problem with this finding
Wording Text says removing filtering 'makes the unfiltered variant perform the worst in every task', but in Table 8 SGLUE Wikipedia (68.01) is below C4, unfiltered (68.04).
- p. 26 Section 3.4.1 text
makes the unfiltered variant perform the worst in every task
- p. 27 Table 8, SGLUE column
68.01
- p. 27 Table 8, SGLUE column
C4, unfiltered 6.1TB 81.46 19.14 78.78 68.04
Judge's note: In Table 8 (p.27 image) C4, unfiltered is lowest in every column except SGLUE, where Wikipedia scores 68.01. Table 16 shows the same values. The gap is within noise, but the sentence is categorical.
Second check: another model agreed
Report a problem with this finding
Wording The SuperGLUE gain from Wikipedia + TBC (71.36 to 73.24) is called 'almost entirely attributable' to MultiRC EM (25.78 to 50.93), but in Table 16 MultiRC accounts for about a third of the summed task gain.
- p. 27 Section 3.4.1 text
This is almost entirely attributable to a boost in performance from 25.78 (baseline, C4) to 50.93
- p. 57 Table 16, rows 8 C4 and Wikipedia + TBC
50.93
Only the authors can say what is right here.
Judge's note: I recomputed both SuperGLUE averages from Table 16 (p.57) and they reproduce 71.36 and 73.24. The MultiRC task score (mean of F1a and EM) rises 5.31, because F1a falls 66.13 to 51.59. The summed change over the 8 tasks is about 15.0; CB, COPA, RTE and WSC also rise. The wording overstates.
Second check: another model agreed
Report a problem with this finding
Wording Sec. 3.4.1 says "using data from Wikipedia produced significant (but less dramatic) gains on SQuAD", but Table 8 shows 81.29 vs the C4 baseline 80.88, a gain of 0.41 against a baseline run-to-run SD of 0.343 (Table 1).
- p. 27 Section 3.4.1 text
using data from Wikipedia produced significant (but less dramatic) gains on SQuAD
- p. 27 Table 8, SQuAD column
81.29
Judge's note: Checked Table 8 on p27: Wikipedia SQuAD is 81.29 (not bold) and C4 is 80.88. Table 1 on p14 gives SQuAD SD = 0.343. The gain is about 1.2 SD. The paper has no formal significance test. Its 2-SD rule only decides boldface relative to the best score in each table, so this is a wording point rather than a breach of a stated criterion.
Second check: another model agreed
Report a problem with this finding
Wording "train from layers 12 − n to 12 in the nth episode": with 12 episodes, n = 1 gives two layers (11–12), not only the final layer as the prose describes, and n = 12 gives a layer 0; the indexing works only if n runs from 0 to 11, which is not stated.
- p. 30 Section 3.5.1
train from layers 12 − n to 12 in the nth episode
Judge's note: On p30 the prose says fine-tuning starts with only the final layer and the process is split into 12 episodes. Taking n = 1..12 against layers 1..12 gives an off-by-one. Taking n = 0..11 is consistent. The text does not say which convention applies, so this is an ambiguity.
Second check: another model agreed
Report a problem with this finding
Typo Textual citation used where a parenthetical one is needed: 'or span extraction Keskar et al. (2019b) tasks'; the parallel citations in the same sentence are parenthetical.
- p. 2 Introduction, paragraph 4
or span extraction Keskar et al. (2019b) tasks
Judge's note: The page 2 crop shows '(McCann et al., 2018)', '(Radford et al., 2019)', then 'Keskar et al. (2019b)' in text form inside the noun phrase. This is a citation-command slip.
Second check: none (judge only)
Report a problem with this finding
Typo Duplicated word: 'TPU pods are are multi-rack ML supercomputers' (p.5).
- p. 5 Section 2.1, last paragraph
TPU pods are are multi-rack ML supercomputers
- p. 5
of Cloud TPU Pods.5 TPU pods are are multi-rack ML super
Judge's note: The page 5 crop shows 'are are' on a single line, so this is not a line-break artefact.
Second check: code check
Report a problem with this finding
Typo Author surname printed 'De Marneff' in the citation 'CB (De Marneff et al., 2019)' and in the reference entry 'Marie-Catherine De Marneff'.
- p. 7 Section 2.3 task list
CB (De Marneff et al., 2019)
- p. 59 Reference list
Marie-Catherine De Marneff
- p. 59 References
Marie-Catherine De Marneff, Mandy Simons, and Judith Tonhauser
Only the authors can say what is right here.
Judge's note: Rests on outside knowledge: the CommitmentBank author's surname is spelled 'de Marneffe'. The page images on p.7 and p.59 show 'De Marneff' in both places, so the spelling is internally consistent.
Second check: none (judge only)
Report a problem with this finding
Typo Duplicated word: 'The first model structure we consider is an an encoder-decoder Transformer' (p.15).
- p. 15 Section 3.2.1
The first model structure we consider is an an encoder-decoder Transformer
- p. 15 Section 3.2.1, first paragraph
is an an encoder-decoder Transformer
- p. 15
odel structure we consider is an an encoder-decoder Tra
Judge's note: The page 15 crop shows 'an an' on a single line.
Second check: code check
Report a problem with this finding
Typo Subject-verb disagreement: 'the large disparity between our baseline and “no pre-training” emphasize how much' (should be 'emphasizes').
- p. 15 Section 3.1.5, first paragraph
the large disparity between our baseline and “no pre-training” emphasize how much
Judge's note: Confirmed on the page 15 crop.
Second check: none (judge only)
Report a problem with this finding
Typo Article mismatch: 'same number of parameters as an 2L-layer language model' ('2L' is read 'two-L', so 'a' is expected).
- p. 19 Section 3.2.2, last sentence of first paragraph
same number of parameters as an 2L-layer language model
Judge's note: The page 19 crop shows 'an 2L-layer'. 'an L+L-layer' is correct elsewhere on the page, which suggests a carry-over. Cosmetic.
Second check: none (judge only)
Report a problem with this finding
Typo Number disagreement: "at what point using a smaller pre-training data sets poses an issue" (should be "data set").
- p. 27 end of Section 3.4.1
using a smaller pre-training data sets poses an issue
Judge's note: The page 27 image shows the text exactly as quoted.
Second check: none (judge only)
Report a problem with this finding
Typo SGLUE for "4× ensembled, fine-tune only" is 71.55 in Table 13 but 71.56 in Table 16.
- p. 35 Table 13, row '4× ensembled, fine-tune only', SGLUE
71.55
- p. 57 Table 16, row 13 '4× ensembled, fine-tune only', SuperGLUE Score
71.56
Judge's note: Both images show the values as quoted. The SuperGLUE average recomputed from the Table 16 subtask scores is 71.556. Because the subtask scores are themselves rounded, 71.55 cannot be ruled out entirely, but the two tables print different values for the same quantity.
Second check: none (judge only)
Report a problem with this finding
Typo Author name printed "Konečnỳ" (grave accent on y) in the p43 citation and in both reference entries on p62.
- p. 43 Section 4.2, The inconvenience of large models
Konečnỳ et al., 2015, 2016
- p. 62 Reference list, Konečný entries
Jakub Konečn`y
- p. 43 4.2 Outlook, first item
Konečn`y et al., 2015, 2016
- p. 62 References
Jakub Konečn`y, Brendan McMahan
Only the authors can say what is right here.
Judge's note: Rests on outside knowledge: the author's surname is spelled Konečný with an acute accent. The images of p43 and p62 show a grave accent over the y.
Second check: none (judge only)
Report a problem with this finding
Typo Missing verb: "These examples selected at random and were not cherry-picked." (should be "were selected").
- p. 45 Appendix C
These examples selected at random and were not cherry-picked.
Judge's note: The p45 image shows the sentence as quoted.
Second check: none (judge only)
Report a problem with this finding
Typo Reference list prints "Marie-Catherine De Marneff" (CommitmentBank entry).
- p. 59
Marie-Catherine De Marneff, Mandy Simons, and Judith Tonhauser.
Only the authors can say what is right here.
Judge's note: Rests on outside knowledge: the author's surname is spelled de Marneffe. The p59 image shows 'De Marneff'.
Second check: none (judge only)
Report a problem with this finding
Typo The GLUE reference (Wang et al., 2018) prints the co-author as "Amapreet Singh", but the SuperGLUE reference (Wang et al., 2019b) on the same page spells the name "Amanpreet Singh".
- p. 66 Reference list, Wang et al. 2018 (GLUE)
Alex Wang, Amapreet Singh, Julian Michael
- p. 66 Reference list, Wang et al. 2019b (SuperGLUE)
Amanpreet Singh
- p. 66
Alex Wang, Amapreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman.
Judge's note: I checked both crops from PDF page 66. The GLUE entry reads "Alex Wang, Amapreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman." The SuperGLUE entry reads "... Nikita Nangia, Amanpreet Singh, Julian Michael ...". The same author is spelled two ways within this reference list, so the GLUE entry is a misspelling with an evident fix: "Amanpreet". This is a cosmetic typo.
Second check: none (judge only)