RoBERTa: A Robustly Optimized BERT Pretraining Approach
2019 · paper · AI and machine learning
A replication study of BERT pretraining that measures the effect of hyperparameters and training data size. It finds BERT was significantly undertrained; the improved recipe, RoBERTa, matches or exceeds later models and reaches state-of-the-art results on GLUE, RACE and SQuAD.
Checked: PDF from arxiv.org, fetched 4 Oct 2026. Page numbers are those of that PDF.
2 findings
Substance BERT-LARGE SST-2 dev accuracy printed as 93.7 in Table 4 but 93.2 in Table 5; both are attributed to Devlin et al. (2019) and both give MNLI 86.6.
- p. 7 Table 4, BERTLARGE with BOOKS + WIKI row, SST-2 column
90.9/81.8 86.6 93.7
- p. 8 Table 5, Single-task single models on dev, BERTLARGE row, SST column
93.2
Only the authors can say what is right here.
Judge's note: Page images p7 and p8 show Table 4 BERT-LARGE 'with BOOKS + WIKI' SST-2 = 93.7 and Table 5 'Single-task single models on dev' BERT-LARGE SST = 93.2. Both captions credit Devlin et al. (2019), and the MNLI value matches (86.6). Nothing in the paper explains the difference; it may come from two different tables in the source. Only the authors can say which value is meant.
Second check: another model agreed
Report a problem with this finding
Reference or label Section 5.1 says 'We explore a slightly wider hyperparameter space, described in the Appendix' for the ensembles/test setting, but the Appendix (Section C, Table 10) gives only the same GLUE grid as the dev setting.
- p. 7 Section 5.1, second setting
We explore a slightly wider hyperparameter space, described in the Appendix
- p. 13 Table 10, GLUE column
{1e-5, 2e-5, 3e-5}
Only the authors can say what is right here.
Judge's note: p12 Appendix sections A to C and p13 Table 10 show GLUE settings: LR {1e-5, 2e-5, 3e-5}, batch size {16, 32}, 10 epochs, warmup 0.06, linear decay. This is identical to the 'limited' dev sweep described in Section 5.1 on p7. No wider space for test submissions is described anywhere in the Appendix.
Second check: another model agreed
Report a problem with this finding
17 minor (typos and the like; most checked by the judge only)
Wording Conclusion states "We find that performance can be substantially improved by ... training on longer sequences", but no experiment isolates sequence length (Sec. 3.1: "We train only with full-length sequences").
- p. 1 Introduction, list of modifications
(3) training on longer se-quences
- p. 10 Section 7 Conclusion
training on longer se-quences
- p. 3 Section 3.1
We train only with full-length sequences.
Only the authors can say what is right here.
Judge's note: Section 3.1 (p.3) describes full-length training as a departure from BERT's reduced-length schedule, but Tables 1–4 do not vary it. The only related evidence is Table 2's SENTENCE-PAIR row, which mixes shorter inputs with a different input format and batch size. Within the Intro, the item reads as a list of modifications; in the Conclusion (p.10) it is worded as a finding. Query to the authors about which result supports it.
Second check: another model agreed
Report a problem with this finding
Wording Conclusion: 'performance can be substantially improved by ... dynamically changing the masking pattern', while Section 4.1 says 'dynamic masking is comparable or slightly better than static masking'.
- p. 10 Section 7 Conclusion
performance can be substantially im-proved by training the model longer
- p. 4 Section 4.1 Results / Table 1
dynamic masking is comparable or slightly better than static masking
Judge's note: p10 conclusion lists dynamic masking and NSP removal under 'substantially improved'. The p4 Table 1 differences (static vs dynamic) are +0.4 on SQuAD 2.0, −0.3 on MNLI-m and +0.4 on SST-2, and Section 4.1 itself calls them comparable or slightly better. 'Substantially' may be meant for the combined recipe, but as written it overstates this item.
Second check: another model agreed
Report a problem with this finding
Wording Table 2 caption says "All models are trained for 1M steps with a batch size of 256 sequences", but the text says SENTENCE-PAIR+NSP and DOC-SENTENCES increased the batch size to keep the token count similar.
- p. 5 Table 2 caption
All models are trained for 1M steps with a batch size of 256 sequences.
- p. 5 SENTENCE-PAIR+NSP and DOC-SENTENCES bullets
we increase the batch size so that the total number of tokens remains similar
Judge's note: P.5 image: the caption and the SENTENCE-PAIR and DOC-SENTENCES bullets disagree on the batch size. The caption is imprecise for two of its rows.
Second check: another model agreed
Report a problem with this finding
Wording Table 3's 256-batch baseline ("256 1M 1e-4 3.99 84.7 92.7") equals the DOC-SENTENCES row of Table 2 (84.7/92.7), not FULL-SENTENCES (84.7/92.5), which the text says is "use[d] ... in the remainder of our experiments".
- p. 6 Table 3, 256 row
256 1M 1e-4 3.99 84.7 92.7
- p. 5 Section 4.2 / Table 2 FULL-SENTENCES row
we use FULL-SENTENCES in the remainder of our experiments
Only the authors can say what is right here.
Judge's note: Checked on the p.5 and p.6 images. The paper does not state the input format for Table 3. The match could be a separate run, but on p.4 the Table 1 'dynamic' row (78.7/84.0/92.9) also equals Table 2's SEGMENT-PAIR row, which suggests rows are reused. The authors should say which configuration produced the Table 3 baseline.
Second check: another model agreed
Report a problem with this finding
Wording Section 5.1 says 'The rest of the hyper-parameters remain the same as during pretraining', but Table 10 gives GLUE weight decay 0.1 while Table 9 (and Section 2.4) give pretraining weight decay 0.01.
- p. 7 Section 5.1
The rest of the hyper-parameters remain the same as during pretraining.
- p. 13 Table 10 (GLUE weight decay) vs Table 9 (weight decay)
Weight Decay 0.1 0.01 0.1
- p. 13 Table 10, Weight Decay row, GLUE column
0.1
- p. 13 Table 9, Weight Decay row
0.01
Only the authors can say what is right here.
Judge's note: Verified on p13: in Table 10, Weight Decay is 0.1 (RACE), 0.01 (SQuAD) and 0.1 (GLUE); in Table 9, Weight Decay is 0.01 for both models. Section 5.1 on p7 names only batch size, learning rate, warmup/decay and epochs as GLUE-specific. The text and the table disagree, and the authors need to say which is right.
Second check: another model agreed
Report a problem with this finding
Wording 'We observe further improvements in performance across all downstream tasks' (adding data at 100K steps) holds for the Table 4 tasks, but appendix Table 8 shows RTE 84.5→82.7 and CoLA 66.3→66.1 for the same models.
- p. 7 Section 5 Results
We ob-serve further improvements in performance across all downstream tasks
- p. 13 Table 8, with BOOKS + WIKI vs + additional data
84.5
Judge's note: p7: the sentence refers to Table 4, where all three tasks improve. p13 Table 8 rows 'with BOOKS + WIKI' and '+ additional data (§3.2)' show RTE 84.5 vs 82.7 and CoLA 66.3 vs 66.1. Table 4's caption points readers to these complete GLUE results. 'All downstream tasks' therefore overstates what the data show; it is a minor wording point.
Second check: another model agreed
Report a problem with this finding
Typo Introduction (p.1, right column) reads "the effects of hyperparmeter tuning"; should be "hyperparameter".
- p. 1 Introduction, para 2
hyperparmeter
- p. 1
effects of hyperparmeter tuning and training set size
Judge's note: Page 1 image shows 'hyperparmeter' in the first Introduction paragraph of the right column, not in the abstract. The abstract spells it correctly.
Second check: none (judge only)
Report a problem with this finding
Typo Footnote 6 expands MRPC as "Microsoft Research Paragraph Corpus"; the cited Dolan and Brockett (2005) work is a corpus of sentential paraphrases (Paraphrase Corpus).
- p. 3 Footnote 6
Microsoft Research Paragraph Corpus (MRPC)
Only the authors can say what is right here.
Judge's note: Rests on outside knowledge: MRPC stands for Microsoft Research Paraphrase Corpus. Footnote 6 on the p.3 image reads 'Microsoft Research Paragraph Corpus (MRPC)'. The reference entry is 'Automatically constructing a corpus of sentential paraphrases', which supports 'Paraphrase'.
Second check: none (judge only)
Report a problem with this finding
Typo Section 3.3: "These results depend on a several task-specific modifications"; the stray 'a' should be deleted.
- p. 3 Section 3.3 GLUE
depend on a several task-specific modifica-tions
- p. 3 Section 3.3 GLUE paragraph
results depend on a several task-specific modifi-cations
- p. 3 Section 3.3 GLUE
depend on a several task-specific modifica-
Judge's note: Visible on the p.3 image, GLUE paragraph, right column.
Second check: none (judge only)
Report a problem with this finding
Typo Section 4.4: "BPE relies on subwords units"; should be "subword units".
- p. 6 Section 4.4
BPE relies on subwords units
Judge's note: Visible on the p.6 image, Section 4.4, first paragraph.
Second check: none (judge only)
Report a problem with this finding
Typo Section 4.3: "train BERT with even larger batche sizes"; should be "batch sizes".
- p. 6 Section 4.3
larger batche sizes
- p. 6 Section 4.3
even larger batche sizes
- p. 6
larger batche sizes, up to 32K sequences
Judge's note: Visible on the p.6 image, left column, final paragraph of 4.3.
Second check: none (judge only)
Report a problem with this finding
Typo Section 4.4: "outweighs the minor degre-dation"; 'degredation' should be 'degradation'.
- p. 6 Section 4.4
minor degre-dation
Judge's note: P.6 image, right column: the word is broken across lines as 'degre-' / 'dation', so the spelling is 'degredation'.
Second check: none (judge only)
Report a problem with this finding
Typo Section 4.4: "the advantages of a universal encoding scheme outweighs"; should be "outweigh".
- p. 6 Section 4.4
the advantages of a univer-sal encoding scheme outweighs
Judge's note: P.6 image, right column: plural subject with a singular verb.
Second check: none (judge only)
Report a problem with this finding
Typo Benchmark name misspelled 'GLUE, SQuaD and RACE' (should be SQuAD), Section 5, p7.
- p. 7 Section 5, last paragraph of left column
GLUE, SQuaD and RACE
Judge's note: The p7 page image shows 'SQuaD' at the bottom of the left column. Everywhere else the paper writes SQuAD.
Second check: none (judge only)
Report a problem with this finding
Typo Garbled accented author names in references: 'Llu'is M'arquez' (Agirre et al. 2007) and 'Kornl Csernai' (Iyer et al. 2016).
- p. 10 References, Iyer et al. 2016
Kornl Cser-nai
- p. 10 References, Agirre et al. 2007
Llu’is M‘arquez
Only the authors can say what is right here.
Judge's note: Rests on outside knowledge: the correct names are Lluís Màrquez and Kornél Csernai. The p10 page image shows the stray apostrophe-like marks in 'Llu'is M'arquez' and shows 'Kornl', so these are not extraction artefacts.
Second check: none (judge only)
Report a problem with this finding
Typo Reference Dagan et al. (2006): 'recognising tectual entailment' should be 'textual'.
- p. 10 References, Dagan et al. 2006
recognising tectual entailment
Judge's note: The p10 page image shows 'recognising tectual entailment' in the Dagan et al. 2006 entry.
Second check: none (judge only)
Report a problem with this finding
Typo Reference Radford et al. (2018) lists co-author as 'Time Salimans' (should be Tim Salimans).
- p. 11 References, Radford et al. 2018
Time Salimans
- p. 10 References, Iyer et al. 2016
Kornl Cser-
Only the authors can say what is right here.
Judge's note: Rests on outside knowledge: the co-author's name is Tim Salimans. The crop of p11 shows 'Time Salimans'. The 'Kornl' part of this cluster duplicates C024 and is not counted again here.
Second check: none (judge only)