Unsupervised Cross-lingual Representation Learning at Scale
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020 · paper · Computational linguistics
Trains XLM-R, a Transformer masked language model on over two terabytes of text in one hundred languages. It reportedly beats multilingual BERT on cross-lingual benchmarks, notably for low-resource languages, and the paper analyzes the trade-offs of multilingual scale.
Checked: PDF from aclanthology.org, fetched 5 Oct 2026. Page numbers are those of that PDF.
6 findings
Substance Table 6, Swedish row prints Tokens (M) "77.8" and Size (GiB) "12.1": the only rows with a non-integer token count are this and Ukrainian, and 12.1 GiB conflicts with Figure 1, where sv sits between uk (84.6 GiB) and th (71.7 GiB).
- p. 11 Table 6, row sv Swedish (Tokens (M), Size (GiB))
77.8 12.1
- p. 3 Figure 1, x-axis order of languages by CommonCrawl size
CommonCrawl
- p. 11 Table 6, row sv Swedish
77.8
- p. 3 Figure 1, x-axis ordering (sv placed between uk and th)
sv
Only the authors can say what is right here.
Judge's note: The page 11 image shows "Swedish 77.8 12.1". Figure 1 is sorted by CommonCrawl size; its x-axis order in the text layer and crop is en, ru, id, vi, fa, uk, sv, th, ja. Table 6 sizes for the neighbours are uk 84.6 and th 71.7, so 77.8 fits as the size and 12.1 does not. The printed ratio is about 6.4 M tokens per GiB against about 172 for Danish and 173 for Norwegian. The cells look transposed with the token count in billions, but the exact intended token count is for the authors. The Ukrainian row is judged in C015.
Second check: another model agreed
Report a problem with this finding
Substance Table 1, cross-lingual transfer, Lample and Conneau (2019) Wiki 100-language rows: the #M = N row prints Avg "71.3" but its 15 entries average 70.7; the #M = 1 row prints Avg "70.7" but its entries average 71.3. The text relies on the printed averages: "We observe a 0.6 decrease in overall accuracy when we go from N models to a single model - going from 71.3 to 70.7".
- p. 7 Table 1, cross-lingual transfer block, Lample and Conneau (2019) Wiki, #M = N, #lg = 100
83.7 76.2 76.6 73.7 72.4 73.0 72.1 68.1 68.4 72.0 68.2 71.5 64.5 58.0 62.4 71.3
- p. 7 Table 1, cross-lingual transfer block, Lample and Conneau (2019) Wiki, #M = 1, #lg = 100
83.2 76.7 77.7 74.0 72.7 74.1 72.7 68.7 68.6 72.9 68.9 72.5 65.6 58.2 62.4 70.7
- p. 6 Section 5.2, XNLI paragraph
We observe a 0.6 decrease in overall accuracy when we go from N models to a single model - going from 71.3 to 70.7
- p. 6 Section 5.2, XNLI paragraph
We observe a 0.6 decrease in overall accuracy when we go from N models to a single model - going from 71.3 to 70.7.
Only the authors can say what is right here.
Judge's note: Read both rows on the page 7 image and checked them against the layout text; the digits agree. N row sum is 1060.8, mean 70.72; single-model row sum is 1068.9, mean 71.26. Each row's mean equals the other row's printed Avg, so either the Avg cells or the per-language cells are swapped. As printed, the single-model row is at least as high as the N row in 14 of 15 languages, which contradicts the stated decrease. No explanation appears in the caption or text. Which half is swapped is for the authors to say.
Second check: another model agreed
Report a problem with this finding
Substance Section 5.2 says XLM-R outperforms mBERT by "1.86 F1 when trained on each language", but Table 2 gives Avg 90.24 (XLM-R, each) and 88.28 (mBERT, each), a difference of 1.96.
- p. 7 Section 5.2, Named Entity Recognition paragraph (left column, below Table 2)
and 1.86 F1 when trained on each language
- p. 7 Table 2, 'each' rows for mBERT and XLM-R, Avg column
88.28
- p. 7 Table 2, XLM-R 'each' row and mBERT 'each' row, Avg column
90.24
- p. 7 Named Entity Recognition paragraph
1.86 F1 when trained
Judge's note: Read on the page 7 image. Both table averages reproduce from their per-language entries (90.245 and 88.28). The companion figure in the same sentence, 2.42, does match the table (80.94 - 78.52). No other pair of rows in Table 2 gives 1.86.
Second check: another model agreed
Report a problem with this finding
Substance Table 6, Ukrainian row prints Tokens (M) "6.5" for Size (GiB) "84.6": 6.5 million tokens cannot be right for an 84.6 GiB corpus, and it is a decimal in a column of integers.
- p. 11 Table 6, row uk Ukrainian (Tokens (M), Size (GiB))
6.5 84.6
- p. 11 Table 6, row uk Ukrainian
6.5
Only the authors can say what is right here.
Judge's note: The page 11 image shows "Ukrainian 6.5 84.6". The printed ratio is 0.08 M tokens per GiB, against about 84 for Russian (23408/278.0) and 95 for Bulgarian (5487/57.5). The size of 84.6 GiB is consistent with uk's sixth position in Figure 1, so the token cell is the anomalous one; it is probably in billions. This is a different row from the Swedish one in C014, so it is kept as a separate finding; the exact value is for the authors.
Second check: another model agreed
Report a problem with this finding
Substance Table 7 gives RoBERTa Base 8 attention heads ("RoBERTaBase 1 bBPE 12 768 3072 8 50k 125M"), while every other L=12, Hm=768 model in the table (BERT Base, mBERT, XLM-R Base) is listed with A=12.
- p. 12 Table 7, RoBERTaBase row, column A
bBPE 12 768 3072 8 50k 125M
- p. 12 Table 7, RoBERTaBase row
RoBERTaBase 1 bBPE 12 768 3072 8 50k 125M
Only the authors can say what is right here.
Judge's note: Rests on outside knowledge: that RoBERTa Base uses the BERT Base architecture with 12 attention heads. The page 12 image does show A=8 for RoBERTa Base and A=12 for BERT Base, mBERT and XLM-R Base at identical L/Hm/Hff; the 8 matches the XLM-15 and Unicoder rows, which suggests a copy slip. A does not enter the caption's parameter formula, so there is no internal arithmetic check, and the paper never states RoBERTa Base's head count elsewhere. Hence a query for the authors.
Second check: another model agreed
Report a problem with this finding
Reference or label Figure 1 y-axis is labelled "Dataset size (in GB)" while the caption of the same figure says "Amount of data in GiB (log-scale)" and Table 6 uses "Size (GiB)".
- p. 3 Figure 1, y-axis label
Dataset size (in GB)
- p. 3 Figure 1 caption
Amount of data in GiB (log-scale)
Judge's note: Seen on the page 3 image and the Figure 1 crop: the axis reads GB, the caption GiB. Table 6 on page 11 and the text ("capped at 60 GiB", "MiB") use binary units throughout. Unit label mismatch; immaterial on a log-scale plot.
Second check: another model agreed
Report a problem with this finding
12 minor (typos and the like; most checked by the judge only)
Reference or label Table 6 prints two rows with the same ISO code and name and different statistics, "my Burmese 15 0.4" and "my Burmese 56 1.6", with nothing to distinguish them (unlike the two zh rows, labelled Simplified / Traditional).
- p. 11 Table 6, right column, two consecutive rows
my Burmese 15 0.4
- p. 11 Table 6, right column, second Burmese row
my Burmese 56 1.6
- p. 11 Table 6
my Burmese 15 0.4 / my Burmese 56 1.6
- p. 11 Table 6 caption / rows
We report the list of 100 languages
Only the authors can say what is right here.
Judge's note: Checked on the page 11 image. Both rows are labelled identically, and a search of the text layer for Burmese found no explanation of a second variant or encoding. One row is missing a qualifier or is mislabelled; only the authors can say which. The reviewer claim of 103 rows is wrong: I counted 50 rows in each half, 100 in total, so the caption's "list of 100 languages" matches the row count with both Burmese rows and the 5 romanized rows included. The defect is the missing label only, not the count.
Second check: another model agreed
Report a problem with this finding
Wording Introduction says XLM-R "outperforms the previous state of the art by 5.1% average accuracy on XNLI, 2.42% average F1-score on Named Entity Recognition", but Section 5.2 gives 2.42 as the margin over mBERT ("outperforms mBERT by 2.42 F1 on average for cross-lingual transfer") and calls XLM-R only "on par with the state of the art" on NER.
- p. 1 Introduction, last paragraph on the page (continues on page 2)
outperforms the previous state of the art by 5.1% average accuracy on XNLI, 2.42% average F1-score
- p. 7 Section 5.2, Named Entity Recognition paragraph
outperforms mBERT by 2.42 F1 on average for cross-lingual transfer
Only the authors can say what is right here.
Judge's note: Read the introduction in the text layer (pages 1-2) and Table 2 and the NER paragraph on the page 7 image. 80.94 - 78.52 = 2.42 is XLM-R vs mBERT with train = en. Table 2 shows Akbik et al. (2018) ahead on en (93.18 vs 92.92) and de (88.27 vs 85.81). mBERT is the only zero-shot transfer baseline in Table 2, so the sentence may be defensible for that setting, but the paper does not say so. Wording issue only.
Second check: another model agreed
Report a problem with this finding
Wording Text says overall XNLI accuracy "decreases from 71.8% to 67.7% as we go from XLM-7 to XLM-100" (Table 5 also gives XLM-7 Wiki Avg 71.8), but the 7-language bar in Figure 4 sits at about 72.9.
- p. 5 Figure 4, bar for 7 languages (Fixed capacity)
Fixed capacity
- p. 4 Section 5.1, Transfer-dilution paragraph
decreases from 71.8% to 67.7% as we go from XLM-7 to XLM-100
- p. 8 Table 5, XLM-7 Wiki, cross-lingual transfer, Avg
71.8
- p. 4 Section 5.1, Transfer-dilution paragraph
decreases from 71.8% to 67.7%
Only the authors can say what is right here.
Judge's note: On the Figure 4 crop the 7-language fixed-capacity bar tops out nearly midway between the 72 and 74 gridlines (about 72.9). The 30-language fixed bar is about 71.9 and the 100-language bar about 67.6, which agrees with 67.7. The Table 5 XLM-7 Wiki row does average to 71.8. The roughly one-point gap is read off a bar height, so it is approximate, and the figure may come from a different run or evaluation set; the paper does not say. Only the authors can settle it.
Second check: another model agreed
Report a problem with this finding
Wording mBERT is listed with "102" languages in Table 1 (Devlin et al. (2018) row, #lg) and Table 3 (#lgs), but with "104" in Table 7 ("mBERT 104 WordPiece").
- p. 7 Table 1, Devlin et al. (2018) row, #lg column
102
- p. 8 Table 3, mBERT row, #lgs column
102
- p. 12 Table 7, mBERT row, #lgs column
104
- p. 12 Table 7, mBERT row
mBERT 104
Judge's note: Seen on the page images: Table 1 (page 7) 102, Table 3 (page 8) 102, Table 7 (page 12) 104. The paper gives no explanation. Either count could be legitimate if different released checkpoints are meant, so this is a minor consistency point.
Second check: another model agreed
Report a problem with this finding
Wording Section 5.4 says "pretraining on CC improved performance by up to 10 points", but Table 5 shows Swahili gains above 10 points (BERT 60.1 to 70.8; XLM-7 cross-lingual 60.8 to 71.2).
- p. 8 Section 5.4
pretraining on CC improved performance by up to 10 points
- p. 8 Table 5, BERT monolingual baselines, sw column (Wiki / CC)
60.1
Judge's note: Read Table 5 on the page 8 image. The Swahili gains are 10.7 and 10.4; the Urdu gain is 7.8. "Up to 10" slightly understates the largest gain; it is probably meant loosely.
Second check: another model agreed
Report a problem with this finding
Wording Section 5.4: "We observed in Table 5 that pretraining on Wikipedia for Swahili and Urdu performed similarly to a randomly initialized model", but Table 5 has no randomly initialized baseline.
- p. 8 Section 5.4, first sentence
We observed in Table 5 that pretraining on Wikipedia for Swahili and Urdu performed similarly to a randomly initialized model
- p. 8 Table 5, Monolingual baselines, BERT Wiki row (sw, ur)
60.1 57.3
Only the authors can say what is right here.
Judge's note: Table 5 on the page 8 image has only BERT (Wiki/CC) and XLM-7 (Wiki/CC) rows. A search of the text for "random" finds no other mention of a random-initialisation result in any table or figure. The comparison cannot be read from the cited table; the baseline is presumably unreported.
Second check: another model agreed
Report a problem with this finding
Typo Misspelling in Section 4: "we fine-tune a multiligual model on each training set" (should be "multilingual").
- p. 4 Section 4, XNLI paragraph, translate-train item
we fine-tune a multiligual
- p. 4 Section 4, XNLI paragraph, translate-train description
we fine-tune a multiligual model on each training set
Judge's note: The text layer reads "multiligual" in the translate-train item. On the page 4 image the word is consistent with that spelling, though at this resolution the missing letter is hard to see by eye. The next item spells "multilingual" correctly.
Second check: none (judge only)
Report a problem with this finding
Typo Table 5, XLM-7 Wiki translate-train-all, zh column prints "78" where every other cell has one decimal place (should be "78.0").
- p. 8 Table 5, XLM-7 Wiki translate-train-all, zh column
78
Judge's note: Seen on the page 8 image: the row reads 84.6 80.1 80.2 75.7 78 68.7 66.7 76.3. With 78.0 the row mean is 76.29, matching the printed 76.3. Formatting only.
Second check: none (judge only)
Report a problem with this finding
Typo Table 3 caption: "Results on MLQA question answering We report the F1 and EM" lacks a full stop after the bold title, and writes "English Squad dataset" where the body uses "SQuAD".
- p. 8 Table 3 caption
question answering We report the F1 and EM
Judge's note: Seen on the page 8 image. The other captions end the bold title with a full stop (Table 1: "Results on cross-lingual classification."). The body text on page 4 writes "SQuAD". Cosmetic.
Second check: none (judge only)
Report a problem with this finding
Typo Reference entry has a garbled author list: "Armand Joulin, Edouard Grave, and Piotr Bojanowski Tomas Mikolov. 2017." ("and" precedes the third name and the last two names run together without a separator).
- p. 9 References, Joulin et al. 2017
Armand Joulin, Edouard Grave, and Piotr Bojanowski Tomas Mikolov. 2017.
- p. 9 References, Joulin et al. 2017
Bo-janowski Tomas Mikolov. 2017.
Judge's note: Seen on the crop of the page 9 reference list; the names are merged in the PDF itself, not by extraction. The Grave et al. (2018) entry in the same list shows Piotr Bojanowski and Tomas Mikolov as separate authors, so the defect is visible from the paper alone.
Second check: none (judge only)
Report a problem with this finding
Typo Reference entry for GLUE prints the second author as "Amapreet Singh" ("Alex Wang, Amapreet Singh, Julian Michael"); reviewers say the name is Amanpreet Singh.
- p. 10 References, Wang et al. 2018
Alex Wang, Amapreet Singh, Julian Michael
Only the authors can say what is right here.
Judge's note: Rests on outside knowledge: that the GLUE paper's second author is spelled "Amanpreet Singh". The crop of page 10 does print "Amapreet Singh". The correct spelling cannot be checked from the files.
Second check: none (judge only)
Report a problem with this finding
Typo Table 7 caption: "remains significantly smaller that recently introduced Transformer models"; "that" should be "than". The same sentence has "a similar number of parameters than XLM-100", which should be "as" or "to".
- p. 12 Table 7 caption
remains significantly smaller that recently introduced Transformer models
Judge's note: Both wordings read on the page 12 image exactly as quoted. Plain typo and grammar slip, cosmetic. The suggested change of "hindsight" to "insight" is a word-choice point and is not included.
Second check: none (judge only)