XLNet: Generalized Autoregressive Pretraining for Language Understanding
2019 · paper · AI and machine learning
Proposes XLNet, a language pretraining method that learns bidirectional context by maximizing likelihood over all permutations of word order, avoiding BERT's masking drawbacks. Under comparable settings, it reportedly outperforms BERT on 20 tasks, including question answering and sentiment analysis.
Checked: PDF from arxiv.org, fetched 4 Oct 2026. Page numbers are those of that PDF.
5 findings
Reference or label A.6: "The one-side masked pattern is very similar to the lower-left part of Fig. 1-(d), with the upper-right triangle masked out" points to a panel that does not exist; Figure 1 has panels (a)-(c) only, and the matching heat map is Fig. 2-(d) "Content-based symmetry".
- p. 15 Appendix A.6, paragraph between Figures 2 and 3
very similar to the lower-left part of Fig.
- p. 15 Appendix A.6, same sentence, next line
1-(d), with the upper-right triangle masked out
- p. 4 Figure 1 caption
(c): Overview of the
- p. 15 Appendix A.6, paragraph after Figure 2
very similar to the lower-left part of Fig. 1-(d)
Judge's note: Page 4 image: Figure 1 is the two-stream attention architecture diagram with panels (a), (b), (c) and a caption listing only those three. Page 15 image: the sentence prints "Fig. 1-(d)"; Figure 2 on the same page is the only figure with a panel (d), and its heat map (diagonal structure in the lower-left) fits the comparison with Fig. 3(c). The pointer as printed is wrong; Fig. 2-(d) is the evident intended target.
Second check: another model agreed
Report a problem with this finding
Reference or label Table 3: the block headed "Test set results on leaderboard (single model, as of Dec 14, 2019)" contains the row "BERT∗ [10] 87.433 93.294", and the caption says "∗ indicates ensembles".
- p. 8 Table 3, test-set sub-header
Test set results on leaderboard (single model, as of Dec 14, 2019)
- p. 8 Table 3, SQuAD1.1 test rows
BERT∗[10]
- p. 8 Table 3 caption
indicates ensembles
- p. 8 Table 3, SQuAD1.1 test rows
BERT∗ [10] 87.433 93.294
Only the authors can say what is right here.
Judge's note: Viewed page 8. In the SQuAD1.1 test rows the second entry is BERT with an asterisk, and the caption defines the asterisk as an ensemble. Nothing in the text says the header applies only to the XLNet rows. Either the header or the row label is wrong, and only the authors can say which.
Second check: another model agreed
Report a problem with this finding
Reference or label Appendix A.1: the under-brace of the second term reads "z_t^(1) = j, z_<t^(2) = z_<t"; by the definition on the line above ("z_t^(1) = i ≠ j = z_t^(2)") it should read z_t^(2) = j.
- p. 12 Appendix A.1, displayed equation, under-brace beneath p_θ(X_j = x | x_{z<t})
=j, z(2) <t =z<t
- p. 12 Appendix A.1, definition line above the equation
= i ̸= j = z(2)
- p. 12 Appendix A.1, displayed equation, second underbrace
z(1) t =j, z(2) <t =z<t
- p. 12 A.1 displayed equation, second under-brace
z_t^(1)=j, z_{<t}^(2)=z_{<t}
Judge's note: Viewed the crop and page 12. The definition line gives z_t^(1) = i and z_t^(2) = j. The first under-brace reads z_t^(1) = i, z_<t^(1) = z_<t. The second under-brace shows superscript (1) on z_t with value j, while its second half carries (2). The superscript is a typographic slip with an evident fix; the argument is not affected.
Second check: another model agreed
Report a problem with this finding
Reference or label Figure 4, bottom-right panel (Factorization order: 4 → 3 → 1 → 2): the upper memory box feeding h3(2) is labelled "mem(0)"; the other three panels label it mem(1).
- p. 16 Figure 4, bottom-right panel (Factorization order: 4 → 3 → 1 → 2), upper memory box
mem(0)
Judge's note: Page 16 image plus a 4x render of the bottom-right panel made from the PDF. In that panel both memory boxes read mem(0). In the other three panels the lower box (feeding h3(1)) is mem(0) and the upper box (feeding h3(2)) is mem(1), which matches A.2, where layer m attends to the cached layer m-1 memory. Label slip in one panel. One reviewer note describes the rows the other way round, but the defect is the same.
Second check: another model agreed
Report a problem with this finding
Wording Section 3.3 says the experiment "reuses the hyper-parameters of RoBERTa, as described in section 3.1", but Section 3.1 never mentions RoBERTa; the only reuse it states is "we reuse all pretraining hyper-parameters as in the original BERT".
- p. 7 Section 3.3, paragraph after Table 2
reuses the hyper-parameters of RoBERTa, as described in section 3.1
- p. 6 Section 3.1, second paragraph
where we reuse all pretraining hyper-parameters as in the
Judge's note: Viewed pages 6 and 7 and searched the text for RoBERTa: it first appears in Section 3.3 and not in Section 3.1 or the hyperparameter appendix. Section 3.1 gives the full-data settings (500K steps, batch size 8192) without attributing them to RoBERTa. The pointer probably means that paragraph, so the attribution is missing at the place pointed to; it is a loose cross-reference, not a wrong one.
Second check: another model agreed
Report a problem with this finding
17 minor (typos and the like; most checked by the judge only)
Wording Abstract says "XLNet outperforms BERT on 20 tasks", but the tables give a BERT number for only 18 distinct tasks; the BERT cells for AG (Table 4) and WNLI (Table 5) are dashes.
- p. 1 Abstract
XLNet outperforms BERT on 20 tasks
- p. 8 Table 4, BERT [35] row, AG column
BERT [35] 4.51 1.89 29.32 0.64 - 2.63 34.17
- p. 8 Table 5, BERT [2] row, WNLI column
BERT [2] 86.6/- 92.3 91.3 70.4 93.2 88.0 60.6 90.0 -
Judge's note: Counted tasks with both a BERT and an XLNet value on pages 7-8. Table 1 has 11 (SQuAD1.1, SQuAD2.0, RACE, MNLI, QNLI, QQP, RTE, SST-2, MRPC, CoLA, STS-B). Table 2 adds ClueWeb09-B. Table 4 adds 6 (IMDB, Yelp-2, Yelp-5, DBpedia, Amazon-2, Amazon-5); the BERT [35] row has '-' under AG. Table 5 has '-' for BERT under WNLI. That gives 18; 20 is reached only by counting AG and WNLI. No text explains the count of 20. The authors may be counting all tasks evaluated, so this is a consistency point, not a clear error.
Second check: another model agreed
Report a problem with this finding
Typo Abstract ends with a full stop on both sides of the footnote mark: "inference, sentiment analysis, and document ranking.1."
- p. 1 Abstract, last line
inference, sentiment analysis, and document ranking.1.
- p. 1 Abstract, last sentence
document ranking.1.
Judge's note: Viewed the crop and page 1. The image shows "ranking." then superscript 1 then a second full stop. It is printed, not an extraction artefact.
Second check: none (judge only) · checked by hand
Report a problem with this finding
Typo Figure 1 caption is missing "to": "which does not have access information about the content".
- p. 4 Figure 1 caption
which does not have access information about the content
- p. 4 Figure 1 caption (b)
does not have access information about the content
Judge's note: Viewed page 4. The caption for (b) reads as quoted; it should read "access to information".
Second check: none (judge only)
Report a problem with this finding
Typo Subject-verb disagreement: "XLNet still lead to substantial gains" should be "leads".
- p. 8 Section 3.3, second bullet after Table 5
XLNet still lead to substantial gains
Judge's note: Viewed page 8, second bullet after Table 5. Printed as quoted.
Second check: none (judge only)
Report a problem with this finding
Typo Duplicated word: "Finally, we unexpectedly find the the next-sentence prediction objective".
- p. 9 Section 3.4, paragraph after Table 6
we unexpectedly find the the
- p. 9
we unexpectedly find the the next-sentence prediction objective
- p. 9
Finally, we unexpectedly find the the next-sentence predi
Judge's note: Viewed page 9. The line ends "find the the" and the next line begins "next-sentence"; both words are printed on the same line.
Second check: code check
Report a problem with this finding
Typo Missing "to": "clearly contribute the superior performance of XLNet over BERT".
- p. 9 Section 3.4, paragraph after Table 6
contribute the superior performance of XLNet over BERT
- p. 9 Section 3.4, para 2
clearly contribute the superior performance
- p. 9 Section 3.4, paragraph after Table 6
clearly contribute the superior performance of XLNet
Judge's note: Viewed page 9, paragraph after Table 6. Printed as quoted; it should read "contribute to the superior performance".
Second check: none (judge only)
Report a problem with this finding
Typo Reference [6] has a malformed URL: "URl: http://http://commoncrawl. org, 2019."
- p. 10 Reference [6]
URl: http://http://commoncrawl. org
- p. 10 Reference [6]
URl: http://http://commoncrawl. org, 2019.
Judge's note: Viewed page 10. The scheme "http://" is printed twice, "URl" has a lower-case l, and there is a space before "org".
Second check: none (judge only)
Report a problem with this finding
Typo Reference [11] title ends "better text generation via filling in the_.", with a single underscore run onto "the".
- p. 10 Reference [11]
filling in the_.
Only the authors can say what is right here.
Judge's note: Rests on outside knowledge: the MaskGAN paper's title ends in a blank ("Filling in the ______"). Viewed page 10: the entry is printed as quoted. If the title does end in a blank, the entry is a compressed rendering of it and not a truncation, so this is cosmetic at most.
Second check: none (judge only)
Report a problem with this finding
Typo Reference [26] repeats the report type and publisher: "English gigaword fifth edition, linguistic data consortium. Technical report, Technical Report. Linguistic Data Consortium, Philadelphia, Tech. Rep., 2011."
- p. 11 Reference [26]
Technical report, Technical Report. Linguistic Data
- p. 11 Reference [26]
Technical report, Technical Report. Linguistic Data Consortium, Philadelphia, Tech. Rep., 2011.
Judge's note: Viewed page 11. "Technical report" appears twice and "Tech. Rep." once, and "linguistic data consortium" appears in both the title and the publisher field.
Second check: none (judge only)
Report a problem with this finding
Typo Appendix A.2 misspells "representation" three times: "Initial represetation:", "Cached layer-m content represetation (memory)", and "employed to update the represetntations:".
- p. 12 Appendix A.2, first label
Initial represetation:
- p. 12 Appendix A.2, second label
Cached layer-m content represetation (memory)
- p. 12 Appendix A.2, sentence before the update equations
employed to update the represetntations:
- p. 12 Appendix A.2
employed to update the represetntations
Judge's note: Viewed page 12. All three misspellings are printed in the image.
Second check: none (judge only)
Report a problem with this finding
Typo A.3.1: "the average length of the passages in RACE are longer than 300" has a singular subject with a plural verb (should be "is longer than 300"), and 300 carries no unit.
- p. 12 Appendix A.3.1
the average length of the passages in RACE are longer than 300
Judge's note: Read on the page 12 image. The sentence is printed as quoted. Grammar slip only; the missing unit (words or tokens) is a minor ambiguity and the meaning is recoverable.
Second check: none (judge only)
Report a problem with this finding
Typo A.2: "employed to update the represetntations:" misspells "representations".
- p. 12 A.2 intro to update equations
update the represetntations:
Judge's note: Read on the page 12 image: the word is printed "represetntations". The same section also prints "Initial represetation" and "content represetation", which are separate misspellings.
Second check: none (judge only)
Report a problem with this finding
Typo A.4.2 defines the term as "Layer-wise decay" while the Table 8 row it explains is labelled "Layer-wise lr decay".
- p. 13 A.4.2 text
“Layer-wise decay” means
- p. 14 Table 8 row label
Layer-wise lr decay
Judge's note: Table 8 row label read on the page 14 image; the A.4.2 wording ("“Layer-wise decay” means exponentially decaying the learning rates...", and "the Layer-wise decay rate is α") checked in the page 13 text. The quoted term drops "lr". The definition makes clear it is the learning rate that decays, so this is a naming inconsistency with no risk of misreading.
Second check: none (judge only)
Report a problem with this finding
Typo A.5.2 says "consider a context-target pair (x, U)" where A.5.1 defines and uses "target-context pair" for the same ordered pair (x, U).
- p. 14 Appendix A.5.2, sentence before the bullets
consider a context-target pair (x, U):
- p. 14 Appendix A.5.1, first paragraph
we define a set of target-context pairs
- p. 14 Appendix A.5.2
consider a context-target pair (x, U)
- p. 14 Appendix A.5.1
we say a target-context pair (x, U) ∈ I is covered
Judge's note: Read on the page 14 image. A.5.1 uses "target-context pairs" / "target-context pair (x, U)" three times, with the target x first. A.5.2 reverses the name for the same pair. Terminology slip; the meaning is unchanged.
Second check: none (judge only)
Report a problem with this finding
Typo A.6: "attends to positions every a few stride apart" is ungrammatical (should be "every few strides apart" or "a few strides apart").
- p. 15 Appendix A.6, paragraph between Figures 2 and 3
attends to positions every a few stride apart
Judge's note: Read on the page 15 image; the phrase is printed as quoted. Cosmetic.
Second check: none (judge only)
Report a problem with this finding
Typo A.7: "such that only tokens the occur before z_t in the permutation can be attended" has "the" for "that".
- p. 16 Appendix A.7, second paragraph
such that only tokens the occur before
- p. 16 A.7
only tokens the occur before zt
- p. 16 A.7
only tokens the occur before
- p. 16
only tokens the occur before zt in the permutation
Judge's note: Read on the page 16 image; printed as quoted. Typo.
Second check: none (judge only)
Report a problem with this finding
Typo Figures 5 and 6, Position-2 View, layer-1 row: the greyed-out pair for position 1 reads "h1(1)" "h1(1)"; the second box should be g1(1).
- p. 17 Figure 5, split view, Position-2 View, layer-1 row, second box
h1(1)
- p. 18 Figure 6, split view, Position-2 View, layer-1 row, second box
h1(1)
Judge's note: Checked on 6x renders of the Position-2 View made from the PDF for pages 17 and 18. In both figures the first pair on the layer-1 row is h1(1), h1(1), while the layer-2 row above it reads h1(2), g1(2), and the Position-3 View on page 17 reads h1(1), g1(1). The boxes are greyed out (inactive in that view), so the slip does not change what the figure conveys; hence cosmetic.
Second check: none (judge only)