Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context

• p. 1 Abstract

  text articles with thousands of tokens

• p. 2 Introduction

  cles with thousands of tokens (see Appendix E)

• p. 8 Section 4.4

  relatively coherent articles with thousands of to-kens

• p. 16 Appendix E

  tokens (500 or 1,000 in our case)

Only the authors can say what is right here.

Judge's note: The Appendix E text on page 16 limits generation to 500 or 1,000 tokens after a seed of at most 512 tokens. No sample cited as support reaches 'thousands' of generated tokens. The authors may have longer unreported samples, so the wording is put as a query.

Second check: another model agreed

• p. 8 Section 4.3, text after Table 8

  The RECL of Transformer-XL is 80% and 450% longer than recurrent net-works and Transformer respectively

• p. 8 Table 8

  Transformer-XL 128M

• p. 1 Abstract

  80% longer than RNNs and 450% longer than vanilla Trans-formers

• p. 8 Section 4.3 text and Table 8

  The RECL of Transformer- XL is 80% and 450% longer than recurrent net- works and Transformer respectively.

Only the authors can say what is right here.

Judge's note: Table 8 on page 8 gives Transformer-XL 151M = 900 and QRNN = 500 at r=0.1, and Transformer-XL 128M = 700 and Transformer = 128. The caption says RECL is computed per model group, so the within-group comparison is legitimate. The sentence does not say which Transformer-XL model each percentage uses, and its subject reads as the 900-word model.

Second check: another model agreed

• p. 14 Appendix D, first paragraph

  Recall that, this model has 16 10-head transformer layers and relies on a memory of length 640.

• p. 6 Section 4.1, WikiText-103 paragraph

  the attention length to 384 during training and 1600 during evaluation

• p. 8 Table 6, Transformer-XL (151M) row

  640

• p. 14 Appendix D

  this model has 16 10-head transformer layers and relies on a memory of length 640

Only the authors can say what is right here.

Judge's note: Checked p.14 (Appendix D, Fig. 5 caption 'previous 640 tokens'), p.6 (Sec. 4.1: 384/1600) and p.8 (Table 6: 640 appears only as the attention length for the 151M ablation model; text says attention length 'can be increased to 640 at test time' in the ablation). I searched the text and found no earlier statement of 16 layers, 10 heads or memory 640 for the SoTA (257M, 18.3 PPL) model. So 'Recall that' points back to something the paper does not say. The visualisation may have used a shorter memory than the main evaluation, but the paper does not say so. Which model and settings were visualised is a question for the authors.

Second check: another model agreed

• p. 12 Table 10, Loss column and header

  Partial

• p. 8 Table 6, Loss column

  Half

• p. 12 Table 10 header

  pplx best

• p. 8 Table 6 header

  PPL init

Judge's note: The page-12 and page-8 images show identical numbers under different labels. Section 4.2 (page 7) defines only 'Full' and 'half' losses, so 'Partial' is undefined. The values sit in the correct columns, so only the terminology and column order differ.

Second check: another model agreed

• p. 14 Appendix C, RECL algorithm step 2

  set c = c′, c′ = c + ∆ and go to step 1.

Judge's note: Read on the page-14 image. Taken literally, step 1 resets c to its initial value and cancels the update made in step 2. The intended loop is clear: recompute g_i(c,c′), i.e. go to step 2.

Second check: another model agreed

• p. 14 Appendix D, first bullet

  For the head 8 in the 1st layer, we see an almost uniform attention over the entire memory span

• p. 15 Appendix D, third bullet

  different from the case of head 8, where a few locations are clearly attended more than others

Only the authors can say what is right here.

Judge's note: p.14 first bullet: head 8 is near-uniform. p.15 third bullet: the relative clause attaches to head 8. Fig. 6a (head 8) on p.15 looks diffuse, while Fig. 6c (head 158) shows scattered peaks. The clause probably means to describe head 158 in contrast to head 8, but as printed the two descriptions of head 8 contradict each other. The wording is ambiguous; I do not take this as evidence of a wrong head number.

Second check: another model agreed

• p. 17 Observations below Table 11, third bullet

  accurately uses the the phrase “another back injury” in the 2013 season paragraph

• p. 17 Table 11, XL Generation, 2012 season

  expected to miss the season as a result of a shoulder injury

• p. 17 Table 11 commentary, third bullet

  accurately uses the the phrase

Judge's note: On the p.17 image, the XL Generation 2012 section mentions 'After an injury to closer Javier Vázquez' and 'expected to miss the season as a result of a shoulder injury'. Neither is a back injury. 'Another' could be read loosely as 'another injury', so calling the use 'accurate' is an overstatement rather than a clear error. The typo parts are covered by C023.

Second check: another model agreed

• p. 1 Abstract

  0.99 on en- wiki8

• p. 4 Section 3.2, para 'Besides achieving extra long context'

  periments on enwiki8, Transformer-XL is up to

• p. 4 Section 3.2

  In our ex-periments on enwiki8

Judge's note: In the page-1 abstract crop the text reads 'en-/wiki8', with the extra 'i'. The page-4 crop reads 'experiments on enwiki8'. The Table 2 caption on page 6 reads 'enwik8'.

Second check: none (judge only)

• p. 1 Introduction, col 2

  wide adaption, RNNs are difficult to optimize

• p. 2 Section 2

  since the ubiquitous adaption of LSTM

Judge's note: Both instances are visible in the page images (p1 col 2, p2 Section 2). The sense is widespread use, so the word should be 'adoption'.

Second check: none (judge only)

• p. 5 Section 3.3, computational procedure, A^n_{τ,i,j}

  + u⊤kτ,j + v⊤Wn k,RRi−j

Judge's note: The page-5 crop shows A^n_{τ,i,j} = q^n⊤k^n + q^n⊤W^n_{k,R}R_{i−j} + u⊤k_{τ,j} + v⊤W^n_{k,R}R_{i−j}. The superscript n on k is missing only in term (c). This is a notational inconsistency, and the meaning is clear.

Second check: none (judge only)

• p. 16 Appendix E

  Wikitext-103 dataset

• p. 6 Section 4.1

  WikiText-103 (Merity et al.,

Judge's note: Both spellings are visible on p.6 and p.16/17. This is a cosmetic consistency point.

Second check: none (judge only)

• p. 7 Section 4.2, last sentence

  relative positional encodings is also superior to

• p. 18 Notes below Table 12

  This example exhibit some additional interesting properties

• p. 14 Figure 5 caption

  There are totally 160 attention heads

• p. 14 Figure 5 caption

  corresponds to a attention head

Judge's note: All phrases are visible in the page-7, page-18 and page-14 crops. They are subject-verb agreement and article errors.

Second check: none (judge only)

• p. 8 Section 4.3

  See Appedix C for more details about RECL.

• p. 8 Section 4.3

  See Appedix C for more details about RECL

Judge's note: Visible in the page-8 image. The pointer target (Appendix C, RECL details) is correct.

Second check: none (judge only)

• p. 8 Section 4.3

  measure by the relative improvement over the best

Judge's note: Visible in the page-8 image (Section 4.3).

Second check: none (judge only)

• p. 14 Appendix C

  we define the set T is the above equation as

Judge's note: Visible on the p.14 page image, first paragraph.

Second check: none (judge only)

• p. 16 Appendix E, first paragraph

  We seed the our Transformer-XL

• p. 16 Appendix E, first paragraph

  denote the Wikipedia page tile, section title and subsection title

Judge's note: Visible on the p.16 page image, Appendix E first paragraph. The 'page tile' typo is handled in C020 and the missing 'on' in C021.

Second check: none (judge only)

• p. 16 Appendix E, first paragraph

  denote the Wikipedia page tile, section title and subsection title

Judge's note: Visible on the p.16 page image.

Second check: none (judge only)

• p. 16 Appendix E

  trained the Wikitext-103 dataset

• p. 16 Appendix E

  We seed the our Transformer-XL

Judge's note: Visible on the p.16 page image, first sentence of Appendix E. The 'the our' part of this cluster is covered by C019.

Second check: none (judge only)

• p. 17 Observations below Table 11

  accurately uses the the phrase

• p. 17 Observations below Table 11

  one earlier injure in the 2012 season

• p. 17 Table 11 commentary, third bullet

  accurately uses the the phrase “another back injury” in the 2013 season paragraph, since it has talked about one earlier injure in the 2012 season

• p. 17 Table 11, XL Generation, 2012 season

  expected to miss the season as a result of a shoulder injury

Judge's note: All three are visible on the p.17 page image in the bullets after Table 11. The shoulder-vs-back injury point is judged in C024.

Second check: code check