Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation
2021 · paper · AI and machine learning
Studies how Transformer language models handle inputs longer than seen in training, and introduces ALiBi, which penalizes attention scores in proportion to distance instead of using position embeddings. A model trained on length 1024 matches one trained on 2048, training 11% faster.
Checked: PDF from arxiv.org, fetched 4 Oct 2026. Page numbers are those of that PDF.
4 findings
Substance Section 2.2: "the L = 1024 sinusoidal model runs at 28.5k words per second (WPS)" citing Appendix Table 1, but Table 1 gives 26.0k for sinusoidal L=1024; 28.5k is the L=512 value.
- p. 4 Section 2.2, T5 Bias paragraph
the L = 1024 sinusoidal model runs at 28.5k words per second (WPS)
- p. 15 Table 1, Sinusoidal rows
26.0k
- p. 15 Table 1, Sinusoidal rows
1024 26.0k 77.8k 19.2 GB
Judge's note: Page 4 text checked against Table 1 on page 15: Sinusoidal 512 = 28.5k, 1024 = 26.0k (train). The sentence's conclusion (no speedup compared with T5 bias at 14.4k) is unaffected.
Second check: another model agreed
Report a problem with this finding
Reference or label In Table 6 the Train Speed↑ entry "11.5k" (Rotary) is bold, but the highest value in that column is 28.3k (ALiBi L=512).
- p. 20 Table 6, Train Speed column
11.5k
Judge's note: The crop of Table 6 on page 20 shows bold on 11.5k in the Train Speed↑ column, which reads 15.3k, 11.5k, 4.3k, 28.3k, 15.5k. In every other column bold marks the best value (247M; 13.6k; 17.60; 18.30). No legend gives bold another meaning. 11.5k is the second-lowest value, so the bold looks misplaced.
Second check: another model agreed
Report a problem with this finding
Wording Figure 2 caption says "The speed differences between our method and the sinusoidal are within 1% during training and 3% for inference", but Table 1 (cited for exact numbers) shows 15.5k vs 15.3k training at L=3072 (1.3%) and 85.8k vs 82.1k inference at L=512 (4.5%).
- p. 3 Figure 2 caption
are within 1% during training and 3% for inference
- p. 15 Table 1
85.8k
- p. 15 Table 1
28.3k 85.8k 15.3 GB
- p. 3 Figure 2 caption
within 1% during training and 3% for inference
Only the authors can say what is right here.
Judge's note: Read Table 1 on page 15. Training gaps: 0.7% (512), 0.8% (1024), 1.3% (3072). Inference gaps: 4.5% (512), 1.8% (1024), 0.5% (3072). The two cases above the stated bounds are both ones where ALiBi is faster, so the caption may mean 'no more than 1%/3% slower'. As written, though, the bounds are two-sided and Table 1 exceeds them. The memory statement (100MB) does match Table 1.
Second check: another model agreed
Report a problem with this finding
Wording Appendix A.2: models with 512 ≤ L < 3072 "are faster to train than the sinusoidal model with L = 3072, but they all achieve greater perplexity scores on the validation set". They achieve lower (better) perplexity: 18.40, 17.96, 17.72, 17.64 vs 18.67.
- p. 19 Section A.2, paragraph after Figure 8
but they all achieve greater perplexity scores on the validation set
- p. 19 Figure 8 caption
achieve better perplexity than the sinusoidal model
Judge's note: Page 19: Table 5, the Lvalid=3072 column, and Figure 8 both show lower perplexity for these ALiBi models. The Figure 8 caption on the same page says 'better perplexity'. Since lower is better for perplexity, 'greater' states the opposite. The data and caption are correct; only the sentence misdescribes them.
Second check: another model agreed
Report a problem with this finding
16 minor (typos and the like; most checked by the judge only)
Wording Section 4.1: "without extrapolation, for every L, our models outperform those using the sinusoidal method" (citing Appendix Table 5), but Table 5 has sinusoidal better at L=64 (28.03 vs 28.46) and L=128 (23.81 vs 23.98), with the sinusoidal values bolded.
- p. 6 Section 4.1, paragraph after Figure 4
without extrapolation, for every L, our models outperform those using the sinusoidal method
- p. 19 Table 5, diagonal vs Sinusoidal* row
28.03
Only the authors can say what is right here.
Judge's note: Page 6 sentence explicitly references Appendix Table 5; Table 5 on page 19 diagonal vs Sinusoidal* row confirms the exceptions at L=64 and 128. The claim holds for L>=256 and for the Figure 4 range.
Second check: another model agreed
Report a problem with this finding
Wording Page 7 says "training the L = 3072 sinusoidal model requires a GPU with more than 16 GB of memory", and the Figure 8 caption says the shorter-L ALiBi models "require less time and memory to train". Table 1, however, lists 15.1 GB for sinusoidal L=3072, against 15.3 GB (ALiBi 512) and 19.3 GB (ALiBi 1024).
- p. 7 Section 4.1, second paragraph
requires a GPU with more than 16 GB of memory
- p. 19 Figure 8 caption
require less time and memory to train
- p. 15 Table 1, Memory column
15.1 GB
- p. 19 Figure 8 caption
even though all of them (except the L = 3072) require less time and memory to train
Only the authors can say what is right here.
Judge's note: The Figure 2 caption (p.3) partly explains this: memory at 3072 is lower 'since we break batches into multiple updates'. So the Table 1 figure is not the unsplit requirement. The paper never states the unsplit memory that would back the >16 GB and 'less memory' claims, so the reported figures do not support them as written.
Second check: another model agreed
Report a problem with this finding
Wording Section 4.2 says that on the smaller datasets "ALiBi consistently outperforms other position methods even when not extrapolating", but Table 5 shows T5 bias better at L=512 (19.65 vs 19.73, T5 bolded) and sinusoidal better at L=64 and 128.
- p. 8 Section 4.2, paragraph on Appendix Table 12
ALiBi consistently outperforms other position methods even when not extrapolating
- p. 19 Table 5, T5 Bias* row, column 512
19.65
Only the authors can say what is right here.
Judge's note: Checked the page 8 text against Table 5 on page 19. Page 7 itself restricts the rotary/T5 comparison to 'Ls of 1024 and 3072'. This is a separate sentence from C004, but the same Table 5 exceptions apply.
Second check: another model agreed
Report a problem with this finding
Wording Conclusion: "when not extrapolating, our method achieves either better perplexity than the sinusoidal method (in models smaller than 1B parameters, trained on less data)". Table 5 shows sinusoidal better at L=64 (28.03 vs 28.46) and L=128 (23.81 vs 23.98).
- p. 9 Section 6 Conclusion
our method achieves either better perplexity than the sinusoidal method (in models smaller than 1B parameters, trained on less data)
- p. 19 Table 5
23.81
Only the authors can say what is right here.
Judge's note: Checked the page 9 conclusion against Table 5 on page 19. The claim holds for L>=256, and the sentence states no restriction. Appendix B (p.24) words this more carefully as 'either superior or similar'.
Second check: another model agreed
Report a problem with this finding
Wording Section 4.2: "In Appendix B, we find that ALiBi's edge over sinusoidal embeddings is largely explained by its improved avoidance of the early token curse", whereas Appendix B states that "ALiBi's decrease in perplexity when given longer sequences" is so explained.
- p. 9 Section 4.2, last paragraph
ALiBi’s edge over sinusoidal embeddings is largely explained by its improved avoidance of the early token curse
- p. 22 Appendix B intro
ALiBi’s decrease in perplexity when given longer sequences is largely explained by its improved avoidance of the early
Only the authors can say what is right here.
Judge's note: Page 9 vs page 22. Under sliding-window evaluation, which curtails the early token curse, ALiBi L=3072 still scores 16.96 (Table 15) against sinusoidal 18.03 (Table 13, p.25; also Figure 11, p.24). The main-text summary therefore describes a broader finding than Appendix B reports.
Second check: another model agreed
Report a problem with this finding
Wording Section B.2 says ALiBi's results "mirror" the T5 bias, for which "perplexity remains relatively flat when evaluating longer sequences (see Table 14)", but Table 14 shows the L=512 T5 model rising from 17.92 to 30.77.
- p. 24 Section B.2
perplexity remains relatively flat when evaluating longer sequences (see Table 14)
- p. 25 Table 14, row 512
30.77
- p. 25 Table 14
512 17.92 18.51 20.36 22.62 30.77
Only the authors can say what is right here.
Judge's note: Page 25, Table 14: T5 bias L=512 reads 17.92, 18.51, 20.36, 22.62, 30.77; L=1024 reads 17.65, 17.87, 18.51, 20.66. Table 15: ALiBi L=512 goes from 17.98 to 18.3. Next to the sinusoidal blow-up in Table 13 (up to 360.12), T5 is flat by comparison. But the 72% rise for L=512 is much larger than ALiBi's 2%, so the words "mirror" and "relatively flat" describe the data loosely. The authors should say what the comparison is against.
Second check: another model agreed
Report a problem with this finding
Typo Duplicated word in Introduction: "the number of tokens on which the the model was trained".
- p. 1 Introduction, paragraph 2
on which the the model was trained
- p. 1
the number of tokens on which the the model was trained
- p. 1
the number of tokens on which the the model was trained.
Judge's note: Visible in the page 1 image, introduction paragraph 2. A plain duplicated article.
Second check: code check
Report a problem with this finding
Typo Method name spelled "AliBi" instead of "ALiBi" in "AliBi incurs a memory increase" and "trained with AliBi on a much larger" (p.6) and "even when AliBi is trained on fewer tokens" (p.21).
- p. 6 Implementation paragraph
AliBi incurs a memory increase
- p. 6 Section 4, first paragraph
trained with AliBi on a much larger
- p. 21 Section A.3
even when AliBi is trained on fewer tokens
Judge's note: All three occurrences are visible in the page 6 image and the page 21 crop. Everywhere else the paper writes ALiBi.
Second check: none (judge only)
Report a problem with this finding
Typo Conclusion: "our ALiBi method offers an extremely simple replacement for existing position approaches and allow models to extrapolate" — 'allow' should agree with 'method' ('allows').
- p. 9 Section 6
existing position approaches and allow models to extrapolate
Judge's note: Visible in the page 9 image. 'allow' is coordinated with 'offers' under the singular subject 'our ALiBi method'.
Second check: none (judge only)
Report a problem with this finding
Typo Missing comma: "the sinusoidal, rotary T5 bias, and ALiBi position methods" (should be 'rotary, T5 bias').
- p. 15 Paragraph after Table 1
the sinusoidal, rotary T5 bias, and ALiBi position methods
Judge's note: Visible in the page 15 image below Table 1. Rotary and T5 bias are separate methods; the Table 1 caption on the same page writes 'rotary, T5 bias'.
Second check: none (judge only)
Report a problem with this finding
Typo Duplicated word across a line break in Appendix A.2: "(the standard deviation for the the sinusoidal model with L = 3072 is 0.24)".
- p. 19 Section A.2, first paragraph
the standard deviation for the the sinusoidal model
- p. 19
; (the standard deviation for the the sinusoidal model wi
Judge's note: Visible in the page 19 image ('for the' at the end of one line, 'the sinusoidal' at the start of the next).
Second check: code check
Report a problem with this finding
Typo Appendix A.2 has a duplicated word: "(the standard deviation for the the sinusoidal model with L = 3072 is 0.24)".
- p. 19 A.2 text after Figure 8
for the the sinusoidal model with L = 3072
Judge's note: The crop of page 19 shows "the" printed twice across the line break, "for the / the sinusoidal model".
Second check: none (judge only)
Report a problem with this finding
Typo Appendix A.3 misspells "conjunction" as "conjuction": "(in conjuction with the English Wikipedia)".
- p. 20 Section A.3
in conjuction with the English Wikipedia
- p. 20 A.3
(in conjuction with the English Wikipedia)
Judge's note: The page 20 crops show "conjuction" as printed.
Second check: none (judge only)
Report a problem with this finding
Typo Appendix A.3 calls the corpus "Toronto BooksCorpus", while elsewhere the paper writes "Toronto BookCorpus" (Fig. 9, Table 8) and "Toronto Book Corpus" (Sec. 4, Tables 9-10).
- p. 20 Appendix A.3, paragraph 3
Toronto BooksCorpus (Zhu et al., 2015), which has been used to train BERT (Devlin et al., 2019) (in conjuction
Judge's note: The crop shows "Toronto BooksCorpus". A text search finds "Toronto BookCorpus" in the Fig. 9 and Table 8 captions and "Toronto Book Corpus" at line 491 and in Tables 9-10, so the name is spelled inconsistently. The "conjuction" misspelling in this note is the same defect as C018.
Second check: none (judge only)
Report a problem with this finding
Typo The Figure 10 caption misspells "sliding": "the five passes required by the siding window approach".
- p. 23 Figure 10 caption
required by the siding window approach
Judge's note: The crop of the Figure 10 caption on page 23 shows "siding window". "Sliding window" is spelled correctly elsewhere in the same caption.
Second check: none (judge only)
Report a problem with this finding
Typo The Early Token Curse paragraph (B.1) misspells "occurring": "predictions occuring early in each subsequence".
- p. 23 Early Token Curse paragraph
predictions occuring
- p. 23 Appendix B.1, Early Token Curse
predictions occuring early
Judge's note: The crop of page 23 shows "occuring" as printed.
Second check: none (judge only)