Paper check About

FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness

Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, Christopher Ré

Advances in Neural Information Processing Systems 35, 2022 · paper · AI and machine learning

Introduces FlashAttention, an exact attention algorithm that reduces reads and writes between GPU memory levels using tiling. It speeds up Transformer training, for example 3× on GPT-2, and enables longer contexts and better results on long-sequence tasks.

Checked: PDF from arxiv.org, fetched 4 Oct 2026. Page numbers are those of that PDF.

8 findings

Substance Section 4 bullet: "speeds up GPT-2 up to 3× over HuggingFace [87] and 1.8× over Megatron [77]", but Section 4.1 and the Table 2 caption say "1.7×", and the Table 2 times support 1.7× at most.

p. 7 Section 4, Training Speed bullet
1.8× over Megatron [77] over standard Transformers
p. 7 Section 4.1 GPT-2 paragraph
1.7× speedup compared to Megatron-LM
p. 8 Table 2 caption
up to 1.7× compared to Megatron-LM
p. 7 Section 4, 'Training Speed' bullet
1.8× over Megatron [77]

Judge's note: Table 2 (p.8): 4.7/2.7 = 1.74 and 11.5/6.9 = 1.67. The 1.8× in Table 2 is Megatron-medium vs HuggingFace (21.0/11.5), a different quantity. The same comparison is printed two ways.

Second check: another model agreed

Report a problem with this finding

Substance The bottom panel of Figure 8 ('FlashAttention Speedup, T4 (Forward Only)') appears bar-for-bar identical to Figure 7 ('FlashAttention Speedup, GTX 3090', combined pass on RTX 3090).

p. 30 Figure 8 bottom panel
FlashAttention Speedup, T4 (Forward Only)
p. 29 Figure 7
FlashAttention Speedup, GTX 3090

Only the authors can say what is right here.

Judge's note: Compared the page 29 and page 30 images. All 15 bars match in height: at 128 ≈2.85/3.55/2.65; at 512 ≈4.0/3.65/2.4; at 2048 ≈4.6/4.35/2.5. The y-axis ranges and legends are also the same. The two figures claim a different GPU and a different pass, so identical values are implausible and one panel may be a duplicate. The T4 forward-only panel also shows larger speedups (up to ~4.6×) than the T4 combined panel (≤3.75×), while the text on page 29 says 'we observe less speedup on T4'. Only the authors can supply the correct panel.

Second check: another model agreed

Report a problem with this finding

Substance The LSformer row of Table 14 (combined pass, masking only), '3.08 3.10 4.26 10.90 31.59 61.72 121.51 241.18', is identical to the LSformer row of Table 11 (combined pass, dropout + masking).

p. 33 Table 14, LSformer row
3.08 3.10 4.26 10.90 31.59 61.72 121.51 241.18
p. 32 Table 11, LSformer row
3.08 3.10 4.26 10.90 31.59 61.72 121.51 241.18

Only the authors can say what is right here.

Judge's note: Compared the page 32 and page 33 images; all eight values are the same. The LSformer forward and backward rows differ between the two settings (Table 9 vs 12, Table 10 vs 13), and the LSformer row of Table 17 (dropout only) also differs. The stated exception on page 31 (measured without masking) covers only Block Sparse, Longformer and BigBird; their rows do repeat between Tables 11 and 17, as expected. Nothing explains an identical LSformer row, which points to a copied row. The combined pass need not equal forward plus backward, so the sums below are only indicative.

Second check: another model agreed

Report a problem with this finding

Reference or label "We reproduce the algorithm description in Algorithm 5 in Appendix B", but Algorithm 5 is printed on p.25 and described in Appendix D.1.

p. 7 Section 3.3, first paragraph
We reproduce the algorithm description in Algorithm 5 in Appendix B.
p. 25 Algorithm 5 / Appendix D.1
We describe the full block-sparse FlashAttention algorithm in Algorithm 5.

Judge's note: Appendix B (Algorithm Details) starts at text line 1146, C (Proofs) at 1570 and D at 1845. Algorithm 5 is at line 1759 on p.25, directly before D.1, which says 'We describe the full block-sparse FlashAttention algorithm in Algorithm 5.'

Second check: another model agreed

Report a problem with this finding

Wording Section 4.3 says block-sparse FlashAttention "is faster than all implementations of exact, sparse, and approximate attention that we know of, across all sequence lengths", yet Table 11 has exact FlashAttention faster at lengths 128 and 256 (0.43 and 0.41 ms against 0.44 and 0.44 ms).

p. 10 Section 4.3 Runtime
is faster than all implementations of exact, sparse, and approximate attention that we know of, across all sequence lengths
p. 32 Table 11 (dropout and masking, fwd+bwd), 128 and 256 columns
0.43 0.41

Only the authors can say what is right here.

Judge's note: I looked at Table 11 on page 32 (forward plus backward pass, with dropout and masking). FlashAttention is in bold as best at 128 and 256, and block-sparse FlashAttention is underlined as second best. FlashAttention is an exact implementation the authors know of, so 'across all sequence lengths' is not borne out by their own table.

Second check: another model agreed

Report a problem with this finding

Wording Appendix B.1 says "q_i and k_j are the i-th and j-th columns of Q and K respectively" and "Let v_j be the j-th column of V, then the i-th columns of the output is", but with Q, K, V, O ∈ R^{N×d} these are rows.

p. 18 Appendix B.1
where qi and kj are the i-th and j-th columns of Q and K respectively
p. 18 Appendix B.1
Let vj be the j-th column of V, then the i-th columns of the output is
p. 18 Appendix B.1, below the definition of S, P, O
are the i-th and j-th columns of Q and K respectively

Judge's note: On p18 the same paragraph declares Q, K, V ∈ R^{N×d} and S = QK^T ∈ R^{N×N}. Then S_ij = q_i^T k_j with i, j ranging up to N only works if q_i and k_j are rows (d-vectors). Likewise o_i = P_{i:}V = Σ_j P_ij v_j is row i of O, a sum over the rows of V. The derivation is correct once 'columns' is read as 'rows'.

Second check: another model agreed

Report a problem with this finding

Wording Proof of Theorem 1: "Let V_{j:j+1} be the slice of V from column jB_c to column (j + 1)B_c − 1" should say rows, since V ∈ R^{N×d} and V_{:j} is defined as "the first jB_c rows of V".

p. 23 Proof of Theorem 1
Let Vj:j+1 be the slice of V from column jBc to column (j + 1)Bc − 1
p. 22 Proof of Theorem 1
the first jBc rows of V
p. 22 Proof of Theorem 1
the the ﬁrst 𝑗𝐵𝑐rows of V
p. 22 Proof of Theorem 1
before the any iteration

Judge's note: p23 shows 'column' for the V slice. p22 defines V_{:j} by rows, and in the chain V_{j:j+1} is stacked under V_{:j} and multiplied by an N×B_c block, so it must be B_c rows. The 'the the' and 'the any' typos raised by reviewers 3 and 4 are covered separately in C039 and C041.

Second check: another model agreed

Report a problem with this finding

Wording Proof of Theorem 1, third line of the O^(j+1) chain: "diag(ℓ^(j)) e^{m^(j)−m^(j+1)} diag(ℓ^(j)) exp(S_{:,:j} − m^(j)) V_{:j}". The second diag(ℓ^(j)) lacks the inverse.

p. 23 Proof of Theorem 1, third line of O^(j+1) derivation
diag(ℓ(j)) exp(S:,:j−m(j))V:j
p. 23 Proof of Theorem 1, third line of the O^(j+1) derivation chain
diag(ℓ(j))e^{m(j)−m(j+1)} diag(ℓ(j)) exp(S:,:j − m(j))V:j

Only the authors can say what is right here.

Judge's note: On the p23 image and in the layout text, the other diag(ℓ^(j+1)) factors carry a −1 but the second diag(ℓ^(j)) on line 3 does not. My own working: P_{:,:j} = softmax(S_{:,:j}) = diag(ℓ^(j))^{-1} exp(S_{:,:j} − m^(j)), so line 2 should become diag(ℓ^(j)) e^{m^(j)−m^(j+1)} diag(ℓ^(j))^{-1} exp(S_{:,:j} − m^(j)) V_{:j} = e^{−m^(j+1)} exp(S_{:,:j}) V_{:j}, which is line 4. As printed, line 3 equals diag(ℓ^(j))^2 e^{−m^(j+1)} exp(S_{:,:j}) V_{:j} and does not lead to line 4. Example: S = [0, 0] gives ℓ = 2, and line 3 then gives [4, 4] instead of [1, 1]. The step is faulty as printed, but Theorem 1 and Algorithm 1 are unaffected once the inverse is restored.

Second check: another model agreed

Report a problem with this finding

37 minor (typos and the like; most checked by the judge only)

Reference or label Theorem 5: "Standard attention (Algorithm 0) backward pass requires Θ(Nd + N^2) HBM accesses" points to the forward Algorithm 0, while the standard backward pass is given as Algorithm 3 (p20).

p. 21 Theorem 5
Standard attention (Algorithm 0) backward pass requires Θ(Nd + N2) HBM accesses
p. 20 Algorithm 3 heading
Algorithm 3 Standard Attention Backward Pass

Judge's note: p20 shows "Algorithm 3 Standard Attention Backward Pass". p21 shows that Theorem 5 cites Algorithm 0, the standard forward implementation, alongside "FlashAttention backward pass (Algorithm 4)". The parenthetical can be read as qualifying 'Standard attention', so this is defensible. Algorithm 3 would be the more precise pointer.

Second check: another model agreed

Report a problem with this finding

Reference or label Proof of Theorem 1: "when we update the statistics in the inner loop (Algorithm 1 line 10)". The update m^new = max(m_i, m̃_ij), ℓ^new = ... is line 11.

p. 22 Proof of Theorem 1, last line
when we update the statistics in the inner loop (Algorithm 1 line 10)

Judge's note: In the Algorithm 1 text, line 10 computes the block-local m̃_ij, P̃_ij and ℓ̃_ij, and line 11 computes m_i^new and ℓ_i^new, which is the update described on p22-23. Line 10 supplies inputs to the update, so the pointer is off by one but partly defensible.

Second check: another model agreed

Report a problem with this finding

Wording Abstract/intro say block-sparse FlashAttention is "faster than any existing approximate attention method" with no qualifier, but in the forward-only Tables 15 and 18 Linformer is faster at sequence lengths 128–1024.

p. 1 Abstract
an approximate attention algorithm that is faster than any existing approximate attention method
p. 3 Introduction, 'Benchmarking Attention' bullet
block-sparse FlashAttention is faster than all existing approximate attention methods that we know of
p. 34 Table 18 (forward pass), Linformer vs Block-Sparse FlashAttention, 128–1024
0.46 0.36 0.45 0.50

Only the authors can say what is right here.

Judge's note: Table 18 (p.34) forward pass: Linformer 0.46/0.36/0.45/0.50 vs Block-Sparse FlashAttention 0.56/0.52/0.63/0.65 ms at 128–1024. The paper itself puts Linformer in bold at 1024. Table 15 (p.33): Linformer 0.47/0.37/0.49/0.52 vs BSFA 0.54/0.51/0.68/0.61. Section 4.3 (p.10) scopes the claim to forward+backward runtime (Fig. 3), where the claim holds (e.g. Table 20). The abstract (p.1) and intro bullet (p.3) give no such scope. This is a wording issue, not a data error.

Second check: another model agreed

Report a problem with this finding

Wording Section 5 says "Our IO-aware implementation of attention is optimal within constants for computing attention on a single GPU", but Proposition 3 only rules out o(N²d²M⁻¹) HBM accesses holding for all M in [d, Nd].

p. 10 Section 5, Multi-GPU IO-Aware Methods
Our IO-aware implementation of attention is optimal within constants for computing attention on a single GPU.
p. 6 Proposition 3
There does not exist an algorithm to compute exact attention with o(N2d2M−1) HBM accesses for all M in the range [d, Nd].
p. 1 Abstract
optimal for a range of SRAM sizes

Only the authors can say what is right here.

Judge's note: I read Proposition 3 and the sentences after it on page 6. The proof uses only the case M = Θ(Nd), and the authors say they leave lower bounds in terms of M as future work. The abstract limits the claim to 'a range of SRAM sizes'. The unqualified 'optimal within constants' says more than was proved.

Second check: another model agreed

Report a problem with this finding

Wording Intro says block-sparse FlashAttention is "2-4× faster than even FlashAttention" without a sequence length or sparsity; in Table 11 the ratio runs from about 0.93× (seq 256) to about 146× (seq 64K).

p. 2 Introduction, paragraph on block-sparse FlashAttention
a sparse attention algorithm that is 2-4× faster than even FlashAttention
p. 32 Table 11 (fwd+bwd, dropout and masking), FlashAttention vs Block-Sparse FlashAttention rows
9341.30

Judge's note: Table 11 (p.32), FA vs BSFA: 0.43/0.44, 0.41/0.44, 0.95/0.45, 2.55/0.89, 9.56/1.95, 37.49/4.12, 9341.30/64.11. The 2–4× range may come from Fig. 2 right (seq 4K, varying sparsity, p.6), but the intro sentence (p.2) gives no setting. The text is unqualified rather than wrong.

Second check: another model agreed

Report a problem with this finding

Wording Intro: "GPT2 (seq. length 1K) 3× faster than baseline implementations from HuggingFace [87] and Megatron-LM [77]", but Table 2 gives about 1.7× over Megatron-LM; the 3× applies to HuggingFace only.

p. 3 Introduction, 'Faster Model Training' bullet
GPT2 (seq. length 1K) 3× faster than baseline implementations from HuggingFace [87] and Megatron-LM [77]
p. 8 Table 2
4.7 days (2.0×)

Only the authors can say what is right here.

Judge's note: Table 2 (p.8): FlashAttention 2.7/6.9 days, Megatron 4.7/11.5, HuggingFace 9.5/21.0. Section 4.1 and the Table 2 caption themselves say 1.7× vs Megatron-LM. Reviewer note 2 is not a defect: the 2.0× in Table 2 is Megatron's speedup over HuggingFace, a different quantity.

Second check: another model agreed

Report a problem with this finding

Wording Algorithm 5 uses the same italic M for 'on-chip SRAM of size M' (and B_c = ⌈M/4d⌉) and for the 'block sparsity mask M ∈ {0, 1}^{N/Br×N/Bc}..' tested as M_ij in line 8. The Require line also ends with a doubled period.

p. 25 Algorithm 5 Require line and line 8
block sparsity mask 𝑀∈{0, 1}𝑁/𝐵𝑟×𝑁/𝐵𝑐..
p. 25 Algorithm 5, Require line and line 8
block sparsity mask M ∈ {0, 1}N/Br ×N/Bc
p. 6 Section 3.3, block form of the mask
for some M ∈ {0, 1}𝑁/𝐵𝑟×𝑁/𝐵𝑐
p. 25 Algorithm 5 Require
𝑁/𝐵𝑟×𝑁/𝐵𝑐..

Judge's note: Checked the page 25 image and crop C046-a. In the Require line, italic M is the SRAM size and is used in M/4d; in the same line the mask is printed as the same italic M, and line 8 then tests 'if M_ij ≠ 0'. In Section 3.3 (page 6) the text layer shows the mask in bold, so the clash is confined to Algorithm 5. The subscript M_ij makes the meaning recoverable, so this is ambiguity rather than an error. The doubled '..' at the end of the Require line is visible on the page.

Second check: another model agreed

Report a problem with this finding

Wording Text and Table 2 caption say "up to 3× speed up compared to Huggingface", but Table 2 lists "2.7 days (3.5×)" for GPT-2 small.

p. 8 Table 2 caption
achieve up to 3× speed up compared to Huggingface
p. 8 Table 2, GPT-2 small - FlashAttention row
2.7 days (3.5×)
p. 7 Section 4.1 GPT-2
Table 2 shows up to 3× end-to-end speedup compared to Huggingface
p. 7 Section 4.1
up to 3× end- to-end speedup compared to Huggingface

Only the authors can say what is right here.

Judge's note: Table 2 (p.8): 9.5/2.7 = 3.52 (printed 3.5×) for small and 21.0/6.9 = 3.04 for medium. 'Up to 3×' understates the maximum the table shows. The data are fine and the wording is conservative.

Second check: another model agreed

Report a problem with this finding

Typo Figure 1 caption: "resulting in an 7.6× speedup" should read "a 7.6× speedup".

p. 2 Figure 1 caption
resulting in an 7.6×

Judge's note: Seen in the Figure 1 caption on p.2. Cosmetic article error.

Second check: none (judge only)

Report a problem with this finding

Typo Algorithm lines list "O_i, . . . , O_Tr" and "ℓ_i, . . . , ℓ_Tr" (and "dO_i, ..., dO_Tr") where index 1 is meant, as in m_1, ..., m_Tr on the same line.

p. 5 Algorithm 1 line 4
Divide O into Tr blocks Oi, . . . , OTr
p. 20 Algorithm 2 line 5
blocks ℓi, . . . , ℓTr
p. 21 Algorithm 4 line 4
dOi, . . . , dOTr
p. 25 Algorithm 5 line 4
Divide O into Tr blocks Oi, . . . , OTr

Judge's note: Seen in Algorithm 1 line 4 (p.5) and Algorithm 5 line 4 (p.25). The text layer shows the same in Algorithm 2 line 5 and Algorithm 4 line 4.

Second check: none (judge only)

Report a problem with this finding

Typo "all implementations (that we know off)" should be "that we know of".

p. 5 Section 3.1 Recomputation
all implementations (that we know off)
p. 5 Recomputation paragraph
that we know oﬀ

Judge's note: Seen in the Recomputation paragraph on p.5.

Second check: none (judge only)

Report a problem with this finding

Typo "divide K, V in to T_c = ... blocks" should be "into" (Algorithm 1 line 3, repeated in later algorithms).

p. 5 Algorithm 1 line 3
divide K, V in to

Judge's note: Seen in Algorithm 1 line 3 (p.5) and Algorithm 5 line 3 (p.25).

Second check: none (judge only)

Report a problem with this finding

Typo "(S ⊙ 1_M̃)_kl = S_kl if M̃_kl = 1 and −∞ if M_kl = 0": the second condition uses the block mask M instead of M̃.

p. 6 Section 3.3, definition below equation
and −∞ if M𝑘𝑙= 0
p. 6 Section 3.3 equation text
−∞if M𝑘𝑙= 0

Judge's note: The crop of p.6 clearly shows a tilde on the first M and none on the second. M is defined as the N/B_r × N/B_c block mask, so M̃_kl is meant.

Second check: none (judge only)

Report a problem with this finding

Typo The word 'that' is repeated in "Table 4 shows that that GPT-2 with FlashAttention" (Section 4.2).

p. 8 Section 4.2 Language Modeling with Long Context
Table 4 shows that that GPT-2
p. 8 Section 4.2
Table 4 shows that that GPT-2 with FlashAttention

Judge's note: The page 8 image shows 'that that'.

Second check: none (judge only)

Report a problem with this finding

Typo A word is missing in "FlashAttention achieves up 2.4× speed-up"; it should read 'up to 2.4×'.

p. 8 Section 4.1 Long-range Arena
achieves up 2.4× speed-up

Judge's note: The page 8 image, in the Long-range Arena paragraph, shows 'achieves up 2.4× speed-up'.

Second check: none (judge only)

Report a problem with this finding

Typo A space is missing after the citation in "Tay et al. [80]and Xiong et al. [90]".

p. 8 Section 4.1 Long-range Arena
Tay et al. [80]and Xiong et al. [90]

Judge's note: The page 8 image shows '[80]and' with no space.

Second check: none (judge only)

Report a problem with this finding

Typo 'violated' is misspelled in "that were allegedly violaged" (Section 4.2, Long Document Classification).

p. 9 Section 4.2 Long Document Classification
that were allegedly violaged
p. 9 Section 4.2, long document classification
allegedly violaged
p. 9
that were allegedly violaged.

Judge's note: The PDF's text layer, taken from the PDF's own text, reads 'violaged'. At the crop's resolution the image fits that reading, though 'g' and 't' are hard to tell apart there.

Second check: none (judge only)

Report a problem with this finding

Typo Reference [1] prints the second author's name in a broken inverted form: "Alok Aggarwal and S Vitter, Jeffrey."

p. 11 Reference [1]
Alok Aggarwal and S Vitter, Jeffrey.

Only the authors can say what is right here.

Judge's note: Rests on outside knowledge: the author is Jeffrey S. Vitter. The page 11 image shows 'S Vitter, Jeffrey', which mixes inverted and natural name order and does not match the first-last order of the other entries.

Second check: none (judge only)

Report a problem with this finding

Typo Reference [71] lists the same author twice: "Raghu Ramakrishnan, Johannes Gehrke, and Johannes Gehrke."

p. 15 Reference [71]
Raghu Ramakrishnan, Johannes Gehrke, and Johannes Gehrke.
p. 15 Reference [71]
Raghu Ramakrishnan, Johannes Gehrke, and Johannes Gehrke
p. 11 Reference [1]
Alok Aggarwal and S Vitter, Jeﬀrey

Judge's note: The page 15 image shows the duplicated name. This verdict covers [71] only; the reference [1] name problem in the second note is covered by C025.

Second check: none (judge only)

Report a problem with this finding

Typo Reference [32] reverses one author's name: "Aaron Gokaslan, Vanya Cohen, Pavlick Ellie, and Stefanie Tellex."

p. 12 Reference [32]
Pavlick Ellie

Judge's note: The page 12 image shows 'Pavlick Ellie'. The other names in the entry are first-last, so the order is inconsistent.

Second check: none (judge only)

Report a problem with this finding

Typo The title in reference [52] misspells BERT: "Albert: A lite BEDRT for self-supervised learning of language representations."

p. 14 Reference [52]
Albert: A lite BEDRT for self-supervised learning
p. 14 Reference [52]
Albert: A lite BEDRT for self-supervised
p. 14
Albert: A lite BEDRT for self-supervised learning of language representations.

Judge's note: The page 14 image shows 'BEDRT'.

Second check: none (judge only)

Report a problem with this finding

Typo The title in reference [90] misspells Nyström: "Nyströmformer: A nystöm-based algorithm".

p. 16 Reference [90]
A nystöm-based algorithm

Judge's note: The page 16 image shows 'nystöm', while the same entry spells 'Nyströmformer' correctly.

Second check: none (judge only)

Report a problem with this finding

Typo Two typos in the appendices: "Out block-sparse FlashAttention" for 'Our' (Appendix A) and "wall-block runtime" for 'wall-clock' (Appendix D.2).

p. 17 Appendix A Sparse Training
Out block-sparse FlashAttention can also be seen
p. 17 Appendix A, Sparse Training
Out block-sparse FlashAttention can also be seen as a ﬁxed lottery
p. 17 Appendix A, Sparse Training
Out block-sparse FlashAttention
p. 26 D.2 Sparse MLP layers
wall-block runtime

Judge's note: The page 17 image shows 'Out block-sparse'. The page 26 image shows 'wall-block runtime', and page 17 itself uses 'wall-clock'.

Second check: none (judge only)

Report a problem with this finding

Typo Appendix A: "dense unconstrained matrix multiply has very optimize implementation" should read "very optimized implementations".

p. 17 Appendix A Efficient ML Models
has very optimize implementation

Judge's note: The crop of p17 shows the phrase exactly as quoted. It is a wrong word form.

Second check: none (judge only)

Report a problem with this finding

Typo Appendix B.1/B.2 use lowercase n for sequence length ("computed with O(n) extra memory", "dO ∈ R^{n×d}"), while N is used everywhere else, including on the same page.

p. 18 Appendix B.1–B.2
dO ∈ R𝑛×𝑑
p. 18 Appendix B.1
computed with 𝑂(𝑛) extra memory

Judge's note: The p18 image shows O(n) in B.1 and dO ∈ R^{n×d} in B.2. Q, K, V ∈ R^{N×d} and dQ, dK, dV ∈ R^{N×d} appear on the same page. This is a notation inconsistency.

Second check: none (judge only)

Report a problem with this finding

Typo Algorithm 4 line 13: "P_ij = diag(l_i)^{-1} exp(S^masked_ij − m_i)" uses l_i where line 10 and the Require line use ℓ_i.

p. 21 Algorithm 4 line 13
P𝑖𝑗= diag(𝑙𝑖)−1 exp(Smasked
p. 21 Algorithm 4 line 13
P𝑖𝑗= diag(𝑙𝑖)−1 exp

Judge's note: The p21 crop shows a plain italic l in line 13 and a script ℓ in line 10 of the same algorithm. This is a notation inconsistency.

Second check: none (judge only)

Report a problem with this finding

Typo Appendix B.5: "Rabe and Staats [66] focuses on the reducing the total memory footprint" has an extra 'the'.

p. 22 Appendix B.5
focuses on the reducing the total memory footprint

Judge's note: The p22 image shows the phrase as quoted.

Second check: none (judge only)

Report a problem with this finding

Typo Proof of Theorem 1: "V_{:j} ∈ R^{jB_c×d} the the first jB_c rows of V" has a duplicated word.

p. 22 Proof of Theorem 1
the the first jBc rows of V

Judge's note: The crop of p22 shows 'the the'.

Second check: none (judge only)

Report a problem with this finding

Typo Proof of Theorem 1: "Let m^j, ℓ^(j), O^(j) be the values" writes m^j where m^(j) is used everywhere else.

p. 22 Proof of Theorem 1
Let 𝑚𝑗, ℓ( 𝑗), O( 𝑗) be the values

Judge's note: The crop shows m^j without parentheses; the very next display uses m^(j).

Second check: none (judge only)

Report a problem with this finding

Typo Proof of Theorem 1: "(i.e., before the any iteration of the outer loop is executed)" has a stray 'the'.

p. 22 Proof of Theorem 1
before the any iteration

Judge's note: The p22 image and crop show the phrase as quoted.

Second check: none (judge only)

Report a problem with this finding

Typo Proof of Theorem 2: "reside in HBM, and the at the end of the algorithm the output O ∈ R^{N×d} is written to HBM" has a stray 'the'; the same phrase recurs in the proof of Theorem 5.

p. 23 Proof of Theorem 2
and the at the end of the algorithm

Judge's note: The p23 image shows 'and the at the end'. The proof of Theorem 5 in the text has the same wording.

Second check: none (judge only)

Report a problem with this finding

Typo Appendix D.2 Sparse MLP layers: 'improve their wall-block runtime' should read 'wall-clock runtime'.

p. 26 Appendix D.2 Sparse MLP layers
wall-block runtime

Judge's note: The crop of page 26 shows 'wall-block runtime'. Elsewhere the paper writes 'wall-clock' (e.g. E.2, 'wall-clock training time'), so this is a misspelling.

Second check: none (judge only)

Report a problem with this finding

Typo Appendix D.2 Kernel machine learning: 'As similar scenario happens in kernel machine learning' should read 'A similar scenario'.

p. 26 Appendix D.2 Kernel machine learning
As similar scenario happens in kernel machine learning
p. 26 D.2 Kernel machine learning
As similar scenario happens

Judge's note: The page 26 crop shows 'As similar scenario happens'. This is a wrong article.

Second check: none (judge only)

Report a problem with this finding

Typo Appendix E.2: 'In Fig. 4, we plot of the validation perplexity' contains an extra 'of'.

p. 26 E.2 GPT-2
we plot of the validation perplexity

Judge's note: The page 26 crop shows 'we plot of the validation perplexity throughout training'. The stray 'of' should be removed.

Second check: none (judge only)

Report a problem with this finding

Typo The Figure 7 plot title reads 'FlashAttention Speedup, GTX 3090', but the caption and text say 'RTX 3090'.

p. 29 Figure 7 title
FlashAttention Speedup, GTX 3090
p. 29 Figure 7 caption
on RTX 3090

Judge's note: On the page 29 image, the plot title reads 'GTX 3090'. The caption says 'on RTX 3090' and the paragraph headed 'RTX 3090' refers to an RTX 3090 GPU. The label inside the figure is inconsistent with both.

Second check: none (judge only)

Report a problem with this finding

Typo Appendix E.6 Baselines: 'Block-Sparse Attention form OpenAI [11]' should read 'from OpenAI'.

p. 30 Appendix E.6 Baselines
Block-Sparse Attention form OpenAI [11]
p. 30 E.6 Baselines
Block-Sparse Attention form OpenAI

Judge's note: The page 30 image shows 'Block-Sparse Attention form OpenAI [11]'. This is a typo.

Second check: none (judge only)

Report a problem with this finding

Typo Appendix E.6: 'Longformer and BigBird do not support sequence lengths longer than 8092' should read 8192, the tested length that the tables report for these methods.

p. 31 Appendix E.6, paragraph after Table 9
support sequence lengths longer than 8092
p. 31 Setup exceptions
longer than 8092
p. 32 Table 10, Longformer/BigBird rows
25.95
p. 31 Table 9 columns
8192

Judge's note: The page 31 image shows '8092'. The tested lengths are powers of two (128–65536). Longformer and BigBird have entries at 8192 in Table 10 (25.95, 27.45), Table 11 (60.44, 63.28) and Table 12, and none beyond. 8092 is evidently a transposition of 8192.

Second check: none (judge only)

Report a problem with this finding

Something wrong with this page, or a finding we missed?