Attention Is All You Need
NeurIPS, 2017 · paper · AI and machine learning
Proposes the Transformer, a network architecture for sequence translation based solely on attention, without recurrence or convolutions. It reports 28.4 BLEU on English-to-German and 41.8 on English-to-French translation, with faster training, and good results on parsing.
Checked: PDF from arxiv.org, fetched 4 Oct 2026. Page numbers are those of that PDF.
2 findings
Substance Section 6.1: "our big model achieves a BLEU score of 41.0" on WMT 2014 EN-FR, but Table 2 (Transformer (big), EN-FR) and the abstract give 41.8.
- p. 8 Section 6.1, second paragraph
our big model achieves a BLEU score of 41.0
- p. 8 Table 2, Transformer (big), EN-FR
41.8
- p. 1 Abstract
new single-model state-of-the-art BLEU score of 41.8
- p. 1 Abstract
BLEU score of 41.8
Judge's note: The page 8 image shows 41.0 in the text and 41.8 in Table 2. The abstract crop on page 1 shows 41.8. The text, the table and the abstract all refer to the same big model on the same newstest2014 EN-FR task. Nothing in the paper explains the difference.
Second check: another model agreed
Report a problem with this finding
Wording Section 6.3 says the Transformer yields "better results than all previously reported models with the exception of the Recurrent Neural Network Grammar [8]", yet Table 4 lists Luong et al. (2015) [23] multi-task at 93.0, above the Transformer's best of 92.7.
- p. 10 Section 6.3 text after Table 4
yielding better results than all previously reported models with the exception of the Recurrent Neural Network Grammar [8]
- p. 10 Table 4, Luong et al. (2015) row
93.0
- p. 10 Table 4
Luong et al. (2015) [23] multi-task 93.0
Only the authors can say what is right here.
Judge's note: The page 10 image of Table 4 shows Transformer (4 layers) semi-supervised at 92.7, Luong et al. (2015) [23] multi-task at 93.0 and Dyer et al. [8] generative at 93.3. [23] is not the RNNG paper, so the claim of a single exception overstates the table, unless it is meant to be restricted to a training regime, which the sentence does not say. In the WSJ-only block, Dyer et al. (2016) [8] at 91.7 is the RNNG paper, so it is covered by the stated exception.
Second check: another model agreed
Report a problem with this finding
11 minor (typos and the like; most checked by the judge only)
Wording Section 6.3: "to select the dropout, both attention and residual (section 5.4)" points to Section 5.4, which describes only residual and embedding-sum dropout and never mentions attention dropout.
- p. 9 Section 6.3, last paragraph
to select the dropout, both attention and residual (section 5.4)
- p. 8 Section 5.4 Residual Dropout
We apply dropout [33] to the output of each sub-layer
Only the authors can say what is right here.
Judge's note: Page 8 shows that Section 5.4 has only "Residual Dropout" (dropout on sub-layer outputs and on embedding+positional-encoding sums) and "Label Smoothing". Attention dropout is not defined there, so the pointer does not support what it refers to. The authors need to say what "attention" dropout means.
Second check: another model agreed
Report a problem with this finding
Typo Page 1 permission notice reads "Provided proper attribution is provided, Google hereby grants permission…", which repeats "provided".
- p. 1 header
Provided proper attribution is provided
Judge's note: The crop of the page 1 header shows the red notice with this wording. It is a redundant repetition in boilerplate text, not an error of content, so it is cosmetic.
Second check: none (judge only)
Report a problem with this finding
Typo Section 3.1, Decoder: "This masking, combined with fact that the output embeddings are offset by one position" is missing the article "the" before "fact".
- p. 3 Section 3.1, Decoder paragraph
combined with fact that the output embeddings
- p. 3 Section 3.1, Decoder paragraph
combined with fact that the output embeddings are offset
- p. 3 3.1 Decoder
combined with fact that the output embeddings are offset by one position
Judge's note: The page 3 crop shows "combined with fact that". This is not an extraction artefact.
Second check: none (judge only)
Report a problem with this finding
Typo Section 4: "As side benefit, self-attention could yield more interpretable models" is missing the article "a" ("As a side benefit").
- p. 7 Section 4, last paragraph
As side benefit, self-attention could yield
Judge's note: The page 7 image shows "As side benefit".
Second check: none (judge only)
Report a problem with this finding
Typo Section 5.2: "For our big models,(described on the bottom line of table 3), step time was 1.0 seconds" has a stray comma before the parenthesis and a lowercase "table".
- p. 7 5.2
For our big models,(described on the
Judge's note: The page 7 image shows the comma immediately followed by "(" with no space. "Table" is capitalised elsewhere in the paper.
Second check: none (judge only)
Report a problem with this finding
Typo Table 2 sets both ConvS2S Ensemble "41.29" and Transformer (big) "41.8" in bold in the EN-FR column, while in the EN-DE column only 28.4 is bold (the previous best, 26.36, is not).
- p. 8 Table 2, ConvS2S Ensemble, EN-FR
41.29
Judge's note: The page 8 image confirms the bolding. The bold convention is not explained and is applied inconsistently between the two columns. This is a presentation point only; the data are unaffected.
Second check: none (judge only)
Report a problem with this finding
Typo Section 6.3 spells the parser "BerkleyParser" (p. 9) but "Berkeley-Parser" (p. 10).
- p. 9 Section 6.3, second paragraph
BerkleyParser corpora
- p. 10 Section 6.3, last paragraph
Berkeley-Parser [29]
- p. 9 Section 6.3, second paragraph
high-confidence and BerkleyParser corpora
- p. 10 Section 6.3, last paragraph
outperforms the Berkeley-Parser [29]
Judge's note: The page 9 image shows "BerkleyParser". The page 10 image shows "Berkeley-Parser" hyphenated across a line break. The name is misspelt in one place and inconsistent between the two.
Second check: none (judge only)
Report a problem with this finding
Typo Section 6.3: "the larger high-confidence and BerkleyParser corpora from with approximately 17M sentences [37]" is garbled ("from with").
- p. 9 Section 6.3, second paragraph
corpora from with approximately 17M sentences
- p. 9 6.3
BerkleyParser corpora from with approximately 17M sentences
Judge's note: The page 9 image shows "corpora from with approximately 17M sentences". Either a word is missing after "from" or one of the two words is surplus.
Second check: none (judge only)
Report a problem with this finding
Typo Table 4 labels [37] as "Vinyals & Kaiser el al. (2014)" (twice), with "el al." for "et al.", while reference [37] gives "Advances in Neural Information Processing Systems, 2015."
- p. 10 Table 4, rows 1 and 9
Vinyals & Kaiser el al. (2014) [37]
- p. 12 Reference [37]
Advances in Neural Information Processing Systems, 2015.
- p. 12 Reference [37]
Grammar as a foreign language. In Advances in Neural Information Processing Systems, 2015.
- p. 10 Table 4
Vinyals & Kaiser el al. (2014)
Judge's note: The page 10 image shows "el al. (2014)" in rows 1 and 9 of Table 4. The page 12 crop shows reference [37] dated 2015. Both defects are in the same label: the year conflicts with the reference list and "et al." is misspelt. The year mismatch could also be read as a citation-label (S2) issue.
Second check: none (judge only)
Report a problem with this finding
Typo Conclusion: "Making generation less sequential is another research goals of ours" has a number disagreement ("another … goals").
- p. 10 Conclusion, third paragraph
is another research goals of ours
- p. 10 Conclusion
another research goals of ours
Judge's note: The page 10 image shows "another research goals of ours".
Second check: none (judge only)
Report a problem with this finding
Typo Reference [37] gives a malformed, surname-only author list, "Vinyals & Kaiser, Koo, Petrov, Sutskever, and Hinton.", unlike every other entry, which gives full names.
- p. 12 Reference [37]
Vinyals & Kaiser, Koo, Petrov, Sutskever, and Hinton.
Judge's note: I opened the crop and checked the text layer. Entry [37] on page 12 reads exactly "Vinyals & Kaiser, Koo, Petrov, Sutskever, and Hinton. Grammar as a foreign language. In Advances in Neural Information Processing Systems, 2015." I searched the reference list for other entries that begin with a bare surname and found only this one; the neighbouring entries [35], [36] and [38] give full first names. The 'Vinyals & Kaiser' form matches the in-text label 'Vinyals & Kaiser el al. (2014) [37]' in Table 4, which suggests a citation label carried over into the bibliography. This is a cosmetic formatting inconsistency. Filling in the authors' first names would need outside bibliographic information, which these files do not contain.
Second check: none (judge only)