An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
2020 · paper · AI and machine learning
Shows that a standard Transformer applied directly to sequences of image patches can classify images well without convolutional networks. When pre-trained on large datasets, the Vision Transformer reportedly matches or beats leading convolutional networks, reaching 88.55% on ImageNet, with less training compute.
Checked: PDF from arxiv.org, fetched 4 Oct 2026. Page numbers are those of that PDF.
6 findings
Substance Table 6 exaFLOPs for R50x1+ViT-L/16 are "859" at 7 epochs and "1668" at 14 epochs, which is not a doubling, unlike the other models trained at both lengths.
- p. 15 Table 6, R50x1+ViT-L/16 rows, exaFLOPs column
859
- p. 15 Table 6, R50x1+ViT-L/16 14-epoch row
1668
Only the authors can say what is right here.
Judge's note: Page 15 image: ViT-L/16 goes 783 → 1567 and ResNet152x2 goes 563 → 1126, both doubling within rounding. R50x1+ViT-L/16 goes 859 → 1668, where doubling 859 gives 1718, a gap of 50. The other hybrids cost about 50 more than their ViT counterparts (106−55, 274−224, 246−196), whereas 859−783 = 76 and 1668/2 = 834 would give 51. This suggests 859 is the outlier, but the paper states no rule that compute scales exactly with epochs, so it is a query for the authors.
Second check: another model agreed
Report a problem with this finding
Reference or label Figure 6 caption (attention maps from the output token to the input space) says "See Appendix D.7 for details", but D.7 is "Attention Distance" (Figure 11); the attention-map method is in D.8 "Attention Maps", which names "Figures 6 and 14".
- p. 8 Figure 6 caption
See Appendix D.7 for details.
- p. 20 Appendix D.8 Attention Maps
To compute maps of the attention from the output token to the input space (Figures 6 and 14), we used Attention Rollout
- p. 20 Appendix D.8 Attention Maps
To compute maps of the attention from the output token to the input space (Figures 6 and 14)
Judge's note: Figure 6 caption read on page 8; D.7 and D.8 read on page 20. D.7 covers attention distance only and cites Figure 11. D.8 describes Attention Rollout and cites Figure 6 explicitly. The pointer should be to Appendix D.8.
Second check: another model agreed
Report a problem with this finding
Reference or label Section 4.5: "This highly localized attention is less pronounced in hybrid models that apply a ResNet before the Transformer (Figure 7, right)", but the right panel of Figure 7 shows only ViT-L/16; the hybrid (R50x1 + ViT-L/16) panel is in Figure 11 (right).
- p. 8 Section 4.5, last paragraph
less pronounced in hybrid models that apply a ResNet before the Transformer (Figure 7, right)
- p. 9 Figure 7, right panel title
ViT-L/16
- p. 18 Figure 11, right panel title
R50x1 + ViT-L/16
- p. 9 Figure 7 caption
Right: Size of attended area by head and network depth.
Judge's note: The right panel of Figure 7 on page 9 is titled "ViT-L/16" and contains no hybrid model; its caption mentions none. Figure 11 on page 18 has the ViT-L/16 and "R50x1 + ViT-L/16" panels side by side. The pointer does not lead to the hybrid comparison; the target should be Figure 11.
Second check: another model agreed
Report a problem with this finding
Reference or label Figure 13 caption labels both panels left: "number of FLOPs (left) and inference time (left)"; the second should be "(right)".
- p. 20 Figure 13 caption
number of FLOPs (left) and inference time (left)
- p. 20
number of FLOPs (left) and inference time (left).
Judge's note: Page 20 image: the left panel's x-axis is 'Total compute [exaFLOPs]' and the right panel's is 'Peak inference speed [img/sec/core]'. The body text below the figure correctly uses 'Figure 13, left' for compute and 'Figure 13, right' for speed.
Second check: another model agreed
Report a problem with this finding
Wording Section 4.2 says "The smaller ViT-L/16 model pre-trained on JFT-300M outperforms BiT-L (which is pre-trained on the same dataset) on all tasks" and the Table 2 caption says JFT ViT models "outperform ResNet-based baselines on all datasets", but Table 2 shows ViT-L/16 (JFT) tied with BiT-L on ImageNet ReaL (90.54 ± 0.03 vs 90.54) and 0.01 lower on VTAB (76.28 ± 0.46 vs 76.29 ± 1.70).
- p. 5 Section 4.2, last paragraph
The smaller ViT-L/16 model pre-trained on JFT-300M outperforms BiT-L (which is pre-trained on the same dataset) on all tasks
- p. 6 Table 2, rows ImageNet ReaL and VTAB (19 tasks), columns ViT-L/16 (JFT) and BiT-L
76.28 ± 0.46
- p. 6 Table 2, BiT-L column, VTAB row
76.29 ± 1.70
- p. 5 Section 4.2, last paragraph
outperforms BiT-L (which is pre-trained on the same dataset) on all tasks
Only the authors can say what is right here.
Judge's note: Checked Table 2 on the page 6 image. ViT-L/16 (JFT) is ahead of BiT-L on ImageNet, CIFAR-10, CIFAR-100, Pets and Flowers; it ties on ReaL and is marginally below on VTAB. The gaps are well inside the printed standard deviations, so this is an overstatement in wording ("all"), not a wrong number. No statement elsewhere qualifies the claim. Rated S3.
Second check: another model agreed
Report a problem with this finding
Wording Appendix B.1 says "Hybrid models are trained with the exact setup as their ViT counterparts." but Table 3 gives R50+ViT-L/32 a base LR of "2 · 10−4" against "6 · 10−4" for ViT-L/32.
- p. 13 Appendix B.1
Hybrid models are trained with the exact setup as their ViT counterparts.
- p. 13 Table 3, row R50+ViT-L/32
2 · 10−4
- p. 13 Table 3, row ViT-L/32
6 · 10−4
Only the authors can say what is right here.
Judge's note: Page 13 image, Table 3: ViT-L/32 is 6·10^-4 and R50+ViT-L/32 is 2·10^-4. The other hybrids match their same-named ViT rows (B/{16,32} at 8·10^-4, L/16 at 4·10^-4). Section 4.4 notes that the hybrid suffix is the downsampling ratio rather than the patch size, but that does not explain why only L/32 differs. Either the table entry or the 'exact setup' sentence is off; only the authors can say which.
Second check: another model agreed
Report a problem with this finding
16 minor (typos and the like; most checked by the judge only)
Reference or label Huang et al. reference reads "Ccnet: Criss-cross attention for semantic segmentation. In ICCV, 2020."; the venue and year pairing looks inconsistent.
- p. 10 Reference list, Huang et al.
Ccnet: Criss-cross attention for semantic segmentation. In ICCV, 2020.
Only the authors can say what is right here.
Judge's note: Rests on outside knowledge: ICCV is held in odd years only and CCNet appeared at ICCV 2019. The page 10 image confirms the entry prints 'In ICCV, 2020', and the in-text citation in Appendix D.6 (page 19) is 'Huang et al., 2020'. Every other ICCV entry in the list is dated 2017 or 2019, which fits the claim but does not prove it from the files.
Second check: another model agreed
Report a problem with this finding
Wording Section 4.2 describes "S4L – supervised plus semi-supervised learning on ImageNet (Zhai et al., 2019a)", whereas the cited reference is titled "S4L: Self-Supervised Semi-Supervised Learning".
- p. 6 Section 4.2, paragraph on Figure 2
S4L – supervised plus semi-supervised learning on ImageNet (Zhai et al., 2019a)
- p. 12 Reference list, Zhai et al. 2019a
S4L: Self-Supervised Semi-Supervised Learning
Only the authors can say what is right here.
Judge's note: Body text read on page 6; the reference entry read on the page 12 crop. The description does not match the title of the paper's own reference entry, and "supervised" looks like a dropped "self-". The authors might mean a loose paraphrase, so this is a query.
Second check: another model agreed
Report a problem with this finding
Typo Djolonga et al. reference prints the co-author's name as "Sylvan Gelly"; the author list and the Kolesnikov and Tschannen entries print "Sylvain Gelly". Separately, Section 4.4 prints "R50x2 R101x1" with no comma in the model list.
- p. 10 Reference list, Djolonga et al. 2020
Sylvan Gelly
- p. 10 Reference list, Djolonga et al. 2020
Sylvan
- p. 1 Author list
Sylvain Gelly
- p. 8 Section 4.4
R50x2 R101x1
Judge's note: Page 10 image shows 'Dan Moldovan, Sylvan / Gelly' in the Djolonga entry. Page 12 image (Tschannen entry) shows 'Sylvain Gelly'. The page 1 author line and the Kolesnikov entry also read 'Sylvain Gelly', and the Section 4.4 line reads 'R50x2 R101x1, R152x1'; these three were checked in the layout text only, not on the page images. The misspelling is internally inconsistent, so no outside knowledge is needed.
Second check: none (judge only)
Report a problem with this finding
Typo Number disagreement in Section 4.1: "We transfer the models trained on these dataset to several benchmark tasks" should read "these datasets".
- p. 4 4.1 Datasets
models trained on these dataset to several benchmark tasks
Judge's note: Read on the page 4 crop; "these" ends one line and "dataset" starts the next, with no plural s. Not an extraction artefact.
Second check: none (judge only)
Report a problem with this finding
Typo Misspellings: Section 4.4 "the total dowsampling ratio in the ResNet backbone" (downsampling); D.4 "the stem of them model" (the model); D.6 "verses the pretraining compute" (versus); D.6 "we have a a row-self-attention" (duplicated "a").
- p. 8 Section 4.4, first paragraph
the total dowsampling ratio in the ResNet backbone
- p. 8 Section 4.4, first paragraph
total dowsampling ratio
- p. 18 D.4
the stem of them model
- p. 19 D.6
verses the pretraining compute
Judge's note: All four are visible in the page images: "dowsampling" on page 8, "them model" on page 18, and "verses" and "a a" on the page 19 crops.
Second check: none (judge only)
Report a problem with this finding
Typo Missing comma in the Section 4.4 model list: "7 ResNets, R50x1, R50x2 R101x1, R152x1, R152x2".
- p. 8 Section 4.4, first paragraph
R50x2 R101x1
Judge's note: On the page 8 image the line begins "R50x2 R101x1, R152x1, R152x2" with no comma between the first two names. The line break falls before R50x2, so it does not account for the missing comma.
Second check: none (judge only)
Report a problem with this finding
Typo Duplicated word: Section 4.5 "cipal components of the the learned embedding filters" and Appendix B.1 "except for the the qkv-projections".
- p. 8 Section 4.5, first paragraph
components of the the learned embedding filters
- p. 8 Section 4.5
principal components of the the learned embedding
- p. 8 Section 4.5
top principal components of the the learned embedding filters
- p. 13 B.1
except for the the qkv-projections
Judge's note: Both instances are visible mid-line in the page images (page 8, Section 4.5; page 13, B.1). Neither is a line-break artefact.
Second check: code check
Report a problem with this finding
Typo Missing article in the Conclusion: "but there is still large gap between self-supervised and large-scale supervised pre-training" (should be "still a large gap").
- p. 9 Section 5, second paragraph
but there is still large gap between
Judge's note: Read on the page 9 image, Section 5, second paragraph; the article is absent.
Second check: none (judge only)
Report a problem with this finding
Typo Duplicated word in Appendix B.1: "except for the the qkv-projections".
- p. 13 Appendix B.1
except for the the qkv-projections
- p. 13
except for the the qkv-projections and directly after adding
- p. 13
every dense layer except for the the qkv-projections and
Judge's note: Visible on the page 13 image within a single line, so it is not a line-break or extraction artefact.
Second check: code check
Report a problem with this finding
Typo Appendix D.4 prints "based on the coordinate on the path in the input"; "path" should be "patch".
- p. 17 Appendix D.4, third bullet
based on the coordinate on the path in the input
Judge's note: Page 17 image, third bullet. The sentence ends 'to get the final positional embedding for that patch', so 'path' is a slip for 'patch'.
Second check: none (judge only)
Report a problem with this finding
Typo Appendix D.4 relative-embedding bullet prints "encode the spatial information as instead of their absolute position" (stray "as") and "we define the relative distance all possible pairs of patches" (missing "between" or "of").
- p. 17 Appendix D.4, fourth bullet
encode the spatial information as instead of their absolute position
- p. 17 Appendix D.4, fourth bullet
we define the relative distance all possible pairs of patches
Judge's note: Both phrases are visible on the page 17 image in the fourth bullet.
Second check: none (judge only)
Report a problem with this finding
Typo Appendix D.4 prints "the stem of them model"; "them" should be "the".
- p. 18 Appendix D.4, continuation after Figure 10
the stem of them model
Judge's note: Visible on the page 18 image, first line under Figure 10.
Second check: none (judge only)
Report a problem with this finding
Typo Duplicated article in Appendix D.6: "we have a a row-self-attention plus an MLP".
- p. 19 Appendix D.6, second paragraph
we have a a row-self-attention plus an MLP
- p. 19 D.6
we have a a row-self-attention
Judge's note: Visible on the page 19 image within a single line.
Second check: none (judge only)
Report a problem with this finding
Typo Appendix D.6 prints "Figure 13, present the performance of Axial ResNet" (should be "Figure 13 presents") and "verses the pretraining compute" (should be "versus"); the same sentence has "example per seconds".
- p. 19 Appendix D.6, third paragraph
Figure 13, present the performance of Axial ResNet
- p. 19 Appendix D.6, third paragraph
verses the pretraining compute
Judge's note: All three are visible on the page 19 image in the third paragraph of D.6. The 'a a' slip in the same passage is handled under C027.
Second check: none (judge only)
Report a problem with this finding
Typo Doubled full stop around the footnote mark in Appendix D.6: "AxialResNet as a baseline model.3."
- p. 19 Appendix D.6, end of first paragraph
AxialResNet as a baseline model.3.
- p. 19 D.6
as a baseline model.3.
Judge's note: Page 19 image shows 'baseline model.' followed by superscript 3 and a second full stop.
Second check: none (judge only)
Report a problem with this finding
Typo Appendix D.6 (PDF page 20) prints "there is a extra MLP per Axial-ViT block"; the article should be "an extra MLP".
- p. 20 Appendix D.6, paragraph after Figure 13
there is a extra MLP per Axial-ViT block
Judge's note: Opened the crop of page 20: the paragraph after the Figure 13 caption reads "there is a extra MLP per Axial-ViT block" in the rendered page, and the text layer (extract/main.txt line 1794) matches, so it is not an extraction artefact. Plain article error, cosmetic.
Second check: none (judge only)