Emerging Properties in Self-Supervised Vision Transformers
2021 IEEE/CVF International Conference on Computer Vision (ICCV), 2021 · paper · AI and machine learning
Examines whether self-supervised training gives Vision Transformers distinctive properties. Features contain explicit segmentation information and work well with k-NN classifiers, reaching 78.3% on ImageNet. The authors propose DINO, a self-distillation method reaching 80.1% linear accuracy with ViT-Base.
Checked: PDF from arxiv.org, fetched 4 Oct 2026. Page numbers are those of that PDF.
7 findings
Substance Sec. 3.1: "can be interpreted as adding a bias term c to the teacher: g_t(x) ← g_t(x) + c", while Eq. (4) defines c as the running mean of teacher outputs and Algorithm 1 subtracts it: "t = softmax((t - C) / tpt, dim=1) # center + sharpen".
- p. 4 Section 3.1, Avoiding collapse
gt (x) ← gt (x) + c
- p. 4 Eq. (4)
c ← mc + (1 − m)
- p. 3 Algorithm 1, function H
t = softmax((t - C) / tpt, dim=1) # center + sharpen
- p. 4 Sec. 3.1, Avoiding collapse
can be interpreted as adding a bias term c to the teacher
Judge's note: Page 4 image shows a plus sign in g_t(x) ← g_t(x) + c, and Eq. (4) c ← mc + (1−m)(1/B)Σ g_θt(x_i). Page 3 Algorithm 1 updates C with the same running mean and applies (t - C). With c defined as the mean, the prose sign is opposite to the algorithm; centering requires subtraction. No statement in the text defines c as the negative mean.
Second check: another model agreed
Report a problem with this finding
Substance Table 5, row "CT [71] VLOG RN50 48.7 46.4 50.0": (J&F)m 48.7 is not the mean of Jm 46.4 and Fm 50.0 (which is 48.2); every other row agrees to within 0.1.
- p. 7 Table 5, row CT [71]
48.7
- p. 7 Table 5, Self-supervised, CT [71] row
CT [71] VLOG RN50 48.7 46.4 50.0
Only the authors can say what is right here.
Judge's note: Read from the Table 5 crop and page 7 image. Recomputed all nine rows: eight match the printed (J&F)m to within rounding of the underlying values (largest gap 0.1, STC), CT is off by 0.5. No footnote or text explains a different definition for this row. The row is quoted from prior work [71], so which of the three numbers is wrong (or whether the source is itself inconsistent) cannot be settled from the PDF; hence an author query. It is a baseline row and does not affect the DINO results or conclusions.
Second check: another model agreed
Report a problem with this finding
Substance Appendix D gives the threshold as "keep 80% of the mass" for ViT-S/16 Jaccard values (Random 22.0, Supervised 27.3, DINO 45.9). Section 4.2.2 and the Figure 4 caption report the same values on PASCAL VOC12 validation images with "keep 60% of the mass".
- p. 17 Appendix D, 'Self-attention maps from supervised versus self-supervised learning'
by thresholding the self-attention maps to keep 80% of the mass
- p. 8 Figure 4 caption
keep 60% of the mass
- p. 7 Section 4.2.2, Probing the self-attention map
obtained by thresholding the self-attention map to keep 60%
- p. 17 Appendix D, Self-attention maps from supervised versus self-supervised learning
by thresholding the self-attention maps to keep 80% of
Only the authors can say what is right here.
Judge's note: Page 7 (Sec. 4.2.2) and the Figure 4 caption on page 8 both say 60%. The Figure 4 table gives ViT-S/16 values 22.0 / 27.3 / 45.9. The Appendix D paragraph on page 17 says 80% and its table repeats exactly 22.0 / 27.3 / 45.9 for the same dataset and protocol. I searched the text and found nothing that explains a different threshold in the appendix. Identical values under two stated thresholds mean one statement is wrong, and only the authors can say which.
Second check: another model agreed
Report a problem with this finding
Substance Table 14 row 5 gives "MoCov2 INCE … 71.4" for MoCo-v2 (ViT-S/16, 300 epochs, no multi-crop, linear), while Table 7 row 8, Table 13 and the Appendix E table all give 71.6 for the same setting.
- p. 15 Table 14, row 5 (MoCov2, INCE, no multi-crop)
MoCov2 INCE
- p. 8 Table 7, row 8
8 MoCov2
- p. 14 Table 13, MoCo-v2 row, ViT-small Linear
71.6
- p. 18 Appendix E table, MoCo-v2, 2 x 224^2, linear
71.6
Only the authors can say what is right here.
Judge's note: Viewed pages 8, 14, 15 and 18. Table 7 row 8 prints 62.0 / 71.6, Table 13 prints 71.6 / 62.0, and the Appendix E table prints 62.0 / 71.6 under 2×224², but Table 14 row 5 prints 71.4. All four are described as ViT-S/16, 300 epochs, linear evaluation, and Table 13 says all numbers were run by the authors. I found no statement of a different run or setting. 71.4 is also the BYOL value in row 7 of Table 14, which points to a copy slip, and the multi-crop value 73.4 agrees between Table 14 and Appendix E. The majority value is 71.6, but only the authors can say which is right. The text's "2 − 4% (… 5 versus 6)" gives 2.0 with 71.4 and 1.8 with 71.6.
Second check: another model agreed · checked by hand
Report a problem with this finding
Reference or label Table 12 row "SimCLRv2 [16] RN50w4 375 63.0 74.4" cites reference [16], which is "Xinlei Chen and Kaiming He. Exploring simple siamese representation learning" (SimSiam), not a SimCLR paper.
- p. 14 Table 12, fifth row
SimCLRv2 [16]
- p. 11 Reference list, entry [16]
Xinlei Chen and Kaiming He. Exploring simple siamese
- p. 5 Table 2, comparison across architectures
SCLR [12]
- p. 11 Reference list, entry 16
Exploring simple siamese
Only the authors can say what is right here.
Judge's note: Viewed page 14 (Table 12) and page 11 (reference list). The other SimCLRv2 rows in Table 12 cite [13] (Chen et al., Big self-supervised models are strong semi-supervised learners), and [16] is the SimSiam paper, so the reference number is wrong. The intended reference is not certain from the files: it is [13] if the row is SimCLRv2, or [12] if the row is really SimCLR, as the reviewer's point about RN50w4 with 375M parameters under SCLR [12] in Table 2 suggests. The authors need to say which.
Second check: another model agreed
Report a problem with this finding
Wording Sec. 4.1: "it still leads to a significant reduction of running time, and larger memory usage"; smaller patches reduce throughput, so running time increases.
- p. 6 Section 4.1, Comparing across architectures
it still leads to a significant reduction of running time, and larger memory usage
- p. 4 Table 1, im/s column
ViT-S/8
Judge's note: Page 6 reads as quoted. Table 1 (page 4) and Table 2 (page 5) show throughput falling from 1007 to 180 im/s (ViT-S) and 312 to 63 im/s (ViT-B) from /16 to /8. Sec. 5.1 (page 8) says the gain "comes at the expense of throughput". The sentence is framed as a cost, so the intended meaning (reduced speed) is recoverable, but the literal wording states the opposite direction.
Second check: another model agreed
Report a problem with this finding
Wording "we observe that a temperature lower than 0.06 is required to avoid collapse", but the table directly below gives k-NN 68.7 at τt = 0.06 (no collapse); collapse (0.1) first appears at 0.08. Should read 'not higher than 0.06'.
- p. 17 Appendix D, Sharpening
we observe that a temperature lower than 0.06 is required to avoid collapse
- p. 17 Appendix D, sharpening table, τt = 0.06
68.7
- p. 17 Appendix D, Sharpening
we observe that a temperature lower than 0.06 is required to
- p. 17 App. D, Sharpening
a temperature lower than 0.06 is required to
Judge's note: Page 17 image: sharpening table reads τt 0 / 0.02 / 0.04 / 0.06 / 0.08 / 0.04→0.07 with k-NN 43.9 / 66.7 / 69.6 / 68.7 / 0.1 / 69.7. The next sentence ('When the temperature is higher than 0.06, the training loss consistently converges to ln(K)') agrees with the table; the first sentence strictly excludes 0.06, which works. Wording imprecision, the data are fine.
Second check: another model agreed
Report a problem with this finding
22 minor (typos and the like; most checked by the judge only)
Wording Table 1 caption: "“im/s” is the inference time on a NVIDIA V100 GPU"; im/s is a throughput, not a time.
- p. 4 Table 1 caption
“im/s” is the inference time on a NVIDIA
Judge's note: Page 4 caption reads as quoted. Table 2's caption on page 5 calls the same column "throughput (im/s)". Meaning is recoverable; loose wording.
Second check: another model agreed
Report a problem with this finding
Wording Appendix A describes the linear protocol as "a linear regression learned with cyanure library [45]", while Table 10 heads the same columns "Logistic" and Table 12 and its text say "a logistic regression trained on frozen features".
- p. 13 Appendix A, k-NN classification
a linear regression learned with cyanure library [45]
- p. 13 Table 10 column group header
Logistic
- p. 14 Table 12 caption
We train a logistic regression on frozen features
- p. 14 Low-shot learning on ImageNet
logistic regression trained on frozen features
Judge's note: Viewed pages 13 and 14. The Table 10 logistic values for Inet 1% and 10% (64.5, 72.2) are the same as the DINO-FROZEN row of Table 12, which is described as a logistic regression trained with cyanure, so the same classifier is meant. "Linear regression" is a loose or wrong term for it; no result is affected.
Second check: another model agreed
Report a problem with this finding
Typo Introduction: "in the form of close procedure in BERT [18]" should read "cloze procedure".
- p. 1 Introduction, second paragraph
in the form of close
- p. 1
in the form of close procedure in BERT
Judge's note: Page 1 image reads "close procedure in BERT [18]". The masked-word task is the cloze procedure; "close" is a misspelling of a standard term.
Second check: another model agreed
Report a problem with this finding
Typo Three grammar slips: "Each networks outputs a K dimensional feature" (Fig. 2 caption), "with an exponentially decay" (Sec. 5.2), "distincts from the distractors" (Sec. 4.2.1).
- p. 2 Figure 2 caption
Each networks outputs a K dimensional feature
- p. 9 Section 5.2, Analyzing the training dynamic
exponentially decay
- p. 6 Section 4.2.1, Copy detection
distincts
- p. 6
distincts from the distractors.
Judge's note: All three read as quoted on the page images (pages 2, 9, 6). Page 4 has the correct form "with an exponential decay". Cosmetic only.
Second check: another model agreed
Report a problem with this finding
Typo Introduction: "the good performance with k-NN only emerge when combining certain components" should be "emerges".
- p. 2 Introduction, paragraph after bullets
the good performance with k-NN only emerge when com-
Judge's note: Page 2 image reads as quoted; singular subject "performance" with plural verb.
Second check: none (judge only)
Report a problem with this finding
Typo Misspellings and a number slip: "on a singe V100 GPU" (single), "at the cost of a slighlty worse throughput" (slightly) on p.17; "Each networks outputs a K dimensional feature" (Each network outputs) in the Figure 2 caption on p.2.
- p. 17 Appendix D, Impact of the number of heads in ViT-S
on a singe V100 GPU
- p. 17 Appendix D, Impact of the number of heads in ViT-S
at the cost of a slighlty
- p. 17
at inference time on a singe V100 GPU
- p. 2 Figure 2 caption
Each networks outputs a K dimensional feature
Judge's note: All three read as quoted in the page images for pages 17 and 2. The 'compare our 300 epochs run' item listed here is the same defect as C051 and is handled there.
Second check: another model agreed
Report a problem with this finding
Typo Page 5: "The code and models to reproduce our results is publicly available." (should be "are"); "We freeze the pretrain model to compute" (should be "pretrained").
- p. 5 Implementation details
The code and models to reproduce our results is publicly available.
- p. 5 Evaluation protocols
We freeze the pretrain model to compute
Judge's note: Both read as quoted on the page 5 image. Cosmetic grammar slips.
Second check: none (judge only)
Report a problem with this finding
Typo Sec. 3.2: "previous works[18, 19, 69]" has no space before the citation bracket.
- p. 5 Sec 3.2, ViT description
previous works[18, 19, 69]
Judge's note: Crop confirms the missing space in the rendered page; other citations on the page are preceded by a space. Not an extraction artefact.
Second check: none (judge only)
Report a problem with this finding
Typo Table 3 header has two columns both labelled "Pretrain" (one for the method: Sup./DINO, one for the dataset: ImNet/GLDv2).
- p. 6 Table 3 header row
Pretrain Arch. Pretrain
Judge's note: Page 6 image shows the header "Pretrain | Arch. | Pretrain". The column contents make the meaning clear, so this is a minor labelling ambiguity, not an error in the data.
Second check: none (judge only)
Report a problem with this finding
Typo Sec. 4.1: "While reducing the patch size do not add parameters, it still leads to a significant reduction of running time" — subject-verb disagreement; should be "does not add".
- p. 6 Sec 4.1, comparing across architectures
While reducing the patch size
Judge's note: Read on the page 6 image, left column, 'Comparing across architectures' paragraph. The subject is the gerund 'reducing', so the verb should be singular. (The same sentence's 'reduction of running time' is also odd given smaller patches run slower, but that was not the finding raised.)
Second check: none (judge only)
Report a problem with this finding
Typo Sec. 4.1: "throughput (1237/sec VS 1007 im/sec)" — unit written two ways inside one parenthesis ("/sec" and "im/sec"), while tables and Fig. 5 use "im/s".
- p. 6 Sec 4.1
throughput (1237/sec VS 1007 im/sec)
Judge's note: Read on the page 6 image, top of left column. The inconsistency is as printed; the meaning is not in doubt. Capitalised 'VS' recurs in the next line ('79.3% VS 79.8%'), so it is the authors' style rather than a slip. Cosmetic only.
Second check: none (judge only)
Report a problem with this finding
Typo Figure 5 legend labels the blue series "DeiT-S" while the caption says "with ViT-B and ViT-S" and the text calls the model ViT-S.
- p. 8 Figure 5 legend
DeiT-S
- p. 8 Figure 5 caption
with ViT-B and ViT-S
- p. 8 Figure 5 caption
with ViT-B and ViT-S.
- p. 8 Figure 5 caption
different input patch sizes with ViT-B and ViT-S
Judge's note: Legend read on the Figure 5 crop: 'ViT-B' and 'DeiT-S'; caption: 'with ViT-B and ViT-S'. The paper states on page 5 that ViT-small 'follows the design of DeiT-S [69]' and the appendix says 'the default model DeiT-S', so both names denote the same model and no reader is misled; a naming inconsistency within one figure only.
Second check: none (judge only)
Report a problem with this finding
Typo Section 5.1 misspells 'addition': "In addtition, comparing rows 3 and 9 highlights the importance of the momentum encoder".
- p. 8 Section 5.1, first paragraph after Table 6
In addtition, comparing rows
- p. 8
In addtition, comparing rows 3 and 9 highlights
- p. 8 Sec 5.1
In addtition, comparing rows 3
Judge's note: The crop of page 8 shows "In addtition", so the misspelling is in the PDF and is not an extraction artefact. It should read "In addition".
Second check: another model agreed
Report a problem with this finding
Typo Table 7 caption title "Important component for self-supervised ViT pre-training." uses the singular although the caption goes on to "the different components"; Table 13 caption writes "DEIT-small" where the paper otherwise writes "DeiT" and the table header says "ViT-small".
- p. 8 Table 7 caption
Important component for self-supervised ViT pre-
- p. 14 Table 13 caption
DEIT-small
Judge's note: Both read on images: Table 7 caption on page 8 ('Important component ... We study the different components that matter'), Table 13 caption on the page 14 crop ('Methodology comparison for DEIT-small and ResNet-50', column header 'ViT-small'). Two unrelated cosmetic slips bundled in one cluster; both are as printed.
Second check: none (judge only)
Report a problem with this finding
Typo Sec. 5.2: "a form of Polyak-Ruppert averaging [51, 59] with an exponentially decay." — should be "an exponential decay".
- p. 9 Sec 5.2, analyzing the training dynamic
with an exponentially decay
Judge's note: Read on the page 9 image, left column, 'Analyzing the training dynamic' paragraph. Adverb used where an adjective is required.
Second check: none (judge only)
Report a problem with this finding
Typo Sec. 5.3: "but encourages an uniform output" — should be "a uniform output".
- p. 9 Sec 5.3
encourages an uniform output
Judge's note: Read on the page 9 image, right column below Table 8. Wrong indefinite article before a consonant sound.
Second check: none (judge only)
Report a problem with this finding
Typo Section 5.4 has 'form' where 'from' is meant: "the gain from adding more views diminishes (+.2% form 6× to 10× 96² crops) for longer trainings".
- p. 10 Section 5.4, first paragraph
form 6× to 10×
- p. 10
diminishes (+.2% form 6× to 10× 962 crops) for
- p. 10 Sec. 5.4, first paragraph on the page
form 6× to 10× 96
- p. 10 5.4
(+.2% form 6× to 10× 962 crops)
Judge's note: The crop of page 10 shows "form 6× to 10×". It should read "from". The meaning is clear and no number is affected.
Second check: another model agreed
Report a problem with this finding
Typo Reference [5]: "Maxim Berman, Hervé Jégou, Vedaldi Andrea, Iasonas Kokkinos, and Matthijs Douze." — one author's name is given surname-first ("Vedaldi Andrea"), unlike every other name in the list and unlike "Andrea Vedaldi" in reference [2].
- p. 10 Reference list, entry [5]
Vedaldi Andrea
- p. 10 citation of reference 5
learning to align and translate. preprint arXiv:1409.0473, 2014. 5
- p. 10 reference list, entry 5
Maxim Berman, Herv´e J´egou, Vedaldi Andrea, Iasonas Kokkinos, and Matthijs
Judge's note: Read on the page 10 image. The inversion is evident from the paper itself (entry [2] prints 'Andrea Vedaldi'). The second reviewer note in this cluster (that reference 5 does not fit its citation) is not a defect: the quoted text 'learning to align and translate. preprint arXiv:1409.0473, 2014. 5' is the tail of entry [4] with its back-reference page number, not a citation of [5]. In the body, [5] is cited for MultiGrain in Table 4 and the copy-detection paragraph on page 6, which fits.
Second check: another model agreed
Report a problem with this finding
Typo Conclusion has three grammar slips: "achieving performance that are comparable", "We have also seen emerged two properties" and "we have evidences that".
- p. 10 Sec 6 Conclusion
performance that are comparable
- p. 10 Sec 6 Conclusion
we have evidences that
- p. 10 Sec 6 Conclusion
We have also seen emerged
Judge's note: Viewed page 10, Section 6. All three phrases are printed as quoted. They are grammatical errors only; the meaning is clear.
Second check: none (judge only)
Report a problem with this finding
Typo Reference [75] repeats a surname: "Qizhe Xie, Zihang Dai Dai, Eduard Hovy, Minh-Thang Luong, and Quoc V. Le."
- p. 12 Reference list, entry [75]
Zihang Dai Dai
- p. 12
Qizhe Xie, Zihang Dai Dai, Eduard Hovy,
Judge's note: Viewed the crop of page 12. The entry prints "Zihang Dai Dai"; the surname is duplicated.
Second check: another model agreed
Report a problem with this finding
Typo In Table 11 the first row is labelled "MMP", but the caption defines the abbreviation as "“MPP” is Masked Patch Prediction".
- p. 13 Table 11, first row
MMP
- p. 13 Table 11 caption
“MPP” is Masked Patch Prediction
- p. 13 Table 11 caption
“MPP” is Masked Patch Prediction.
- p. 13 Table 11 first row
MMP JFT-300M
Judge's note: Page 13 crops: the caption defines MPP, and the table row under 'Pretrain on additional data' reads MMP (JFT-300M, 384, [19], 79.9). The text layer also gives MMP at line 1796. MMP is defined nowhere else, so the letters are transposed.
Second check: none (judge only)
Report a problem with this finding
Typo Grammar slips in Appendix D: "leading to worse performance compare our 300 epochs run" (compared to); "By aiming a target obtained with a teacher" (aiming at); "τ → 0 (extreme sharpening) correspond to the argmax operation and leads" (corresponds).
- p. 17 Appendix D, Longer training
pare our 300 epochs run
- p. 17 Longer training
worse performance com- pare our 300 epochs run
- p. 17 The teacher outperforms the student
By aiming a target obtained
- p. 17 Sharpening
correspond to the argmax operation
Judge's note: All three read as quoted in the page 17 image. The reviewer's mention of 'Compare to random initialization' on p.13 is also present in the text layer (same slip).
Second check: none (judge only)