Learning Transferable Visual Models From Natural Language Supervision
2021 · paper · AI and machine learning
Trains image models by predicting which caption matches which image, using 400 million image-text pairs from the internet. Natural language then enables zero-shot transfer to over 30 vision tasks, matching the original ResNet-50's ImageNet accuracy without its training examples.
Checked: PDF from arxiv.org, fetched 4 Oct 2026. Page numbers are those of that PDF.
11 findings
Substance Text-encoder vocabulary size is given as "a 49,152 vocab size" in Sec 2.4 (p.5) but as "Vocabulary size 49408" in Table 18 (p.48).
- p. 5 Section 2.4, text encoder paragraph
with a 49,152 vocab size
- p. 48 Table 18
Vocabulary size 49408
- p. 48 Table 18
49408
Only the authors can say what is right here.
Judge's note: Both images confirm the two values. Both are labelled as the vocabulary size of the same BPE text encoder. I found no statement in the paper that one count excludes byte or special tokens, so only the authors can say which figure is meant or how they differ.
Second check: another model agreed
Report a problem with this finding
Substance The Figure 5 UCF101 bar "+7.7" and the text "outperforms a ResNet-50's features by 7.7% on UCF101" (p.8) do not match Table 11 L/14-336px zero-shot UCF101 76.9 minus Table 10 ResNet-50 linear-probe UCF101 71.2 = 5.7.
- p. 8 Figure 5 / text
Zero- shot CLIP also outperforms a ResNet-50’s features by 7.7% on UCF101
- p. 43 Table 11, L/14-336px, UCF101 and KITTI
21.9 34.9 63.0 76.9 61.3
- p. 40 Table 10, ResNet 50 row
67.1 11.7 82.5 71.2 46.8
- p. 8 Figure 5 bar for UCF101 and the text
+7.7
Only the authors can say what is right here.
Judge's note: I checked the column order in both table images: Table 11 has ...KITTI, Country211, PCam, UCF101, Kinetics700...; Table 10 has the same order. Other Figure 5 bars reproduce exactly from this model pair (Cars 78.8-49.9, Country211 34.9-11.7, Kinetics 61.3-46.8, ImageNet 76.2-74.3, EuroSAT 59.6-96.7, PCam 63.0-82.5), but UCF101 gives 5.7, not 7.7. The paper does not say Figure 5 uses a different model or run for UCF101. The KITTI bar is handled separately in C011.
Second check: another model agreed
Report a problem with this finding
Substance The Figure 5 KITTI Distance bar "-34.0" (p.8) does not match Table 11 L/14-336px zero-shot KITTI 21.9 minus Table 10 ResNet-50 linear-probe KITTI 67.1 = -45.2.
- p. 8 Figure 5 bar for KITTI Distance
-34.0
- p. 43 Table 11, L/14-336px row, KITTI column
21.9
- p. 40 Table 10, ResNet 50 row, KITTI column
67.1
Only the authors can say what is right here.
Judge's note: Same model pair as the other Figure 5 bars, which reproduce exactly (see C010). The KITTI column was checked against the headers on p.40 and p.43. The paper does not explain the discrepancy. This is a different bar from C010's UCF101 value, so it is kept as a separate item.
Second check: another model agreed
Report a problem with this finding
Substance Country211 is described as "sampling 200 photos for training and 100 photos for testing, for each country" over 211 countries (p.37), but Table 9 (p.39) lists train size "43,200". 211×200 = 42,200.
- p. 37 A.1, Country211 description
by sampling 200 photos for training and 100 photos for testing, for each country
- p. 39 Table 9, Country211 row
211 43,200 21,100
- p. 37 A.1 Datasets, Country211
sampling 200 photos for training and 100 photos for testing, for each country
- p. 39 Table 9, Country211
Country211 211 43,200 21,100
Judge's note: The text says the dataset is balanced, with exactly 200 training and 100 test photos per country. The test count matches (211×100 = 21,100), but the training count does not, and no validation split or extra images are mentioned.
Second check: another model agreed
Report a problem with this finding
Substance Table 9 lists "Food-101 102 75,750 25,250"; the class count should be 101.
- p. 39 Table 9, Food-101 row
Food-101 102 75,750 25,250
- p. 42 Figure 21, Food101 panel
correct rank: 1/101
Judge's note: The split sizes divide exactly by 101, and the Figure 21 Food101 panel shows "correct rank: 1/101". The 102 appears to have been copied from neighbouring rows (Caltech-101, Flowers102).
Second check: another model agreed
Report a problem with this finding
Substance Table 9 gives "Facial Emotion Recognition 2013 8" classes, while the Figure 21 FER2013 panel ranks among 7 ("correct rank: 5/7").
- p. 39 Table 9, FER2013 row
Facial Emotion Recognition 2013 8 32,140 3,574
- p. 42 Figure 21, FER2013 panel
correct rank: 5/7
- p. 39 Table 9
Facial Emotion Recognition 2013 8
Only the authors can say what is right here.
Judge's note: The two internal figures conflict, and the paper gives no explanation for an eighth class. That FER2013 has 7 classes is outside knowledge, but the internal mismatch is enough to raise the query.
Second check: another model agreed
Report a problem with this finding
Substance Table 12, Country211 zero-shot: YFCC "5.2", WIT "5.3", Δ printed "+0.1". Under the table's YFCC−WIT convention this is −0.1.
- p. 44 Table 12, Country211 row, Zero Shot
5.2 5.3 +0.1
Judge's note: Every other Δ in the crop equals YFCC−WIT, e.g. Birdsnap 19.9−4.5 = +15.4, GTSRB 6.9−7.0 = −0.1, Cars 3.8−10.9 = −7.1. Rounding cannot yield a positive difference when the YFCC value rounds below the WIT value.
Second check: another model agreed
Report a problem with this finding
Reference or label The "noisy-student variants (B0-B7, L2-475, and L2-800) (Tan & Le, 2019)" are cited to the original EfficientNet paper; elsewhere the paper attributes Noisy Student to Xie et al. (2020).
- p. 37 A.2, EfficientNet paragraph
the noisy-student variants (B0-B7, L2-475, and L2-800) (Tan & Le, 2019)
- p. 37 Appendix A.2 EfficientNet paragraph
the noisy-student variants (B0-B7, L2-475, and L2-800)
- p. 37 A.2 EfficientNet
noisy-student variants (B0-B7, L2-475, and L2-800) (Tan & Le, 2019)
Judge's note: Page 4 and the Figure 10 caption cite Xie et al. (2020) for the Noisy Student EfficientNet, so the A.2 citation is inconsistent with the paper's own attribution.
Second check: another model agreed
Report a problem with this finding
Reference or label A.2 says "VirTex has a similar model design to CLIP-AR", but no model named CLIP-AR is defined; the autoregressive model is introduced as "LM RN50".
- p. 38 A.2, VirTex paragraph
similar model design to CLIP-AR
- p. 38 A.2 VirTex
VirTex has a similar model design to CLIP-AR
- p. 37 A.2 LM RN50
LM RN50 This is a multimodal model that uses an au-
Only the authors can say what is right here.
Judge's note: CLIP-AR appears only in the VirTex paragraph on page 38. The autoregressive model is defined on page 37 and listed in Table 10 as LM RN50. It most likely refers to LM RN50, but the authors should confirm.
Second check: another model agreed
Report a problem with this finding
Reference or label Table 14: the SST-2 SOTA entry is printed "97.5^e", but the caption's footnote list runs "d(Lippe et al., 2020) f(Jaderberg et al., 2014)" and has no entry e, so the source of the 97.5 is not given.
- p. 46 Table 14, SST-2 SOTA and footnote list
97.5e
- p. 46 Table 14 caption
d(Lippe et al., 2020) f(Jaderberg et al.,
- p. 46 Table 14 footnote
d(Lippe et al., 2020) f(Jaderberg et al., 2014)
Only the authors can say what is right here.
Judge's note: I checked the crops of page 46. In the table, SOTA values carry superscripts a to e, and the SST-2 value carries e. The Table 14 caption lists footnotes a, b, c, d, f, g, h, i and has no e. Only the authors can say which reference was meant. The e entry under Table 13 (Yu et al., 2020) belongs to a different table.
Second check: another model agreed
Report a problem with this finding
Reference or label Table 17 caption: "c(Vo et al., 2017) c(Weyand et al., 2016)": the letter c is used twice. The table marks PlaNet with d, so Weyand et al. should be labelled d.
- p. 47 Table 17 footnotes
c(Vo et al., 2017) c(Weyand et al., 2016)
- p. 47 Table 17 footnote
c(Vo et al., 2017) c(Weyand et al.,
Judge's note: I checked the crop of page 47. In the table, Deep-Ret+ has superscript c and PlaNet has superscript d. The caption labels both Vo et al. and Weyand et al. with c and has no d.
Second check: another model agreed
Report a problem with this finding
28 minor (typos and the like; most checked by the judge only)
Substance Zero-shot ViT-L/14@336px ImageNet-A accuracy is "77.1" in Figure 13 (p.15) but "77.2" in Table 16 (p.47).
- p. 15 Figure 13 right panel, ImageNet-A row
77.1
- p. 47 Table 16 Zero-Shot CLIP IN-A
77.2
- p. 47 Table 16, Zero-Shot CLIP row
76.2 70.1 77.2 88.9 72.3 60.2
- p. 15 Figure 13 table
2.7 77.1
Only the authors can say what is right here.
Judge's note: Figure 13's caption says the right panel uses the best zero-shot CLIP model, ViT-L/14@336px, which is the same model as Table 16's Zero-Shot CLIP row. The other shared values agree (IN-R 88.9, ObjectNet 72.3, Sketch 60.2). The 1.9% ImageNet-A change quoted in the text matches 77.2 (77.2-75.3), so Figure 13's 77.1 looks inconsistent. The gap is only 0.1.
Second check: another model agreed
Report a problem with this finding
Wording Section 2.5 says RN50x4/x16/x64 'use approximately 4x, 16x, and 64x the compute of a ResNet-50'. The GFLOPs in Figures 4 and 9 (6.1 to 265.9), and the text's 'across a 44x increase in model compute', give about 3.5x, 12x and 44x.
- p. 5 Sec 2.5
use approximately 4x, 16x, and 64x the compute of a
- p. 10 Sec 3.1.5
across a 44x increase in model compute
Judge's note: The GFLOPs come from the axis labels of Figures 4 and 9. The x4/x16/x64 labels may be nominal scaling targets or a different compute measure; the paper does not say which. It is a wording inconsistency.
Second check: another model agreed
Report a problem with this finding
Wording 'On aYahoo, CLIP achieves a 95% reduction in the number of errors', but Table 1 (72.4 to 98.4) gives a 94.2% error reduction.
- p. 7 Sec 3.1.3
CLIP achieves a 95% reduction
Judge's note: Computed from Table 1 on page 7. The printed figure is loose rounding, about 94% rather than 95%.
Second check: another model agreed
Report a problem with this finding
Wording Section 3.1.5 says 'On STL10, CLIP achieves 99.3% overall'. Table 11 gives 99.4 for L/14-336px, the model the paper defines as 'CLIP'; 99.3 is the L/14 (224px) value.
- p. 8 Section 3.1.5
CLIP achieves 99.3% overall
- p. 43 Table 11, L/14-336px row, STL10
99.4
- p. 8 Sec 3.1.5 text
CLIP achieves 99.3% overall which appears to be a new
- p. 43 Table 11, L/14-336px STL10
99.4 59.6 71.7
Only the authors can say what is right here.
Judge's note: The paper states that 'CLIP' means ViT-L/14@336px unless otherwise specified. Figure 5's +3.0 equals 99.4-96.4, using Table 10's ResNet-50 STL10 score, so the figure agrees with 99.4. The text gives no other model name.
Second check: another model agreed
Report a problem with this finding
Wording Caltech101 is listed among the 5 datasets where zero-shot comes within '≤3 point difference' of linear probe. The tables give 96.0 (linear probe, Table 10) and 92.8 (zero-shot, Table 11), a 3.2-point gap.
- p. 10 Figure 8 caption and text
approach linear probe performance (≤3 point difference)
- p. 43 Table 11 L/14-336px Caltech101
92.8
- p. 40 Table 10 L/14-336px Caltech101
96.0
Judge's note: Both values are for L/14-336px. The 3.2-point gap is just over the stated threshold.
Second check: another model agreed
Report a problem with this finding
Wording Sec 5 step 3: "We also calculate 99.5% Clopper-Pearson confidence intervals on Dirty as another check" uses the subset name 'Dirty', which is never defined. Only Overlap, Clean and All are defined.
- p. 18 Section 5, step 3
99.5% Clopper-Pearson confidence intervals on Dirty as
Judge's note: The crop of p.18 shows 'Dirty' in monospace. A grep of the text finds no other occurrence and no definition. It evidently means Overlap.
Second check: another model agreed
Report a problem with this finding
Wording In Table 5 the combined-gender rows for Zero-Shot CLIP and Linear Probe Instagram have no Average entry, whereas the Linear Probe CLIP combined row shows 97.5.
- p. 21 Table 5, Zero-Shot CLIP and Linear Probe Instagram third rows
96.7 95.9 98.0 97.5 98.0 96.3 96.6
Judge's note: The table image confirms the empty Average cells.
Second check: another model agreed
Report a problem with this finding
Wording Zero-shot CLIP MNIST accuracy is 88.4 in Table 14 but 88.3 in Table 11 (L/14-336px row).
- p. 46 Table 14, ZS CLIP row
CLIP 88.4 51.0 90.0 63.3 67.9
- p. 43 Table 11, L/14-336px row, MNIST
88.3
- p. 46 Table 14, ZS CLIP MNIST
88.4 51.0 90.0 63.3 67.9
- p. 43 Table 11, L/14-336px row, MNIST
78.3 88.3 57.7
Only the authors can say what is right here.
Judge's note: The other zero-shot values in Table 14 (Hateful Memes 63.3, SST-2 67.9) match Table 11 exactly. No different prompt set or evaluation is stated for Table 14, so the authors should say which value is right.
Second check: another model agreed
Report a problem with this finding
Typo Page 1: 'labels learned representations which preformed similarly' should read 'performed similarly'.
- p. 1 Introduction
labels learned representations which preformed similarly
Judge's note: The crop shows 'preformed' as printed. It is a misspelling.
Second check: none (judge only)
Report a problem with this finding
Typo Misspellings: "learned representations which preformed similarly" (p.1, performed), "pairs collected form a variety" (p.3, from), "the ImageNet datatset" (p.15, dataset), heading "EfficietNet" (p.37, EfficientNet).
- p. 3 Section 2.2
text) pairs collected form a variety
- p. 1 Intro
preformed similarly
- p. 3 Sec 2.2
collected form a variety
- p. 15 Sec 3.3
the ImageNet datatset
Judge's note: All four misspellings are visible in the page images: p.1 right column, p.3 Sec 2.2, p.15 Sec 3.3 left column, p.37 A.2 run-in heading. They are cosmetic. The 'form' item is also reported as C003 and 'EfficietNet' as C032; this cluster is kept as the umbrella for preformed/datatset.
Second check: none (judge only)
Report a problem with this finding
Typo Sec 2.3: "multiplicative scalar to avoid turning as a hyper-parameter" should read "tuning".
- p. 4 Section 2.3, final sentence
to avoid turning as a hyper-parameter
Judge's note: The crop of p.4, right column, shows "turning". The context (temperature optimized rather than hand-set) requires 'tuning'.
Second check: none (judge only)
Report a problem with this finding
Typo Page 8: the prompt example ''A photo of a big {label}”' opens with a single quote but closes with a double quote.
- p. 8 Section 3.1.4
‘A photo of a big {label}” and
Judge's note: The crop shows the mismatched quote marks.
Second check: none (judge only)
Report a problem with this finding
Typo Sec 3.2: "The EffcientNet also slightly outperforms CLIP" should read "EfficientNet".
- p. 12 Section 3.2
The EffcientNet also slightly outperforms
- p. 12
The EffcientNet also slightly outper-
Judge's note: Visible in the crop of p.12, right column; the surrounding sentences spell it correctly.
Second check: none (judge only)
Report a problem with this finding
Typo Page 13: 'They propose this distinction because in part because they find' repeats 'because'.
- p. 13 Section 3.3
because in part because they find
Judge's note: The crop shows the repeated word.
Second check: none (judge only)
Report a problem with this finding
Typo Figure 13 legend reads "Exisiting robustness techniques"; should be "Existing".
- p. 15 Figure 13 legend
Exisiting robustness techniques
Judge's note: The misspelling is visible in the legend of the left panel of Figure 13.
Second check: none (judge only)
Report a problem with this finding
Typo "the ImageNet datatset" should read "dataset".
- p. 15 Section 3.3
the ImageNet datatset
Judge's note: The typo is visible on page 15.
Second check: none (judge only)
Report a problem with this finding
Typo "Oxford IIT Pets" (Section 4 and the Table 2 caption) is inconsistent with "Oxford-IIIT Pets", which is used elsewhere in the paper.
- p. 16 Section 4
in the test split of the Oxford IIT Pets dataset
- p. 17 Table 2 caption
Comparison of human performance on Oxford IIT Pets.
- p. 16 Sec 4
the Oxford IIT Pets dataset
- p. 16 Section 4
Oxford IIT Pets dataset
Judge's note: Both occurrences of "IIT" are confirmed on the page images. The paper uses "Oxford-IIIT Pets" on pages 7, 8, 39 and 42.
Second check: none (judge only)
Report a problem with this finding
Typo "due to at least a two factors" contains a stray article.
- p. 17 Sec 4
due to at least a two factors
- p. 17 Section 4
at least a two factors
Judge's note: Visible on page 17.
Second check: none (judge only)
Report a problem with this finding
Typo Sec 8: "This areas of research dates back to the mid-90s" has a number-agreement error ('This area' or 'These areas').
- p. 26 Sec 8
This areas of research dates back to the mid-90s with the
- p. 26 Section 8
This areas of research dates back
- p. 26
This areas of research dates back to the mid-90s
Judge's note: Visible on p.26, left column. The secondary point (mid-90s vs the cited Mori et al. 1999) is a characterisation question and is not judged a defect.
Second check: none (judge only)
Report a problem with this finding
Typo "co-occuring" should be "co-occurring".
- p. 26 Section 8
co-occuring with images
Judge's note: Visible on page 26.
Second check: none (judge only)
Report a problem with this finding
Typo The reference entries He et al. 2016a and 2016b ("Deep residual learning for image recognition", pp. 770–778) are identical duplicates.
- p. 30 References
pp. 770–778, 2016b.
- p. 30 References
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778, 2016b.
Judge's note: Both entries appear on page 30 with the same title, venue and pages.
Second check: none (judge only)
Report a problem with this finding
Typo Appendix misspellings: heading "EfficietNet" (p.37), "using ViT-L/14 archiecture" (p.38), "achives 80.5% accuracy" and "MSCOCO datsets" (p.45).
- p. 37 A.2 heading
EfficietNet
- p. 38 A.4
archiecture
- p. 45 E.2
achives 80.5% accuracy
- p. 38 A.4
ViT-L/14 archiecture
Judge's note: All four misspellings are visible in the page crops. They are cosmetic.
Second check: none (judge only)
Report a problem with this finding
Typo Appendix A.2 model heading is misspelled "EfficietNet"; it should read "EfficientNet".
- p. 37 A.2 Models
EfficietNet
Judge's note: The crop of page 37 shows the bold heading "EfficietNet", while the body text spells "EfficientNet" correctly.
Second check: none (judge only)
Report a problem with this finding
Typo E.2 reads "zero-shot CLIP performs a bit more respectively"; this presumably should be "respectably".
- p. 45 E.2, IIIT5K sentence
CLIP performs a bit more respectively
- p. 45 E.2
performs a bit more respectively
- p. 45 E.1
high transfer performance transfer
Judge's note: The crop of page 45 shows the phrase exactly as quoted. Reviewer note 3 also mentions the duplicated word "transfer", which is handled separately in C037.
Second check: none (judge only)
Report a problem with this finding
Typo E.1 reads "achieve high transfer performance transfer on exactly what it is pre-trained for"; the second "transfer" is a duplicated word.
- p. 45 E.1
achieve high transfer performance transfer on exactly
- p. 45 E.1
high transfer performance transfer on exactly
Judge's note: The crop of page 45 shows the phrase exactly as quoted.
Second check: none (judge only)
Report a problem with this finding
Typo E.2, SST-2 paragraph: "Fitting a linear classifier on CLIP's representation of rendered sentences achives 80.5% accuracy": 'achives' should read 'achieves'.
- p. 45 E.2
achives 80.5% accuracy
- p. 45 E.2 OCR, SST-2 paragraph
sentences achives 80.5% accuracy
- p. 45
rendered sentences achives 80.5% accuracy.
Judge's note: I checked the crop of page 45, right column. The misspelling 'achives' is on the page image.
Second check: none (judge only)
Report a problem with this finding
Typo E.1: "text and image retrieval on the Flickr30k and MSCOCO datsets": 'datsets' should read 'datasets'.
- p. 45 E.1
MSCOCO datsets
Judge's note: I checked the crop of page 45, left column. The misspelling 'datsets' is on the page image.
Second check: none (judge only)
Report a problem with this finding
Typo E.2 reads "CLIP's OCR performance is strongest Hateful Memes and SST-2"; the word "on" is missing.
- p. 45 E.2
strongest Hateful Memes and SST-2
Judge's note: The sentence on page 45 lacks the preposition.
Second check: none (judge only)