wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations

Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, Michael Auli

2020 · paper · AI and machine learning

Presents a framework that learns speech representations from unlabeled audio by masking latent features and solving a contrastive task, then fine-tuning on transcribed speech. It reports 1.8/3.3 word error rates on Librispeech and 4.8/8.2 with only ten minutes of labeled data.

Checked: PDF from arxiv.org, fetched 4 Oct 2026. Page numbers are those of that PDF.

9 findings

Substance Sec. 5.1, p.5: "The LARGE model pre-trained on LV-60k and fine-tuned on only 10 minutes of labeled data achieves a word error rate of 5.2/8.6 on the Librispeech clean/other test sets". Table 1 (p.6), the abstract and the conclusion give 4.8/8.2 for the same setting.

p. 5 Section 5.1, last paragraph
a word error rate of 5.2/8.6 on the Librispeech clean/other test sets
p. 6 Table 1, 10 min labeled, LARGE LV-60k Transf.
4.8
p. 1 Abstract
still achieves 4.8/8.2 WER
p. 8 Conclusion
we achieve a WER of 4.8/8.2 on test-clean/other of Librispeech

Judge's note: On p.6, Table 1, the 10 min labeled LARGE LV-60k Transf. row reads 4.6 7.9 4.8 8.2. The conclusion on p.8 says 4.8/8.2. Sec. 5.1 states that most results use the Transformer LM, and no 5.2/8.6 pair for this model appears in the text layer. The appendix 4-gram and no-LM rows give other values. The same quantity is reported two ways.

Second check: another model agreed

Report a problem with this finding

Substance Eq. (1), p.3, is printed as p_{g,v} = exp(l_{g,v}+n_v)/τ / Σ_k exp(l_{g,k}+n_k)/τ. With /τ outside the exponential, τ cancels, so the temperature has no effect.

p. 3 Equation (1)
exp(lg,v + nv)/τ

Judge's note: The page image shows '/τ' outside the exp(...) parentheses in both the numerator and the denominator. I reworked it: [exp(a)/τ] / Σ_k[exp(b_k)/τ] = exp(a)/Σ_k exp(b_k), which does not depend on τ. That contradicts the text, which calls τ a temperature and anneals it from 2 to 0.5/0.1 in Sec. 4.2. The evident intended form is exp((l_{g,v}+n_v)/τ).

Second check: another model agreed

Report a problem with this finding

Substance Appendix D: "3696 utterances of average length 13.6 sec, or 563k discrete latents" does not fit the stated 49 Hz encoder rate or the 'five hours' size of TIMIT.

p. 15 Appendix D, paragraph 1
3696 utterances of average length 13.6 sec, or 563k discrete latents
p. 4 Section 4.1
ﬁve hours of audio recordings
p. 15 Appendix D, first paragraph
The training data contains 3696 utterances of average length 13.6 sec, or 563k discrete latents.

Only the authors can say what is right here.

Judge's note: §4.2 (p5) gives the encoder output frequency as 49 Hz (about 20 ms stride). 3696 × 13.6 s is about 50,266 s (about 14.0 h), which is more than the five hours stated in §4.1 for the whole of TIMIT, and would give about 2.46M latents. 563k latents correspond to about 3.1 s per utterance. Only the authors can say which figure is wrong.

Second check: another model agreed

Report a problem with this finding

Substance Appendix E: "further decreases the error rate to 33.8 for LARGE on LS-960". Table 9 gives 38.3 for LARGE LV-60k (10 min, no LM, dev-clean), and LARGE LS-960 is 43.0.

p. 16 Appendix E, paragraph 2
further decreases the error rate to 33.8 for LARGE on LS-960
p. 14 Table 9, 10 min labeled, LARGE LV-60k, None, dev clean
38.3
p. 16 Appendix E, paragraph 2
decreases the error rate to 33.8 for LARGE on LS-960
p. 16 Appendix E, paragraph 3
The LARGE LV-60k model achieves WER 38.3 on dev-clean

Judge's note: Table 9 (p14), 10-min labelled, None rows, dev-clean: BASE 46.1, LARGE LS-960 43.0, LARGE LV-60k 38.3. No 33.8 appears anywhere. The next paragraph on p16 itself says 'The LARGE LV-60k model achieves WER 38.3 on dev-clean'. So both the number and the model label (LS-960 for LV-60k) are wrong.

Second check: another model agreed

Report a problem with this finding

Substance Appendix E says that adding a Transformer LM to the 10-min LARGE LV-60k model "gives a large WER improvement to 5.0" on dev-clean. Table 9 gives 4.6 for that cell.

p. 16 Appendix E, paragraph 3
gives a large WER improvement to 5.0
p. 14 Table 9, 10 min labeled, LARGE LV-60k, Transf., dev clean
4.6
p. 14 Table 9, 10 min LARGE LV-60k Transf.
Transf. 4.6 7.9 4.8 8.2

Judge's note: Table 9 (p.14), 10 min LARGE LV-60k Transf. row: dev-clean 4.6, dev-other 7.9, test-clean 4.8, test-other 8.2. The sentence is about dev-clean (38.3 without an LM). None of the four cells in that row is 5.0.

Second check: another model agreed

Report a problem with this finding

Substance Appendix E: "Increasing the amount of unlabeled training data further decreases the error rate to 33.8 for LARGE on LS-960". The model with more unlabeled data is LARGE on LV-60k, and Table 9 gives 38.3 for it, not 33.8.

p. 16 Appendix E, paragraph 2
Increasing the amount of unlabeled training data further decreases the error rate to 33.8 for LARGE on LS-960.

Judge's note: Checked Table 9 (p.14), 10 min labeled, LM None, dev-clean: BASE LS-960 46.1, LARGE LS-960 43.0, LARGE LV-60k 38.3. The sentence before it already uses LARGE LS-960 (43), and 'increasing unlabeled data' can only mean LV-60k. Also, 33.8 appears nowhere in that row. The next paragraph itself says 'The LARGE LV-60k model achieves WER 38.3 on dev-clean', so 33.8 looks like transposed digits and LS-960 is the wrong label.

Second check: another model agreed

Report a problem with this finding

Reference or label p.3: "We use the straight-through estimator [26]". Ref. [26] is D. Jiang et al., "Improving transformer-based speech recognition using unsupervised pre-training", which is not a source for the straight-through estimator.

p. 3 Section 2, Quantization module
We use the straight-through estimator [26]
p. 10 Reference [26]
Improving transformer-based speech recognition using unsupervised pre-training

Only the authors can say what is right here.

Judge's note: Rests on outside knowledge: the straight-through estimator originates elsewhere (e.g. Bengio et al. 2013), not in Jiang et al. 2019. The reference list shows [26] = Jiang et al., arXiv 1910.09932, on transformer ASR pre-training. The paper also cites [26] for masking (p.2 '[4, 26]' and '[26, 56]'). No Bengio et al. straight-through reference appears in the list. The authors should confirm which reference was intended.

Second check: another model agreed

Report a problem with this finding

Reference or label §5.4 calls the ablation metric "(dev PER)", but it is the average WER on Librispeech dev-clean/dev-other (Table 4 header 'avg. WER').

p. 8 Section 5.4, paragraph 1
the concatenation of dev-clean and dev-other (dev PER)
p. 8 Table 4 header
avg. WER
p. 8 Sec 5.4
dev-clean and dev-other (dev PER)
p. 8 Sec 5.4
(dev PER)

Judge's note: Page 8: the §5.4 sentence reads 'We report the average WER ... on the concatenation of dev-clean and dev-other (dev PER)'. The Table 4 caption and header say WER. PER (phoneme error rate) is used only for TIMIT (Table 3).

Second check: another model agreed

Report a problem with this finding

Reference or label Reference [23] author list reads "M. G. A. Hyvärinen. Noise-contrastive estimation...", which looks like two authors merged into one name.

p. 10 Reference [23]
M. G. A. Hyvärinen. Noise-contrastive estimation
p. 10 Ref [23]
M. G. A. Hyvärinen

Only the authors can say what is right here.

Judge's note: Rests on outside knowledge: that the AISTATS 2010 NCE paper is by M. Gutmann and A. Hyvärinen. The page 10 image shows the malformed 'M. G. A. Hyvärinen' as printed, so this is not an extraction artefact.

Second check: another model agreed

Report a problem with this finding

16 minor (typos and the like; most checked by the judge only)

Reference or label Ref [43] (ELMo) is given as "In Proc. of ACL, 2018".

p. 11 Reference [43]
Deep contextualized word representations. In Proc. of ACL, 2018.

Only the authors can say what is right here.

Judge's note: Rests on outside knowledge: that Peters et al. 2018 appeared at NAACL-HLT 2018, not ACL. The page 11 image shows 'Proc. of ACL, 2018'.

Second check: another model agreed

Report a problem with this finding

Reference or label The Figure 2 y-axis is labelled "Percent of all spans", but its values (0 to about 0.27) behave like fractions.

p. 12 Figure 2 y-axis
Percent of all spans

Only the authors can say what is right here.

Judge's note: Page 12 image: the curve peaks at about 0.27, sits at about 0.03 over roughly lengths 10-20, and then has a small tail. Over integer span lengths this adds up to about 1, which fits proportions. Read as percentages, it would total about 1%. The unit label looks wrong, but the binning is not stated, so the authors should confirm.

Second check: another model agreed

Report a problem with this finding

Wording Sec. 5.3, p.7: "We fine-tune as for the 10 hour subset of Libri-light but do not use a language model". Table 6 lists TIMIT with channel mask prob. 0.012 and 40k updates, against 0.004 and 20k for 10 hours.

p. 7 Section 5.3
We ﬁne-tune as for the 10 hour subset of Libri-light but do not use a language model.
p. 13 Table 6, rows 10 hours and TIMIT
0.012

Judge's note: The Table 6 crop confirms 10 hours = 0.065 / 0.004 / 20k and TIMIT = 0.065 / 0.012 / 40k. Only the timestep mask probability is the same. 'As for' probably means the general recipe, but the sentence does not mention the differing hyperparameters.

Second check: another model agreed

Report a problem with this finding

Typo Introduction, p.2: "followed by a contextualized representations with a self-attention model": the singular article does not agree with the plural noun.

p. 2 Introduction, paragraph after Figure 1
followed by a contextualized representations with a

Judge's note: The crop of p.2 shows the text exactly as quoted. The fix is to write either 'a contextualized representation' or 'contextualized representations'.

Second check: none (judge only)

Report a problem with this finding

Typo Sec. 3.2, p.4: "the softmax disribution does not contain the gumbel noise" misspells 'distribution'.

p. 4 Section 3.2, Diversity Loss
the softmax disribution does not contain
p. 4
the softmax disribution does not contain the gumbel noise

Judge's note: The misspelling appears on the p.4 page image, in the first line.

Second check: none (judge only)

Report a problem with this finding

Typo Sec. 4.1, p.4: "evaluate on the standard Librispech dev-other/clean and test-clean/other sets" misspells 'Librispeech'.

p. 4 Section 4.1
the standard Librispech
p. 4 Sec 4.1
standard Librispech
p. 4
standard Librispech dev-other/clean and test-clean/other sets

Judge's note: The misspelling appears on the p.4 page image, in the Sec. 4.1 paragraph.

Second check: none (judge only)

Report a problem with this finding

Typo Sec. 3.3, p.4: "apply a modified version of SpecAugment [41] by masking to time-steps and channels" has a stray word 'to'.

p. 4 Section 3.3
by masking to time-steps and channels

Judge's note: The p.4 image shows the phrase as quoted. It should read 'by masking time-steps and channels'.

Second check: none (judge only)

Report a problem with this finding

Typo Sec. 4.2, p.5: "Entries are of size d/G = 128 for BASE amd d/G = 384 for LARGE" ('amd' for 'and'). The Table 13 caption on p.19 also has "hyper-parmeter".

p. 5 Section 4.2
for BASE amd d/G = 384
p. 19 Table 13 caption
hyper-parmeter
p. 4 Sec 4.2 / 3.2
disribution
p. 5 Sec 4.2
BASE amd d/G

Judge's note: Both misspellings appear on the page images of p.5 and p.19. The 'disribution' instance belongs to C004.

Second check: none (judge only)

Report a problem with this finding

Typo Misspellings: "complimentary to pre-training" (p7, for complementary), "BASE amd d/G = 384" (p5), "standard Librispech" (p4), "softmax disribution" (p4), "hyper-parmeter choices" (Table 13 caption, p19).

p. 7 Section 5.2
self-training is likely complimentary to pre-training
p. 5 Sec 4.2
BASE amd d/G = 384
p. 4 Sec 4.1
Librispech
p. 19 Table 13 caption
hyper-parmeter

Judge's note: Each misspelling can be seen in the page images: p7 §5.2 'complimentary', p5 §4.2 'amd', p4 §4.1 'Librispech', p4 §3.2 'disribution', p19 Table 13 caption 'hyper-parmeter'.

Second check: none (judge only)

Report a problem with this finding

Typo p.6, below Table 1: "reducing WER by a about a third" has a duplicated article.

p. 6 Paragraph below Table 1
reducing WER by a about a third
p. 16 Appendix E
anyone →any one, awhile →a while

Judge's note: The page image of p.6 confirms the text. The side point in one reviewer's note about Appendix E's 'anyone → any one' wording is not part of this defect, and I did not adjudicate it.

Second check: none (judge only)

Report a problem with this finding

Typo §5.4: "We motivate this choice by an ablating for which" should read 'an ablation'.

p. 8 Section 5.4
We motivate this choice by an ablating
p. 8 §5.4, first paragraph
We motivate this choice by an ablating for which

Judge's note: Seen on the page 8 image, §5.4 first paragraph.

Second check: none (judge only)

Report a problem with this finding

Typo §5.4: "increases from 62% to 78.0%" gives the two values of the same accuracy at different precision.

p. 8 Sec 5.4
increases from 62% to 78.0%

Judge's note: Seen on page 8. This is a cosmetic inconsistency only.

Second check: none (judge only)

Report a problem with this finding

Typo Ref [59] lists author "T. Schaiz", probably for T. Schatz.

p. 11 Ref [59]
T. Schaiz

Only the authors can say what is right here.

Judge's note: Rests on outside knowledge: that the co-author of Zeghidour et al. ICASSP 2018 is Thomas Schatz. The page 11 image shows 'T. Schaiz'.

Second check: none (judge only)

Report a problem with this finding

Typo Appendix A: "increased prediction accuracy for the self-supervised but the task becomes trivial" is missing a noun (e.g. 'task').

p. 12 Appendix A, paragraph 2
increased prediction accuracy for the self-supervised but

Judge's note: Seen on page 12, Appendix A paragraph 2.

Second check: none (judge only)

Report a problem with this finding

Typo Table 13 caption misspells 'hyper-parameter': "Ablation of various hyper-parmeter choices".

p. 19 Table 13 caption
hyper-parmeter choices
p. 19 Table 13 caption
hyper-parmeter
p. 19 last paragraph
unsmasked
p. 19
Ablation of various hyper-parmeter choices

Judge's note: Seen in the Table 13 caption on the p.19 image. The 'unsmasked' typo that one reviewer also raised here is judged under C036.

Second check: none (judge only)

Report a problem with this finding

Typo Appendix F misspells 'unmasked': "next to these unsmasked spans".

p. 19 Appendix F, final paragraph
next to these unsmasked spans

Judge's note: Seen in the last paragraph on the p.19 image. Elsewhere in the same paragraph the word is spelled 'unmasked'.

Second check: none (judge only)

Report a problem with this finding

Something wrong with this page, or a finding we missed?