Neural Discrete Representation Learning
2017 · paper · AI and machine learning
Introduces VQ-VAE, a generative model whose encoder outputs discrete codes and whose prior is learned, using vector quantisation. It avoids "posterior collapse" and is shown to generate images, video and speech, including speaker conversion.
Checked: PDF from arxiv.org, fetched 4 Oct 2026. Page numbers are those of that PDF.
5 findings
Substance Equation 3, "L = log p(x|zq(x)) + ∥sg[ze(x)] − e∥² + β∥ze(x) − sg[e]∥²", is called "the overall loss function" with first term "the reconstruction loss", but that term is a log-likelihood with a positive sign added to two squared-error penalties; it needs a minus sign (−log p(x|zq(x))).
- p. 4 Equation 3
L = log p(x|zq(x)) + ∥sg[ze(x)] −e∥2
- p. 4 Section 3.2, text before Equation 3
The first term is the reconstruction loss (or the data term)
- p. 4 Section 3.2, text before Equation 3
Equation 3 specifies the overall loss function.
Judge's note: Read page 4 image: no minus sign is printed before log p in Equation 3. The text calls L "the overall loss function", the first term "the reconstruction loss", and speaks of the encoder changing its output "to lower the reconstruction loss", so L is minimised. The second and third terms are squared distances that are minimised (they pull embeddings and encoder outputs together). A log-likelihood grows as reconstruction improves, so with the printed sign the three terms pull in inconsistent directions: minimising L lowers the likelihood, maximising it enlarges the squared errors. I found no statement in the paper that defines the first term as a negative log-likelihood or otherwise explains the sign.
Second check: another model agreed
Report a problem with this finding
Reference or label Section 3.2 text writes the reconstruction loss as "the reconstruction loss log p(z|zq(x))"; Equation 3 and the rest of the paper have log p(x|zq(x)).
- p. 4 Section 3.2, paragraph starting 'Equation 3 specifies'
the reconstruction loss log p(z|zq(x))
- p. 4 Equation 3
L = log p(x|zq(x))
- p. 4 Section 3.2, para 2
reconstruction loss log p(z|zq(x))
Judge's note: Viewed the crop and page 4 image: the prose prints log p(z|z_q(x)) with z as the first argument, so it is not an extraction artefact. Equation 3 on the same page prints L = log p(x|z_q(x)) + ..., and the log-likelihood derivation at the foot of the page and on page 5 also uses p(x|z_q(x)). No distribution of z given z_q(x) is defined anywhere in the paper. The x was evidently mistyped as z; the equation itself is unaffected, so this is graded as a wrong symbol and not a wrong result.
Second check: another model agreed
Report a problem with this finding
Reference or label Figure 4 caption reads "Samples (128x128) from a VQ-VAE with a PixelCNN prior trained on frames captured from DeepMind Lab", but the text says the DeepMind Lab experiment used "84x84x3 frames" with a "21x21x1 latent space", and the Figure 5 caption says the same decoder outputs "84x84".
- p. 6 Figure 4 caption
Figure 4: Samples (128x128) from a VQ-VAE with a PixelCNN prior trained on frames captured
- p. 6 Section 4.2, paragraph after Figure 4
84x84x3 frames drawn from the DeepMind Lab environment
- p. 7 Figure 5 caption
decoded with standard VQ-VAE decoder to 84x84
- p. 6 Section 4.2, paragraph below Figure 4
We also repeat the same experiment for 84x84x3 frames drawn from the DeepMind Lab environment
Only the authors can say what is right here.
Judge's note: Read the Figure 4 caption and the paragraph below it on page 6, and the Figure 5 caption on page 7. The text describes only an 84x84 DeepMind Lab model (21x21 latents, a factor of 4, as 128 to 32 for ImageNet). The "(128x128)" wording is identical to the Figure 3 ImageNet caption, so it looks copied. No statement that the samples were resized for display was found. Query rather than correction because the samples could have been upscaled.
Second check: another model agreed
Report a problem with this finding
Reference or label Section 4.3 says of the audio reconstruction "as shown in Figure 7, the reconstruction has the same content (same text contents), but the waveform is quite different"; Figure 7 shows DeepMind Lab video frames, and the waveforms are in Figure 6.
- p. 7 Section 4.3, third paragraph
shown in Figure 7, the reconstruction has the same content
- p. 7 Figure 6 caption
Figure 6: Left: original waveform, middle: reconstructed with same speaker-id
- p. 8 Figure 7 caption
Figure 7: First 6 frames are provided to the model
- p. 8 Figure 7 caption
Figure 7: First 6 frames are provided to the model, following frames are generated conditioned on an
Judge's note: Read page 7 and page 8. Figure 6 (page 7) shows three waveforms: original, reconstructed with the same speaker id, reconstructed with a different speaker id. Figure 7 (page 8) is two rows of video frames, 'Provided' and 'Generated', and is cited correctly in Section 4.4. Figure 6 is not cited anywhere else in the text. The pointer should be to Figure 6.
Second check: another model agreed
Report a problem with this finding
Wording The general description of VAEs says "an encoder network which parameterises a posterior distribution q(z|x) of discrete latent random variables z", but the next paragraph says "Typically, the posteriors and priors in VAEs are assumed normally distributed with diagonal covariance", i.e. continuous.
- p. 3 Section 3, first paragraph
a posterior distribution q(z|x) of discrete latent random
- p. 3 Section 3, second paragraph
Typically, the posteriors and priors in VAEs are assumed normally distributed
- p. 3 Section 3, first paragraph
posterior distribution q(z|x) of discrete latent random
- p. 3 Section 3, second paragraph
Typically, the posteriors and priors in VAEs are assumed normally distributed with diagonal covari-
Only the authors can say what is right here.
Judge's note: Read page 3 image, Section 3. The first paragraph is about what "VAEs consist of" in general and calls the latent variables discrete; the second says VAE posteriors and priors are typically Gaussian; the third introduces discrete latents as what this work adds ("In this work we introduce the VQ-VAE where we use discrete latent variables"). Section 4.1 also calls normal VAEs those "with continuous variables". The word "discrete" in the general description is therefore at odds with the paper's own account. The authors may have been anticipating their own model, so this is put as a query.
Second check: another model agreed
Report a problem with this finding
15 minor (typos and the like; most checked by the judge only)
Typo Misspelling: "generative models succesfully combining the variational autoencoder (VAE) framework" should read "successfully".
- p. 1 Section 1, last paragraph
generative models succesfully combining
- p. 1
generative models succesfully combining the variational
Judge's note: Read page 1 image, last paragraph of Section 1: the word is printed "succesfully" with one s. The paper spells it "successfully" on page 2, so this is a plain misspelling.
Second check: none (judge only)
Report a problem with this finding
Typo Malformed dashes in the abstract: "“posterior collapse” -— where the latents are ignored when they are paired with a powerful autoregressive decoder -— typically observed"; each is a hyphen followed by an em dash.
- p. 1 Abstract
“posterior collapse” -— where the latents are ignored
Judge's note: Viewed the crop of the abstract: both dashes are visibly printed as a short hyphen followed by a long dash, twice. This is in the rendered page, not an extraction artefact. A single dash was intended each time.
Second check: none (judge only)
Report a problem with this finding
Typo Sentence does not parse: "the usefulness of generic representations trained in an unsupervised fashion is still far from being the dominant approach"; the subject "usefulness" cannot be an "approach".
- p. 1 Section 1, first paragraph
the usefulness of generic representations trained in
Judge's note: Read page 1 image, first paragraph of Section 1. The sentence is printed as quoted. The meaning (using generic unsupervised representations is not yet the dominant approach) is recoverable, so this is loose wording and no more.
Second check: none (judge only)
Report a problem with this finding
Typo Subject-verb agreement errors: "a discrete latent model (VQ-VAE) perform as well as"; "the first discrete latent VAE model that get similar performance as its continuous counterparts"; "The NVIL [27] estimator use a single-sample objective"; "Learning representations with continuous features have been the focus of many previous work".
- p. 2 Section 1, contributions, second bullet
a discrete latent model (VQ-VAE) perform as well
- p. 2 Section 1, top of page
that get similar performance as its continuous counterparts
- p. 2 Section 2, second paragraph
The NVIL [27] estimator use a single-sample
- p. 1 Section 1, third paragraph
have been the focus of many previous work
Judge's note: Read pages 1 and 2. All four phrases are printed as quoted: singular subjects with plural verbs ("model ... perform", "model that get", "estimator use", "Learning ... have been") and "many previous work". Meaning is unaffected. C004, C009 and C010 each repeat one of these items.
Second check: none (judge only)
Report a problem with this finding
Typo Misspelling: "a new continuous reparemetrisation based on the so-called Concrete [25] or Gumbel-softmax [19] distribution" should read "reparametrisation".
- p. 2 Section 2, third paragraph
a new continuous reparemetrisation
- p. 2 Related Work, para 3
continuous reparemetrisation
- p. 2
a new continuous reparemetrisation based on the
Judge's note: Read page 2 image, Section 2, third paragraph: printed "reparemetrisation". The paper uses "reparameterisation" in the next paragraph and "reparametrisation" on page 3, so this is a misspelling and not a spelling convention.
Second check: none (judge only)
Report a problem with this finding
Typo "Theis et. al. [36] use scalar quantisation": stray full stop after "et"; should be "et al."
- p. 2 Section 2, last paragraph
Theis et. al.
Judge's note: Read page 2 image, last paragraph of Section 2: printed "Theis et. al." with a full stop after "et".
Second check: none (judge only)
Report a problem with this finding
Typo Figure 1 caption: "The output of the encoder z(x) is mapped to the nearest point e2"; the encoder output is z_e(x) in the figure itself and throughout the text.
- p. 4 Figure 1 caption
output of the encoder z(x) is mapped to the nearest point e2
- p. 4 Figure 1 caption
The output of the encoder z(x) is mapped to the nearest point e2.
- p. 3 Section 3.1
passed through an encoder producing output ze(x)
Judge's note: Viewed the crop of Figure 1 and its caption: the caption prints z(x) with no subscript, while the figure labels the encoder output z_e(x) (left panel and right panel) and Section 3.1 defines it as "an encoder producing output ze(x)". Unsubscripted z is the discrete latent elsewhere. The subscript was dropped; the meaning is recoverable from the figure.
Second check: none (judge only)
Report a problem with this finding
Typo Extra word: "It is has three components that are used to train different parts of VQ-VAE" should read "It has three components".
- p. 4 Section 3.2, paragraph starting 'Equation 3 specifies'
It is has three components
Judge's note: Viewed the crop and page 4 image, Section 3.2: printed "It is has three components".
Second check: none (judge only)
Report a problem with this finding
Typo Section 4.2 prints the reduction "128×128×3×8 / 32×32×9 ≈ 42.6 in bits"; the fraction evaluates to 42.67, which rounds to 42.7 (42.6 is the truncated value).
- p. 5 Section 4.2, second paragraph
≈42.6 in bits
Judge's note: Read the fraction on the page 5 image: numerator 128×128×3×8, denominator 32×32×9, printed ≈ 42.6. Recomputed 393216/9216 = 42.667. The printed value is truncated rather than rounded; with the ≈ sign this is cosmetic.
Second check: none (judge only)
Report a problem with this finding
Typo Figure 5 caption: "The reconstructions are generated by sampled from the second PixelCNN prior" should read "generated by sampling from".
- p. 7 Figure 5 caption
reconstructions are generated by sampled from
Judge's note: Read the Figure 5 caption on the page 7 image; the wording is as quoted.
Second check: none (judge only)
Report a problem with this finding
Typo Section 4.3: "sound like babbling , samples from VQ-VAE" has a stray space before the comma.
- p. 7 Section 4.3, para 4
sound like babbling ,
Judge's note: Checked the page 7 image: there is a visible space between "babbling" and the comma, so it is in the PDF and not an extraction artefact.
Second check: none (judge only)
Report a problem with this finding
Typo Figure 5 caption: "Top original images, Bottom: reconstructions" is missing the colon after "Top".
- p. 7 Figure 5 caption
Top original images, Bottom: reconstructions
Judge's note: Read the Figure 5 caption on the page 7 image; "Top" has no colon while "Bottom:" does.
Second check: none (judge only)
Report a problem with this finding
Typo Section 4.3: "(which was not used any way to train the VQ-VAE)" is missing a word; should read "not used in any way".
- p. 8 Section 4.3, last paragraph
which was not used any way to train
- p. 8 Section 4.3, last para
which was not used any way to train the
Judge's note: Read the last paragraph of Section 4.3 on the page 8 image; the wording is as quoted.
Second check: none (judge only)
Report a problem with this finding
Typo Footnote 1 on page 8: "e.g.. bi/tri-grams" has a doubled period, and "would results in higher accuracy" should read "would result in".
- p. 8 Footnote 1
e.g.. bi/tri-grams
- p. 8 Footnote 1
would results in higher accuracy
Judge's note: Read footnote 1 on the page 8 image; both slips are present as quoted.
Second check: none (judge only)
Report a problem with this finding
Typo Section 4.3: "we mapped every of the 128 possible latent values" is missing a word; should read "every one of" or "each of".
- p. 8 Section 4.3, last paragraph
we mapped every of the 128 possible latent values
Judge's note: Read the last paragraph of Section 4.3 on the page 8 image; the wording is as quoted.
Second check: none (judge only)