Auto-Encoding Variational Bayes

• p. 3 Section 2.2, naive estimator

  f(z)∇qφ(z) log qφ(z)

Judge's note: Page 3 image: the left side is ∇_φ E_{qφ(z)}[f(z)], but both right-hand gradients carry subscripts q_φ(z) and q_φ(z^(l)). The log-derivative identity gives ∇_φ E_q[f] = E_q[f ∇_φ log q_φ], so the subscript should be φ. The Monte Carlo sum also has f(z) where f(z^(l)) is meant. This is a notation slip and the intended meaning is clear, so I rated it minor.

Second check: another model agreed

• p. 6 Section 4, para 4

  is equivalent to maximizing the conditional en-

• p. 6 Section 4 Related work, para 4

  of the mutual information is equivalent to maximizing the conditional en-

• p. 6 Related work

  Maximiz- ing (w.r.t. parameters) of the mutual information

Judge's note: Page 6 image, paragraph 4. My own working: I(X;Z) = H(X) − H(X|Z), and H(X) does not depend on the parameters, so maximizing I is equivalent to minimizing H(X|Z), i.e. maximizing −H(X|Z). The expected log-likelihood E[log p(x|z)] is ≤ −H(X|Z) (Gibbs' inequality), so it lower-bounds the negative conditional entropy, not H(X|Z). The sentence as printed has the sign reversed. The word 'negative' is evidently missing.

Second check: another model agreed

• p. 7 Figure 2 caption

  more latent variables does not result in more overfitting

• p. 7 Likelihood lower bound paragraph

  superfluous latent variables did not result in overfitting

• p. 7 Figure 2, Frey Face panels Nz=2,5,10,20

  Frey Face, Nz =20

Only the authors can say what is right here.

Judge's note: Read from Figure 2 on p.7. In the Frey Face panels the AEVB train and test curves nearly coincide at Nz=2 (about 900 vs 870). At Nz=10 and Nz=20 the train curve reaches about 1450–1500 while the test curve levels off near 1300. The test bound itself does not fall, so the authors may mean overfitting in the sense of a falling test score. Readings are by eye, so this is a wording query for the authors, not a factual correction.

Second check: another model agreed

• p. 14 Eq. (24)

  log pθ(x(i)z(i))

• p. 14 Sentence after eq. (24)

  simply denote the j-th element of vectors µ(i) and σ(i).

Only the authors can say what is right here.

Judge's note: I checked the crops of page 14. In Eq. (24) the term is printed as log pθ(x^(i) z^(i)) with no '|' between x and z, unlike q_φ(z|x) elsewhere in the paper, so the bar has evidently been lost. The equation sums over l=1..L and writes σ^(l)_{z,j}, μ^(l)_{z,j}, σ^(l)_{θ,j} and μ^(l)_{θ,j}, but it also uses x^(i) and z^(i) without defining i. The sentence that follows defines μ_j^(i) and σ_j^(i), neither of which appears in the equation. So the index notation does not match between the equation and its gloss. The missing bar has an obvious fix. Which index is intended (l or i) is for the authors to say.

Second check: another model agreed

• p. 3 Section 2.2, last sentence

  This gradient estimator exhibits exhibits very high variance

• p. 3

  (i)). This gradient estimator exhibits exhibits very high variance

Judge's note: Visible in the page 3 image at the end of Section 2.2.

Second check: code check

• p. 3 Section 2.3, after eq. (4)

  for chosing such an approriate distribution

• p. 3 Sec 2.3

  chosing such an approriate distribution

• p. 3

  approriate distribution p(ϵ)

Judge's note: Visible in the page 3 image, in the line after eq. (4).

Second check: none (judge only)

• p. 3 Eq. (6)

  z(i,l) = gφ(ϵ(i,l), x(i))

• p. 4 Eq. (7)

  ϵ(l) ∼p(ϵ)

Judge's note: Pages 3 and 4 images: both eq. (6) and eq. (7) use ϵ^(i,l) inside g_φ but define ϵ^(l) ~ p(ϵ). Eq. (5) and the text on page 4 use ϵ^(l). This is a minor indexing inconsistency and the meaning can be recovered.

Second check: none (judge only)

• p. 4 Section 2.3, after eq. (8)

  Derivatives ∇θ,φ eL(θ; XM) can be taken

Judge's note: Page 4 image: the printed text is ∇_{θ,φ} L̃(θ; X^M). Eq. (8) and Algorithm 1 use L̃^M(θ, φ; X^M). This is a notation slip.

Second check: none (judge only)

• p. 4 Section 2.3, last paragraph

  while the second term is a an expected negative reconstruction error

• p. 4 Sec 2.3, last paragraph

  a an expected negative reconstruction error

Judge's note: Page 4 image, last paragraph of Section 2.3: both slips are present.

Second check: none (judge only)

• p. 4 Section 2.4 / eq. (4)

  z = gφ(ϵ, x)

• p. 5 Section 2.4 top and Section 3

  gφ(x(i), ϵ(l)) = µ(i) + σ(i) ⊙ϵ(l)

Judge's note: Page 4 images show g_φ(ϵ, x). The page 5 image shows g_φ(x, ϵ^(l)) at the top of the page and g_φ(x^(i), ϵ^(l)) in Section 3. The notation is inconsistent but the meaning is clear.

Second check: none (judge only)

• p. 5 Section 2.4, univariate Gaussian example

  let z ∼p(z|x) = N(µ, σ2)

Judge's note: Page 5 image: the example is generic and has no subscript, so it can be read as an arbitrary conditional. It is still inconsistent with the paper's convention (p_θ(z|x) is the true posterior, q_φ(z|x) the approximation being reparameterised). I treat it as a minor notational observation.

Second check: none (judge only)

• p. 5 Section 3, para 2

  posterior takes on a ap-

• p. 5 Sec 3

  takes on a ap-

Judge's note: Page 5 image, Section 3, second paragraph.

Second check: none (judge only)

• p. 6 Section 4, para 4

  the negative reconstrution error

• p. 6 Section 4, last para

  directed proabilistic models

Judge's note: Page 6 image: 'reconstrution' appears in paragraph 4 and 'proabilistic' in the last paragraph (the [RMW14] sentence).

Second check: none (judge only)

• p. 6 Section 4, para 2

  introduced a control variate schemes

• p. 6 Section 4 Related work, para 2

  introduced a control variate schemes to reduce

Judge's note: Page 6 image, Section 4, second paragraph.

Second check: none (judge only)

• p. 6 Related work

  negative reconstrution error

• p. 6

  i.e. the negative reconstrution error

Judge's note: The page image (crop C016-a) shows 'reconstrution'. The same word is spelled correctly elsewhere, e.g. 'negative reconstruction error' on p.3.

Second check: none (judge only)

• p. 7 Figure 2 caption

  with a Intel Xeon CPU

Judge's note: Visible in the caption on p.7 (crop C018-a).

Second check: none (judge only)

• p. 9 References, [HDFN95]

  The” wake-sleep” algorithm

Judge's note: The p.9 page image shows a closing-quote glyph attached to 'The', then a space before 'wake'.

Second check: none (judge only)

• p. 10 Appendix B, first integral

  qθ(z) log p(z) dz

• p. 11 Appendix B, second integral and KL

  qθ(z) log qθ(z) dz

Judge's note: The p.10 and p.11 images show a bold θ subscript in all the integrands, while the text and the left-hand side −D_KL((q_φ(z)||p_θ(z)) use φ. In this paper θ denotes the generative parameters. The derivation is otherwise unaffected.

Second check: none (judge only)

• p. 11 Appendix B, 'Therefore:' line

  −DKL((qφ(z)||pθ(z))

Judge's note: The crop shows two opening parentheses and one closing one.

Second check: none (judge only)

• p. 11 Appendix D, first para

  (less then 5 dimensions)

• p. 11 Appendix D

  less then 5 dimensions

Judge's note: Visible on the p.11 page image.

Second check: none (judge only)

• p. 12 Appendix D estimator and derivation, last line

  pθ(z)pθ(x(i)|z(l))

Judge's note: Both the estimator and the last line of its derivation on p.12 show p_θ(z) in a sum over l, while the likelihood carries z^(l). In the integral that comes just before, p_θ(x^(i),z) = p_θ(z)p_θ(x^(i)|z), so in the Monte Carlo sum the prior must be evaluated at z^(l). The meaning is clear, so this is a notation slip.

Second check: none (judge only)

• p. 12 Appendix E

  weight updates steps using the acquired sample

Judge's note: Visible on the p.12 page image.

Second check: none (judge only)

• p. 13 Appendix F.1

  pα(θ) = N(z; 0, I)

Judge's note: The crop shows N(z;0,I) for p_α(θ), straight before the identical expression for p_θ(z). This looks like a copy-paste slip; the intended meaning is clear.

Second check: none (judge only)

• p. 13 Appendix F.1

  approximatily Gaussian

Judge's note: The crop of page 13 (Appendix F.1) shows "assume that the true posteriors are approximatily Gaussian with an approximately diagonal covariance". This is a misspelling. The correct spelling 'approximately' appears in the same sentence.

Second check: none (judge only)