A Fast Learning Algorithm for Deep Belief Nets
Neural Computation, 2006 · paper · AI and machine learning
Describes a fast, greedy algorithm that learns deep belief networks one layer at a time, using "complementary priors" to handle inference difficulties. A three-hidden-layer network models handwritten digits and their labels well, and classifies digits better than the best discriminative methods.
Checked: PDF from cs.toronto.edu, fetched 4 Oct 2026. Page numbers are those of that PDF.
7 findings
Reference or label Reference "Mayraz, G. and Hinton, G. E. (2001). ... IEEE Transactions on Pattern Analysis and Machine Intelligence, 24:189–197" and the in-text citation "(Mayraz and Hinton, 2001)" give 2001 for volume 24, which the same list dates 2002 (Belongie et al., "24(4):509–522"); the year should be 2002.
- p. 10 References, Mayraz and Hinton
Mayraz, G. and Hinton, G. E. (2001)
- p. 10 References, Belongie et al.
gence, 24(4):509–522.
- p. 4 Section 3
(Mayraz and Hin- ton, 2001)
- p. 4 Section 3, in-text citation
ton, 2001)
Judge's note: The page 10 image shows both entries in the same journal and volume 24 with different years; pages 189–197 precede 509–522 in that volume. TPAMI volume 24 is the 2002 calendar-year volume, and the Mayraz and Hinton paper appeared in its February 2002 issue. The in-text citation on page 4 carries the same wrong year. The internal inconsistency is visible in the PDF; the choice of 2002 as the correct year rests on the bibliographic record.
Second check: another model agreed
Report a problem with this finding
Reference or label Reference Hinton (2002): "Neural Computation, 14(8):1711–1800"; the article occupies pages 1771–1800.
- p. 10 References, Hinton (2002)
14(8):1711–1800.
- p. 10 References, Hinton (2002)
14(8):1711–1800
Judge's note: The page 10 image shows "14(8):1711–1800". The printed range would make the article 90 pages; the published pagination of 'Training products of experts by minimizing contrastive divergence' is 1771–1800. This relies on the external bibliographic record, which cannot be checked from the PDF itself.
Second check: another model agreed
Report a problem with this finding
Reference or label Reference Freund (1995): "Information and Computation, 12(2):256 – 285"; the volume is 121, not 12.
- p. 10 References, Freund (1995)
Information and Computation, 12(2):256 – 285.
- p. 10 References, Freund (1995)
Information and Computation, 12(2):256 – 285
Judge's note: The page 10 image shows "12(2):256 – 285". Freund's 'Boosting a weak learning algorithm by majority' is Information and Computation 121(2):256–285 (1995); volume 12 of that series dates from the 1960s. This relies on the external bibliographic record.
Second check: another model agreed
Report a problem with this finding
Reference or label Reference "LeCun, Y., Bottou, L., and Haffner, P. (1998). Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324" omits co-author Y. Bengio.
- p. 10 References, LeCun et al.
LeCun, Y., Bottou, L., and Haffner, P. (1998). Gradient-based
Judge's note: The page 10 image shows three authors. The paper identified by this title, journal, volume, issue and pages is by LeCun, Bottou, Bengio and Haffner. This relies on the external bibliographic record.
Second check: another model agreed · checked by hand
Report a problem with this finding
Reference or label Reference Pearl (1988): "Probabilistic Inference in Intelligent Systems: Networks of Plausible Inference"; the book's title is "Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference".
- p. 10 References, Pearl (1988)
Probabilistic Inference in Intelligent Sys-
Judge's note: The page 10 image shows the title with "Inference" in place of "Reasoning". Publisher (Morgan Kaufmann, San Mateo), year and subtitle identify the well-known book. This relies on the external bibliographic record.
Second check: another model agreed
Report a problem with this finding
Reference or label Eq. 18 prints the pairwise term as "Ψi,j(x′i, y′i)" under a sum over i,j; the second argument should be y′j, as in Eq. 16 and 17 ("Ψi,j(xi, yj)") and in the α term of Eq. 18 itself ("αj(y′j)").
- p. 14 Appendix A, Eq. 18
Ψi,j(x′i, y′i)
- p. 14 Appendix A, Eq. 16
Ψi,j(xi, yj)
Judge's note: Checked on the page 14 image and on a 300 dpi re-render of Eq. 18 from the PDF: the subscript on y′ in the Ψ term is clearly i. Eq. 23 sets P(x(0),y(0)) = f(x(0),y(0)), which has to match Eq. 16, so the intended index is j. It is an index slip with an evident fix; the derivation is unaffected, so I rated it S2 rather than S1.
Second check: another model agreed
Report a problem with this finding
Wording Abstract: "This generative model gives better digit classification than the best discriminative learning algorithms" and introduction item 3: "outperforms discriminative methods on the MNIST database" are unqualified, but Table 1 lists discriminative methods at 0.4, 0.56, 0.8 and 0.95% against 1.25% for the generative model; the claim holds only for the permutation-invariant task.
- p. 1 Abstract
better digit classification than the best discrimi-
- p. 1 Introduction, item 3
lent generative model which outperforms discrimina-
- p. 12 Table 1
0.4
Judge's note: Viewed page 1 and Table 1 on page 12. Table 1 gives 1.25% for the generative model and lower errors for four discriminative entries on unpermuted images, including LeNet5 at 0.95% without extra data. The restriction to the permutation-invariant task without geometric knowledge is stated on page 2 (section 1) and in section 6.1, so the body is correct, but neither the abstract sentence nor item 3 carries it. Overstatement in wording, not a data error.
Second check: another model agreed
Report a problem with this finding
16 minor (typos and the like; most checked by the judge only)
Typo "a contrastive version of the wake-sleep algorithm Hinton et al. (1995) that does not suffer from the": textual citation form used where a parenthetical citation "(Hinton et al., 1995)" is needed.
- p. 2 Section 1, left column
gorithm Hinton et al. (1995) that does not suffer from the
Judge's note: The crop of page 2 shows the wording exactly as quoted, with the author names running into the sentence. Elsewhere (footnote 2, appendix B) the same citation is parenthetical.
Second check: none (judge only)
Report a problem with this finding
Typo Footnote 2 is a run-on: "used in the wake-sleep algorithm (Hinton et al., 1995) for the models described in this paper no variational approximation is required"; punctuation or a conjunction (e.g. "but") is missing after the citation.
- p. 3 Footnote 2
wake-sleep algorithm (Hinton et al., 1995) for the models described
Judge's note: The crop of page 3 shows no punctuation between the citation and "for the models described in this paper", nor after "paper". Two clauses are fused and the sentence can be misread as saying wake-sleep inference was used for this paper's models.
Second check: none (judge only)
Report a problem with this finding
Typo Figure 4 caption repeats a word: "applying Eq. 1 to the inputs received from the the current states of the visible units".
- p. 4 Figure 4 caption
received from the the current states
- p. 4
inputs received from the the current states of the visible
- p. 4
inputs received from the the current states
- p. 4
1 to the inputs received from the the current states of t
Judge's note: Checked the crop and page 4. The repeated "the the" is on one line of the Figure 4 caption, so it is in the published text and not caused by a line break or by text extraction.
Second check: another model agreed, code check
Report a problem with this finding
Typo Figure 4: the upper row of time labels is misplaced and overprinted; "t = 1", "t = 2" and a truncated "infinit" are stacked on top of each other above the right-hand hidden-unit box, while only "t = 0" is correctly placed.
- p. 4 Figure 4, labels above the top row of units
t = infinit
Judge's note: The crop and the full page 4 image both show the garbled upper labels ("t = /1 t = 2nf init") at the right end, over the break mark and the last box. The lower labels (t = 0, t = 1, t = 2, t = infinity) are fine. This may be specific to the rendering of the embedded figure in this author-hosted PDF, but it is in the file under review.
Second check: none (judge only)
Report a problem with this finding
Typo Footnote 9: "was 1.18% for the epoch on which number of training errors was smallest"; missing article, should read "on which the number of".
- p. 8 Footnote 9
for the epoch on which number of
Judge's note: The crop of page 8 shows the footnote text exactly as quoted, with no "the" before "number".
Second check: none (judge only)
Report a problem with this finding
Typo Sanger (1989) reference title is truncated: "Optimal unsupervised learning in a single-layer linear feedforward neural. Neural Networks, 2(6):459–473." The word "network" is missing after "neural".
- p. 11 References, Sanger (1989)
single-layer linear feedforward neural. Neural Networks,
- p. 11 References, Sanger (1989)
single-layer linear feedforward neural. Neural Networks
Judge's note: Page 11 image shows the title ending in "feedforward neural." followed directly by the italic journal name. The text layer agrees, so it is not an extraction artefact.
Second check: none (judge only)
Report a problem with this finding
Typo The Simard et al. (2003) reference misspells Recognition: "International Conference on Document Analysis and Recogntion (ICDAR)".
- p. 11 References, Simard et al. (2003)
ment Analysis and Recogntion (ICDAR)
- p. 11 References, Simard et al. (2003)
Document Analysis and Recogntion (ICDAR)
Judge's note: The page 11 image shows "Recogntion" in the italic venue name of the Simard, Steinkraus and Platt (2003) entry. This is a plain misspelling.
Second check: none (judge only)
Report a problem with this finding
Typo Welling, Hinton and Osindero (2003) reference has a garbled editor list: "In S. Becker, S. T. and Obermayer, K., editors". The second editor is reduced to the initials "S. T." with no surname, and the name order is mixed.
- p. 11 References, Welling et al. (2003)
In S. Becker, S. T. and Ober-
Judge's note: Page 11 image shows the entry exactly as quoted. The NIPS 15 editors are Becker, Thrun and Obermayer, so the surname Thrun is missing (that identification is from general knowledge, not from the PDF).
Second check: none (judge only)
Report a problem with this finding
Typo Table 1, last row, first column reads "Unpermuted images" with a capital U, while every other first-column entry is lower case ("unpermuted images", "permutation-invariant").
- p. 12 Table 1, last row
Unpermuted images
Judge's note: Page 12 image: the final row (Backprop in LeNet5, 0.95) is the only capitalised entry in the first column. Purely cosmetic.
Second check: none (judge only)
Report a problem with this finding
Typo Appendix A: "An intuitive way of validating of this assertion is as follows." The second "of" is superfluous ("validating this assertion").
- p. 14 Appendix A, paragraph after Eq. 17
An intuitive way of validating of this assertion
Judge's note: Page 14 image, paragraph after Eq. 17, reads exactly as quoted.
Second check: none (judge only)
Report a problem with this finding
Typo Appendix B pseudo-code variable names are inconsistent: "postopprobs = logistic(...)" and "postopstates = waketopprobs > rand(1, numtop));", but later lines use waketopstates; "-hidvis = hidvis + r*poshidstates'*(...)"; "waketopsates".
- p. 15 Appendix B, bottom-up pass
postopstates = waketopprobs > rand(1, numtop));
- p. 15 Appendix B, positive phase statistics
poslabtopstatistics = targets’ * waketopstates;
- p. 15 Appendix B, bottom-up pass
postopprobs
- p. 15 Appendix B, next line
postopstates
Judge's note: Checked pages 15 and 16 and the crops. The probabilities are stored in postopprobs, but the next line reads waketopprobs, which is never assigned. The states are stored in postopstates, but the positive statistics and the loop setup read waketopstates, which is never assigned. That same line has an extra closing parenthesis ("numtop));"), as does the matching negtopstates line in the loop. On page 16, the hidvis update starts with a stray "-" and uses poshidstates, which is never defined (the defined name is wakehidstates). The topbiases update misspells waketopstates as "waketopsates". These are naming and typing slips in pseudo-code; the intended algorithm can still be worked out.
Second check: another model agreed
Report a problem with this finding
Typo Two Appendix B pseudo-code lines have an unmatched closing parenthesis: "postopstates = waketopprobs > rand(1, numtop));" and "negtopstates = negtopprobs > rand(1, numtop));".
- p. 15 Appendix B, bottom-up pass
waketopprobs > rand(1, numtop));
- p. 15 Appendix B, Gibbs loop
negtopstates = negtopprobs > rand(1, numtop));
Judge's note: Page 15 image: both lines end in "))" with only one opening parenthesis. The analogous hid and pen lines end "rand(1, numhid);" and "rand(1, numpen);".
Second check: another model agreed
Report a problem with this finding
Typo Every comment line in the Appendix B listing is printed with a literal backslash before the percent sign, e.g. "\% UP-DOWN ALGORITHM" and "\% PREDICTIONS". MATLAB comments begin with "%"; the backslash is a LaTeX escape left in verbatim text.
- p. 15 Appendix B, code listing
\% UP-DOWN ALGORITHM
- p. 16 Appendix B, code listing
\% PREDICTIONS
Judge's note: Pages 15 and 16 images show "\%" on all comment lines, including the inline "\% to initialize loop". It is rendered in the PDF, not an extraction artefact.
Second check: none (judge only)
Report a problem with this finding
Typo Appendix B: "-hidvis = hidvis + r*poshidstates’*(data-pvisprobs);" uses "poshidstates", which is never defined. The up-pass hidden states are named "wakehidstates".
- p. 16 Appendix B, updates to generative parameters
= hidvis + r*poshidstates’*(data-pvisprobs);
- p. 15 Appendix B, bottom-up pass
wakehidstates = wakehidprobs > rand(1, numhid);
- p. 16 Appendix B, updates to generative parameters
-hidvis = hidvis + r*poshidstates’*(data-pvisprobs);
Judge's note: Page 16 image confirms "poshidstates". Searching the listing on pages 15–16 finds no assignment to it. The up-pass defines wakehidstates, and the neighbouring lines use wakepenstates and wakehidstates, so wakehidstates is intended. It is an identifier slip in pseudo-code, less serious than the ERROR rating. The stray leading minus on the same line is recorded under C027.
Second check: none (judge only)
Report a problem with this finding
Typo Appendix B: the hidvis update line begins with a stray minus sign, "-hidvis = hidvis + r*poshidstates’*(data-pvisprobs);".
- p. 16 Appendix B, updates to generative parameters
-hidvis
Judge's note: Page 16 image shows the leading "-" before hidvis under "UPDATES TO GENERATIVE PARAMETERS". No other update line has one. This is a different defect from the undefined name on the same line (C026).
Second check: none (judge only)
Report a problem with this finding
Typo Appendix B: "topbiases = topbiases + r*(waketopsates - negtopstates);" misspells the identifier; "waketopsates" should be "waketopstates".
- p. 16 Appendix B, updates to top level associative memory parameters
topbiases = topbiases + r*(waketopsates - negtopstates);
Judge's note: Page 16 image shows "waketopsates" with the "t" missing. The name used elsewhere, on page 15, is waketopstates.
Second check: another model agreed