Dropout: A Simple Way to Prevent Neural Networks from Overfitting
2012 · paper · AI and machine learning
Presents dropout, a technique that randomly drops units from a neural network during training to reduce overfitting. The authors report it improves performance over other regularization methods and gives state-of-the-art results on many benchmarks in vision, speech, text classification and computational biology.
Checked: PDF from jmlr.org, fetched 4 Oct 2026. Page numbers are those of that PDF.
7 findings
Substance Table 1 lists the CIFAR-10/100 Training Set as "60K" (Test Set "10K"), but Appendix B.3 says the data sets "consists of 50,000 training and 10,000 test images each"; 60K is the total, not the training set.
- p. 8 Table 1, row CIFAR-10/100, column Training Set
CIFAR-10/100
- p. 26 Appendix B.3, first sentence
consists of 50,000 training and 10,000 test images
- p. 8 Table 1, row CIFAR-10/100, Training Set column
60K
- p. 26 B.3, first sentence
The CIFAR-10 and CIFAR-100 data sets consists of 50,000 training and 10,000 test images
Judge's note: Page 8 image: the CIFAR-10/100 row reads 60K under Training Set and 10K under Test Set. Page 26 image: B.3 gives 50,000 training and 10,000 test images each, with 5,000 of the training images used for validation. Other rows (MNIST 60K/10K, SVHN 600K/26K, matching B.2) give training size only, and nothing in the paper defines the column as a total. The same B.3 sentence also has a grammar slip ("data sets consists"), S4.
Second check: another model agreed
Report a problem with this finding
Substance Table 1 gives ImageNet dimensionality as "65536 (256 × 256 color)", which omits the three colour channels, whereas the SVHN and CIFAR rows "3072 (32 × 32 color)" include them.
- p. 8 Table 1, row ImageNet (ILSVRC-2012), column Dimensionality
65536 (256 × 256 color)
- p. 8 Table 1, rows SVHN and CIFAR-10/100
3072 (32 × 32 color)
Only the authors can say what is right here.
Judge's note: Page 8 image confirms both entries. 32*32*3 = 3072 as printed, so the column counts channels for colour images; 256*256 = 65536 is the pixel count only, and with three channels it would be 196608. The paper gives no explanation for the ImageNet row using a different convention. Only the authors can say whether the number or the description is intended.
Second check: another model agreed
Report a problem with this finding
Substance Section 6.2 describes TIMIT as "consists of recordings from 680 speakers covering 8 major dialects of American English"; the standard description of the corpus is 630 speakers.
- p. 12 Section 6.2, first paragraph
consists of recordings from 680 speakers covering 8 major dialects
Only the authors can say what is right here.
Judge's note: Rests on outside knowledge: that the TIMIT corpus has 630 speakers (8 dialect regions, ten sentences each). The page 12 image does print "680 speakers". The paper gives no other speaker count (searched the text, including Appendix B.4), so this cannot be verified from the files; the authors should confirm.
Second check: another model agreed
Report a problem with this finding
Substance Text says "for an 8-layer net the error reduces from 20.5% to 19.7%", but Table 7 lists "DBN-pretrained NN (8 layers) (Mohamed et al., 2010)" at 20.7; 20.5 is the mcRBM-DBN-pretrained 5-layer row.
- p. 13 Section 6.2, first paragraph on the page
for an 8-layer net the error reduces from 20.5% to 19.7%
- p. 13 Table 7, row DBN-pretrained NN (8 layers)
DBN-pretrained NN (8 layers) (Mohamed et al., 2010)
- p. 13 Section 6.2, first paragraph on page
Similarly, for an 8-layer net the error reduces from 20.5% to 19.7%.
- p. 13 Table 7, row 'DBN-pretrained NN (8 layers) (Mohamed et al., 2010)'
20.7
Judge's note: Page 13 image: Table 7 has the DBN-pretrained 8-layer net at 20.7, the mcRBM-DBN-pretrained 5-layer net at 20.5, and the 8-layer net with dropout at 19.7. The preceding sentence for the 4-layer net (22.7 to 19.7) matches its table rows exactly. No other 8-layer baseline of 20.5 appears anywhere in the text. The text baseline should be 20.7 per the table.
Second check: another model agreed
Report a problem with this finding
Reference or label Section 8.1: the joint distribution is "exp(v⊤Wh + a⊤h + b⊤v)" (a with hidden units, b with visible units), but the conditionals use the opposite assignment: "P(hj = 1|rj, v) = 1(rj = 1)σ(bj + Σi Wij vi)" and "P(vi = 1|h) = σ(ai + Σj Wij hj)".
- p. 19 Section 8.1, joint distribution
exp(v⊤Wh + a⊤h + b⊤v)
- p. 19 Section 8.1, last equation on the page, P(hj = 1|rj, v)
P(hj = 1|rj, v)
- p. 20 Section 8.1, equation for P(vi = 1|h)
P(vi = 1|h)
- p. 19 Section 8.1, P(hj = 1|rj, v)
bj +
Judge's note: Checked on the page 19 and page 20 images. Both joint expressions (the RBM and the Dropout RBM) print a⊤h + b⊤v. The hidden-unit conditional at the foot of page 19 uses b_j and the visible-unit conditional on page 20 uses a_i. Under the printed joint the conditionals would be σ(a_j + Σ_i W_ij v_i) and σ(b_i + Σ_j W_ij h_j). The only definition is θ = {W, a, b}, with no redefinition between the equations. It is a symbol-labelling inconsistency; swapping the symbols in either place fixes it.
Second check: another model agreed
Report a problem with this finding
Reference or label Reference list entry for Vincent, Larochelle, Lajoie, Bengio and Manzagol, "Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion", gives the venue as "In Proceedings of the 27th International Conference on Machine Learning, pages 3371–3408. ACM, 2010."; the page range 3371–3408 appears to belong to the journal article (Journal of Machine Learning Research 11, 2010), not to ICML proceedings.
- p. 30 References, Vincent, Larochelle, Lajoie, Bengio and Manzagol
In Proceedings of the 27th International Conference on Machine Learning, pages 3371–3408. ACM, 2010.
Only the authors can say what is right here.
Judge's note: Rests on outside knowledge: that this title was published in Journal of Machine Learning Research 11:3371–3408 (2010) and not in the ICML 2010 proceedings. Viewed the crop of PDF page 30: the entry reads exactly as quoted. Internal evidence supports the finding: the range spans 38 pages and starts at page 3371, whereas the neighbouring ICML entries on the same page run 8–9 pages with low page numbers (1096–1103, 118–126), so the range is a journal-volume pagination rather than a conference one. The true venue cannot be verified from the files, hence author_query.
Second check: another model agreed
Report a problem with this finding
Wording Section 7.3: "Figure 9b shows the test error obtained as a function of p. We notice that the magnitude of errors for small values of p has reduced by a lot compared to Figure 9a (for p = 0.1 it fell from 2.7% to 1.7%)"; the quoted values match the training-error curves at p = 0.1 in Figure 9, whereas the test-error curves there are at about 3.2% (9a) and about 2.4% (9b).
- p. 17 Section 7.3, last paragraph
for p = 0.1 it fell from 2.7% to 1.7%
- p. 17 Figure 9a and 9b, points at p = 0.1
Test Error
- p. 17 Section 7.3, last paragraph
(for p = 0.1 it fell from 2.7% to 1.7%)
- p. 17 Figure 9a/9b at p = 0.1
Test Error / Training Error
Only the authors can say what is right here.
Judge's note: Read Figure 9 on the page 17 image. At p = 0.1, panel (a) has the green test-error point at about 3.2 and the blue training-error point at about 2.7; panel (b) has test at about 2.4 and training at about 1.7. The parenthesis therefore gives training errors, unlabelled, straight after a sentence about test error. "Magnitude of errors" could loosely cover both curves, so this is graded as an unlabelled or ambiguous quantity rather than a wrong value; the authors can say which curve was meant. Values are read by eye from the plot, but the two curves are clearly separated at p = 0.1.
Second check: another model agreed
Report a problem with this finding
21 minor (typos and the like; most checked by the judge only)
Typo Figure 2 caption: "The output at test time is same as the expected output at training time" is missing "the" ("is the same as").
- p. 3 Figure 2 caption
The output at test time is same as the expected output
Judge's note: Read the Figure 2 caption on the page 3 image. The article is missing; the body text lower on the same page has "is the same as the actual output".
Second check: none (judge only)
Report a problem with this finding
Typo Stray comma between subject and verb: "A neural net with n units, can be seen as a collection of 2^n possible thinned neural networks."
- p. 3 Sec. 1, paragraph after Figure 2
A neural net with n units, can be seen as a collection of 2n possible thinned neural networks.
Judge's note: Page 3 image shows the comma after "n units". The "2n" in the quote is the text layer's rendering of 2^n; the image shows the superscript correctly, so only the comma is a defect.
Second check: none (judge only)
Report a problem with this finding
Typo Section 4 feed-forward equations print the layer output as y^l and ỹ^l (no parentheses) in "z_i^(l+1) = w_i^(l+1) y^l + b_i^(l+1)" and "z_i^(l+1) = w_i^(l+1) ỹ^l + b_i^(l+1)", while the text and the adjacent line use y^(l) and ỹ^(l).
- p. 5 Section 4, first equation of the standard network and third equation of the dropout network
z(l+1)
Judge's note: Page 5 image: in both z equations the superscript l on y and ỹ has no parentheses, while the line directly above reads ỹ^(l) = r^(l) * y^(l) and the text defines y^(l). Notational inconsistency, not an extraction artefact.
Second check: none (judge only)
Report a problem with this finding
Typo Missing verb: "We note that it possible to obtain better results" should read "it is possible".
- p. 9 Section 6.1.1, second paragraph on the page
We note that it possible to obtain better results
Judge's note: Seen on the page 9 image, Section 6.1.1, second paragraph.
Second check: none (judge only)
Report a problem with this finding
Typo Duplicated word: "errors as seen as by the two separate clusters of trajectories" should read "as seen by".
- p. 9 Section 6.1.1, paragraph beside Figure 4
errors as seen as by the two separate clus-
- p. 9 Sec. 6.1.1, paragraph beside Figure 4
as seen as by the two separate clus-
Judge's note: Seen on the page 9 image, in the paragraph beside Figure 4.
Second check: none (judge only)
Report a problem with this finding
Typo Em dash in place of a hyphen in a compound modifier: "DBM—pretrained dropout nets achieve a test error of".
- p. 9 Section 6.1.1, second paragraph on the page
DBM—pretrained dropout nets achieve a test error of
Judge's note: Page 9 image shows a long dash between DBM and pretrained. Table 7 uses the hyphenated form "DBN-pretrained". Cosmetic.
Second check: none (judge only)
Report a problem with this finding
Typo Wrong verb form: "used Bayesian hyperparameter optimization to obtained an error rate of 14.98%" should read "to obtain".
- p. 10 Section 6.1.3
used Bayesian hyperparameter optimization to obtained an error rate of 14.98%
- p. 10 Section 6.1.3
to obtained an error rate of 14.98%
Judge's note: Seen on the page 10 image, Section 6.1.3.
Second check: none (judge only)
Report a problem with this finding
Typo Subject-verb disagreement: "A 4-layer net pretrained with a stack of RBMs get a phone error rate of 22.7%" should read "gets".
- p. 13 Sec. 6.2, first paragraph
A 4-layer net pretrained with a stack of RBMs get a phone error rate of 22.7%
Judge's note: Seen on the page 13 image, first paragraph.
Second check: none (judge only)
Report a problem with this finding
Typo Inconsistent term: "occurrence of alternate splicing in certain tissues" on page 14, against "alternative splicing" everywhere else (page 13, Table 1, Table 8, Appendix B.6).
- p. 14 Sec. 6.4, top of page
occurrence of alternate splicing in certain tissues
- p. 13 Sec. 6.4
predict the occurrence of alternative splicing based on RNA features
Judge's note: Page 14 image shows "alternate splicing"; page 13 has "alternative splicing" twice in the preceding sentences, and the Table 8 caption and B.6 use "alternative". Both forms occur in the literature, so this is a consistency point.
Second check: none (judge only)
Report a problem with this finding
Typo Figure 8 caption has a duplicated word: "most units have a smaller mean mean activation of about 0.7".
- p. 16 Figure 8 caption
units have a smaller mean mean activation of about 0.7
- p. 16 Figure 8 caption, Right
smaller mean mean activation of about 0.7
- p. 16 Figure 8 caption
a smaller mean mean activation of about 0.7
- p. 16
hat most units have a smaller mean mean activation of about
Judge's note: Read the Figure 8 caption on the page 16 image. "mean mean" is printed on one line, so it is not a line-break artefact. The parallel sentence for the left panel reads "a mean activation of about 2.0".
Second check: code check
Report a problem with this finding
Typo Section 8.1: the displayed equation ends with a full stop and is followed by a capitalised fragment, "Where θ = {W, a, b} represents the model parameters and Z is the partition function."
- p. 19 Sec. 8.1, after RBM distribution
Where θ = {W, a, b} represents the model parameters and Z is the partition function.
Judge's note: Seen on the page 19 image: the equation ends "+ b⊤v)." and the next line starts with an indented, capitalised "Where". It is a punctuation and capitalisation slip that is common in mathematical writing, and the meaning is unaffected.
Second check: none (judge only)
Report a problem with this finding
Typo Section 8.1: missing article in "The distribution over v conditioned on h is same as that of an RBM" ("is the same as").
- p. 20 Sec. 8.1, after Figure 12
The distribution over v conditioned on h is same as that of an RBM
Judge's note: Read on the page 20 image, below Figure 12. The same omission recurs two lines later in "is same as the distribution that an RBM would impose".
Second check: none (judge only)
Report a problem with this finding
Typo Section 9.1: "Let X ∈ R^(N×D) be a data matrix of N data points. y ∈ R^N be a vector of targets." The second sentence has no governing "Let" and is a fragment.
- p. 22 Sec. 9.1
y ∈RN be a vector of targets.
Judge's note: Read on the page 22 image. The full stop after "data points" cuts "y ∈ R^N be a vector of targets" off from "Let"; the fix is "and" or a second "Let".
Second check: none (judge only)
Report a problem with this finding
Typo Appendix A.2: missing word in "Another way to reduce the effect the noise is to use a high momentum" ("the effect of the noise").
- p. 25 Appendix A.2
Another way to reduce the effect the noise is
Judge's note: Read on the page 25 image; "of" is missing in print.
Second check: none (judge only)
Report a problem with this finding
Typo A stray full stop is printed on a line of its own between the heading "Appendix B. Detailed Description of Experiments and Data Sets" and the first paragraph.
- p. 25 Directly below the heading "Appendix B. Detailed Description of Experiments and Data Sets"
Appendix B. Detailed Description of Experiments and Data Sets
- p. 25 Appendix B heading
Appendix B. Detailed Description of Experiments and Data Sets .
Judge's note: Visible on the page 25 image: a lone "." at the left margin under the Appendix B heading, before "This section describes...". It is in the print, not an extraction artefact.
Second check: none (judge only)
Report a problem with this finding
Typo Appendix A.2: subject-verb disagreement in "Using high learning rate and/or momentum significantly speed up learning." ("speeds up"; also "a high learning rate").
- p. 25 A.2, last sentence
Using high learning rate and/or momentum significantly speed up learning.
Judge's note: Read on the page 25 image. The gerund subject takes a singular verb. Grammar only; the meaning is clear.
Second check: none (judge only)
Report a problem with this finding
Typo Appendix A.4: stray comma between subject and verb in "p = 1, implies no dropout and low values of p mean more dropout."
- p. 25 A.4
p = 1, implies no dropout
Judge's note: Read on the page 25 image; the comma after "p = 1" is in the print.
Second check: none (judge only)
Report a problem with this finding
Typo Appendix B.3: wrong word in "rotate it onto its principle components" ("principal components").
- p. 26 Appendix B.3, last line of the page
rotate it onto its principle components
Judge's note: Read on the last text line of the page 26 image.
Second check: none (judge only)
Report a problem with this finding
Typo Appendix B.3: missing word in "Global contrast normalization means that for image and each color channel in that image, we compute the mean" (presumably "for each image and each color channel").
- p. 26 Appendix B.3
means that for image and each color channel in that image
- p. 26 B.3
for image and each color channel in that image
Judge's note: Read on the page 26 image; "for image and each color channel" is as printed.
Second check: none (judge only)
Report a problem with this finding
Typo Appendix B.3: subject-verb disagreement in "The CIFAR-10 and CIFAR-100 data sets consists of 50,000 training and 10,000 test images each." ("consist").
- p. 26 Appendix B.3, first sentence
The CIFAR-10 and CIFAR-100 data sets consists of
Judge's note: Read on the page 26 image, first sentence of B.3.
Second check: none (judge only)
Report a problem with this finding
Typo Appendix B.3: "The network architecture and dropout rates are same as that for SVHN" lacks the article and has a singular pronoun for a plural subject ("are the same as those for SVHN").
- p. 27 B.3, first line
The network architecture and dropout rates are same as that for
- p. 27 Appendix B.3, last sentence (top of page)
dropout rates are same as that for
Judge's note: Read at the top of the page 27 image.
Second check: none (judge only)