Understanding the Difficulty of Training Deep Feedforward Neural Networks
2010 · paper · AI and machine learning
Investigates why standard gradient descent from random initialization trains deep feedforward networks poorly, examining activation functions and how activations and gradients vary across layers. The authors find logistic sigmoids unsuited to deep networks and propose an initialization scheme that speeds convergence.
Checked: PDF from proceedings.mlr.press, fetched 4 Oct 2026. Page numbers are those of that PDF.
2 findings
Substance Eq. 13 prints "∀i, Var[∂Cost/∂s^i] = [nVar[W]]^{d−i} Var[x]", but Eq. 6, which it specialises, has Var[∂Cost/∂s^d] as the factor and contains no Var[x].
- p. 5 Eq. (13), right column
nV ar[W]
- p. 5 Eq. (6), left column
We write V ar[W i′] for the shared scalar variance of all
- p. 5 Eq. 6
Var[∂Cost/∂s^i] = Var[∂Cost/∂s^d] ∏_{i'=i}^{d} n_{i'+1}Var[W^{i'}]
- p. 5 Eq. 7
∏_{i'=i}^{d-1} n_{i'+1}Var[W^{i'}]
Judge's note: Read Eqs. 5, 6, 7, 13 and 14 on the page 5 image. Eq. 6 is Var[∂Cost/∂s^i] = Var[∂Cost/∂s^d] ∏ n_{i'+1}Var[W^{i'}]; with equal widths and variances this gives [nVar[W]]^{d−i} Var[∂Cost/∂s^d]. Eq. 13 as printed ends in Var[x], the input-feature variance defined on the same page ("= Var[x]"), which belongs to the forward expression (Eq. 5). Eq. 14 carries both Var[x] and Var[∂Cost/∂s^d] as separate factors, so Var[x] is not shorthand for the output-gradient variance. No statement in the text equates the two. The second point raised here (the product limits in Eq. 6) is handled under C022.
Second check: another model agreed · checked by hand
Report a problem with this finding
Wording Eq. 6 prints the product as "∏_{i'=i}^{d} n_{i'+1}Var[W^{i'}]" (d−i+1 factors), while Eq. 7 prints the same product with upper limit d−1 and Eq. 13 has exponent d−i (d−i factors).
- p. 5 Eq. (6)
Then for a network with d layers,
- p. 5 Eq. (7)
V ar[x]V ar
- p. 5 Eq. (13)
∀i, V ar
Only the authors can say what is right here.
Judge's note: Read on the page 5 image: Eq. 6 has upper limit d; the second product in Eq. 7 has upper limit d−1 with the same factor n_{i'+1}Var[W^{i'}]; Eq. 13 has exponent d−i and Eq. 14 exponent d (i factors plus d−i). Eq. 6 is the odd one out by one factor; at i = d it would give a non-trivial factor between identical quantities. No indexing convention stated in the text reconciles the three. Which limit the authors intend (upper d−1, or lower i+1) is for them to say.
Second check: another model agreed
Report a problem with this finding
24 minor (typos and the like; most checked by the judge only)
Wording After Eq. (1): "n is the size of the previous layer (the number of columns of W)", whereas Section 4.2.1 writes "s^i = z^i W^i + b^i" and uses W^{i+1}_{k,•} in Eq. (2), a convention under which the previous layer's size is the number of rows of W.
- p. 3 After Eq. (1)
of columns of W)
- p. 5 Section 4.2.1, definitions before Eq. (2)
and si
- p. 5 Section 4.2.1, definitions before Eq. (2)
si = ziW i + bi
Judge's note: Page 3 image (text after Eq. 1) and page 5 image (definitions before Eq. 2, and Eq. 2 with the row index k for the input unit). With a row vector z multiplying W from the left, the input dimension is the row count; "columns" matches the s = Wz convention instead. The paper does not state a change of convention. No result depends on it, so this is a notation inconsistency rather than an error in a value.
Second check: another model agreed
Report a problem with this finding
Wording Figure 12 caption says the legends are "ordered from top to bottom in decreasing final error", but the MNIST legend (Sigmoid, Softsign N, Tanh, Tanh N, Softsign) and the CIFAR10 legend (Sigmoid, Softsign, Tanh, Softsign N, Tanh N) do not follow the final test errors in Table 1 (MNIST: Tanh 1.76 above Softsign N 1.72, Tanh N 1.64 equal to Softsign 1.64; CIFAR-10: Tanh 55.9 above Softsign 55.78).
- p. 8 Figure 12, MNIST panel legend and caption
schemes (ordered from top to bottom in decreasing final
- p. 7 Table 1, MNIST column
1.76
- p. 8 Figure 12 caption and MNIST panel legend (Sigmoid, Softsign N, Tanh, Tanh N, Softsign)
ordered from top to bottom in decreasing final
- p. 8 Figure 12 caption and CIFAR10 legend
ordered from top to bottom in decreasing final error
Judge's note: Read both legends on the page 8 image and Table 1 on the page 7 image. The legend orders and the table values are as the reviewers quote them, so the legend order disagrees with the Table 1 ranking for Softsign N vs Tanh on MNIST and Softsign vs Tanh on CIFAR-10. The differences are small (0.04 and 0.12 points) and the curves are nearly coincident at the right edge of both panels. The paper does not say whether Table 1 is taken at the end of the plotted curves or at a validation-selected point (the text says only that the learning rate is chosen on the validation set), so the legend may be ordered by curve end values that differ slightly from the table. I could not read the curve end values precisely enough at this resolution to confirm the reviewers' figures of about 1.71 and 1.67, so those are not part of the confirmed statement. Either form could be legitimate; only the authors can say which.
Second check: another model agreed
Report a problem with this finding
Typo Abstract: "saturated units can move out of saturation by themselves, albeit slowly, and explaining the plateaus sometimes seen when training neural networks" - the participle "explaining" is joined by "and" to a finite clause.
- p. 1 Abstract
and explaining the plateaus
Judge's note: Read on the page 1 image. The sentence is printed as quoted; "and explaining" has no parallel verb form to coordinate with. Cosmetic grammar slip, meaning is clear.
Second check: none (judge only)
Report a problem with this finding
Typo Grammar slips: "the non-linear activations functions" (abstract); "The size of the images are arbitrary" (Sec. 2.1); "the fact that an hidden unit output of 0" (Sec. 3.1); "the gradient of the cost function on the inputs biases" (Sec. 4.2.1); "may yield to ill-conditioning" (Sec. 4.3).
- p. 3 Section 3.1, hypothesis paragraph
an hidden unit
- p. 1 Abstract
non-linear activations func-
- p. 2 Section 2.1, last paragraph
The size of the images are arbitrary
- p. 6 Section 4.3, right column
may yield to ill-conditioning
Judge's note: Each phrase seen as quoted on the page images for pages 1, 2, 3 and 4; "may yield to ill-conditioning" on page 6 was checked in the text layer only. All are one-word copy-edits with no effect on meaning.
Second check: none (judge only)
Report a problem with this finding
Typo Section 2.2: "tools, flowers and fruits Figure 1 (bottom) shows randomly chosen examples." - the full stop after "fruits" is missing.
- p. 2 Section 2.2, Small-ImageNet paragraph
tools, flowers and fruits Figure 1 (bottom) shows randomly
- p. 2 Sec. 2.2, Small-ImageNet
flowers and fruits Figure 1 (bottom) shows randomly
Judge's note: Page 2 image, right column: no punctuation between "fruits" and "Figure 1"; both are on the same line, so it is not a line-break artefact.
Second check: none (judge only)
Report a problem with this finding
Typo Section 2.2: "The MNIST digits (LeCun et al., 1998a), dataset has 50,000 training images" - stray comma between the citation and "dataset".
- p. 2 Section 2.2, first sentence
The MNIST digits (LeCun et al., 1998a), dataset has
Judge's note: Page 2 image, left column, first sentence of 2.2 is printed as quoted. The comma splits the noun phrase "The MNIST digits dataset".
Second check: none (judge only)
Report a problem with this finding
Typo Section 2.2: "Small-ImageNet which is a set of tiny 37×37 gray level images dataset computed from the higher-resolution and larger set at http://www.image-net.org, with labels from the WordNet noun hierarchy." - a sentence with no main verb, and "a set of ... images dataset" is doubled.
- p. 2 Sec. 2.2
Small-ImageNet which is a set of tiny 37×37 gray level
Judge's note: Page 2 image, right column. The sentence consists of a subject and a relative clause only. Cosmetic.
Second check: none (judge only)
Report a problem with this finding
Typo Page 2 uses both "grey-scale" (Sections 2.1 and 2.2: "translation and grey-scale", "28×28 grey-scale pixel image") and "gray level" ("tiny 37×37 gray level images").
- p. 2 Sec. 2.1 and 2.2
grey-scale
- p. 2 Sec. 2.2
gray level
Judge's note: Both spellings seen on the page 2 image. A spelling-convention inconsistency only; either form is acceptable on its own.
Second check: none (judge only)
Report a problem with this finding
Typo The dataset is written "CIFAR10" in the Figure 12 caption and panel title but "CIFAR-10" in Section 2.2, Section 5 and the Table 1 header.
- p. 8 Figure 12, CIFAR10 panel legend (Sigmoid, Softsign, Tanh, Softsign N, Tanh N)
CIFAR10, for various activation functions and initialization
- p. 7 Table 1, CIFAR-10 column, rows Softsign and Tanh
55.78
- p. 8 Figure 12 caption
MNIST and CIFAR10
- p. 2 Sec. 2.2
CIFAR-10 (Krizhevsky & Hinton, 2009)
Judge's note: This cluster mixes two points. The naming inconsistency is confirmed: the Figure 12 caption and panel title on page 8 read CIFAR10, while the text on pages 2 and 6 and the Table 1 header on page 7 read CIFAR-10. Cosmetic. The other point, that the CIFAR10 legend lists Softsign above Tanh although Table 1 gives Tanh 55.9 and Softsign 55.78, is real but is the same defect already recorded under C033, so it is not repeated here.
Second check: none (judge only)
Report a problem with this finding
Typo Section 3.1: "because of its none-zero mean that induces important singular values in the Hessian" - "none-zero" for "non-zero".
- p. 3 Section 3.1, first paragraph
none-zero mean
- p. 3 Sec 3.1, first paragraph
because of its none-zero mean
- p. 3
because of its none-zero mean that induces
Judge's note: Page 3 image, left column, first paragraph of 3.1 prints "none-zero".
Second check: none (judge only)
Report a problem with this finding
Typo Section 3.1: "Inversely, the others layers have a mean activation value that is above 0.5"; page 7: "Others methods can alleviate discrepancies between layers" - "others" used as an adjective.
- p. 3 Section 3.1, paragraph after Figure 2
the others layers
- p. 7 Right column, paragraph after the bullet list
Others methods can alleviate
Judge's note: "the others layers" read on the page 3 image (right column, paragraph after Figure 2). "Others methods" checked in the page 7 text layer only (plain running text, no typographic risk).
Second check: none (judge only)
Report a problem with this finding
Typo Section 3.1: "The big surprise is that for intermediate number of hidden layers (here four), the saturation regime may be escaped" - article missing before "intermediate number".
- p. 3 Sec. 3.1
for intermediate number of hidden lay-
Judge's note: Page 3 image, right column. Printed as quoted; "an" is missing. Cosmetic.
Second check: none (judge only)
Report a problem with this finding
Typo Section 3.1 (pages 3-4): "a solution that is of poorer quality (also in terms of generalization) then those found with symmetric activation functions" - "then" for "than".
- p. 4 Top of left column (end of Section 3.1)
then those found with symmetric activation functions
- p. 3 Sec 3.1, first lines (continuing from p.3)
poorer quality (also in terms of generalization)
- p. 4
then those found with symmetric activation functions, as
Judge's note: End of page 3 and top of page 4 images: the comparative "poorer quality ... then those found" is printed with "then".
Second check: none (judge only)
Report a problem with this finding
Typo Figure 3 caption prints "Top:98 percentiles" with no space after the colon; the same caption later has "Bottom: 98 percentiles".
- p. 4 Figure 3 caption
Top:98 percentiles
Judge's note: Read the Figure 3 caption on the page 4 image: "Top:98 percentiles" is set without a space, while "Bottom: 98 percentiles" has one.
Second check: none (judge only)
Report a problem with this finding
Typo Figure 3 caption has "98 percentiles (markers alone)" (twice) while Figure 10 caption has "98 percentile (markers alone)" for the same kind of plot.
- p. 4 Figure 3 caption
98 percentiles
- p. 7 Figure 10 caption
98 percentile (markers alone)
Judge's note: Both captions read on the page images (pages 4 and 7). The wording differs between plural and singular; either form could be intended, so this is a consistency point only.
Second check: none (judge only)
Report a problem with this finding
Typo Sec. 4.2.1: "the gradient of the cost function on the inputs biases at each layer"; "inputs biases" is ungrammatical ("input biases").
- p. 4 Sec. 4.2.1
the gradient of the cost function on the inputs biases at each
Judge's note: Read on the page 4 image, right column, first paragraph of 4.2.1. The printed text is "inputs biases". Grammar slip; the meaning is recoverable.
Second check: none (judge only)
Report a problem with this finding
Typo Heading capitalisation varies: "3.2 Experiments with the Hyperbolic tangent" and "4.2 Gradients at initialization" against "3.3 Experiments with the Softsign", "4.1 Effect of the Cost Function" and "4.2.2 Gradient Propagation Study".
- p. 4 Sec. 3.2 and 4.2 headings
Experiments with the Hyperbolic tangent
- p. 4 Sec. 4.2 heading
Gradients at initialization
Judge's note: Headings read on the page 4 image; 4.2.2 heading from the text of page 6. Most headings use title case; "tangent" and "initialization" are lower case. Cosmetic only.
Second check: none (judge only)
Report a problem with this finding
Typo Sec. 4.2.1: "The variances will be expressed with respect to the input, outpout and weight initialization randomness." — "outpout" should be "output".
- p. 5 Section 4.2.1, after Eq. (3)
outpout and weight initialization randomness
- p. 5
The variances will be expressed with respect to the input, outpout and weight initialization randomness.
Judge's note: Read on the page 5 image, left column, below Eq. 3: the word is printed "outpout". Not an extraction artefact.
Second check: none (judge only)
Report a problem with this finding
Typo Sec. 4.2.1: "that the inputs features variances are the same (= Var[x])"; "inputs features variances" is ungrammatical ("input feature variances").
- p. 5 Sec. 4.2.1
that the inputs features variances are the same
Judge's note: Read on the page 5 image, left column, paragraph below Eq. 3. Printed as quoted. Grammar slip; meaning is clear.
Second check: none (judge only)
Report a problem with this finding
Typo Sec. 4.3: "having gradients of very different magnitudes at different layers may yield to ill-conditioning and slower training"; "yield to" should be "lead to" (or "yield").
- p. 6 Sec. 4.3
may yield to ill-conditioning and slower training
Judge's note: Checked in the page 6 text layer only (plain prose, not sign- or layout-sensitive); the page image was not opened. "yield to" means to give way to, which is not the intended sense. Wording slip.
Second check: none (judge only)
Report a problem with this finding
Typo Figure 12 ("Test error curves during training on MNIST and CIFAR10") is never cited in the body text; Section 5 refers only to "Figure 11" and "Table 1".
- p. 8 Figure 12 caption
Figure 12: Test error curves during training on MNIST and
- p. 6 Section 5, first paragraph
Figure 11
- p. 8 Figure 12 caption
Test error curves during training on MNIST and CIFAR10
- p. 6 Section 5
Figure 11 shows such curves with online training on Shapeset-3 × 2, while Table 1 gives final test error
Judge's note: Searched the extracted text for 'Figure 12' and 'Fig. 12': the only occurrence is the caption on page 8. Section 5 (page 6) says "Figure 11 shows such curves with online training on Shapeset-3 x 2, while Table 1 gives final test error for all the datasets studied" and later "Several conclusions can be drawn from these error curves" without pointing to Figure 12. A missing callout, not a wrong pointer.
Second check: none (judge only)
Report a problem with this finding
Typo Table 1, CIFAR-10 column prints "53.8" (Softsign N) and "55.9" (Tanh) to one decimal, while every other entry in the table has two decimals (including "15.60" with a trailing zero).
- p. 7 Table 1, CIFAR-10 column
53.8
- p. 7 Table 1, CIFAR-10 column
55.9
Judge's note: Read Table 1 on the page 7 image and the crop. The CIFAR-10 column reads 55.78, 53.8, 55.9, 52.92, 57.28; all other columns have two decimals throughout. Formatting inconsistency only; the values may simply be 53.80 and 55.90.
Second check: none (judge only)
Report a problem with this finding
Typo Section 5 reads "Others methods can alleviate discrepancies between layers during learning"; "Others" should be "Other".
- p. 7 Sec. 5
Others methods can alleviate discrepancies
Judge's note: Visible on the page 7 image, right column, first paragraph after the bullet list, and in the text layer. Plain grammatical slip.
Second check: none (judge only)
Report a problem with this finding
Typo The x-axis label of Figure 11 and of both panels of Figure 12 reads "# exemples seen"; the English spelling is "examples".
- p. 8 Figure 11 and Figure 12 x-axis labels
# exemples seen
Judge's note: The label is part of the figure images, so it is absent from the text layer. It is legible in all three plots on the page 8 image and in the enlarged crops of Figure 12.
Second check: none (judge only)