Deep Sparse Rectifier Neural Networks

• p. 9 References

  LeCun, Y., Bottou, L., Orr, G., and Muller, K. (1998).

• p. 2 Section 2.1

  hurts gradient-based optimization (LeCun et al., 1998;

• p. 5 MNIST dataset

  MNIST (LeCun et al., 1998)

• p. 5 4.1 dataset list, MNIST

  MNIST (LeCun et al., 1998): 50k/10k/10k

Judge's note: Page 9 image: both 1998 LeCun entries are present and neither has a/b suffixes. Pages 2 and 5 cite 'LeCun et al., 1998' plainly. Context lets a reader guess the intended work (Efficient backprop for optimisation, the Proc. IEEE paper for MNIST), but the labels are missing.

Second check: another model agreed

• p. 6 Table 1 caption

  Test error on networks of depth 3.

• p. 6 Main results paragraph

  Table 1 summarizes the results on networks of 3 hidden layers of 1000 hidden units each,

• p. 6 Experimental setup

  two hidden layers with respectively 4000 and 2000 units

• p. 6 Main results, first sentence

  Table 1 summarizes the results on networks of 3 hidden layers of 1000 hidden units each

Judge's note: Page 6 image. The setup paragraph does state the NORB exception ("For all experiments except on the NORB data ... three hidden layers and 1000 units per layer"; the Nair and Hinton architecture for NORB). The Table 1 caption and the 'Main results' sentence still describe all four columns as 3x1000 networks, which is wrong for NORB. Because the exception is stated a few lines earlier, this is an overgeneralised description, not a wrong result.

Second check: another model agreed

• p. 6 Main results, first bullet

  cal minima of greater or equal quality than those

• p. 6 Table 1, with pre-training

  49.52%

• p. 6 Table 1 NORB Softplus rows vs. second bullet

  contrary to what is experienced using tanh or softplus

• p. 6 Table 1

  Softplus 1.17% 49.52% 33.27% 19.19%

Judge's note: Page 6 image, Table 1. The first sub-claim ('greater or equal quality' than softplus) is not a defect. The bold marks are column-wise statistical equivalence to the best result across both conditions, and in MNIST, CIFAR10 and NISTP both rectifier and softplus (pre-trained) are bold, so the claim fits the paper's own test. The reviewer's reading that bold only compares with/without pre-training is a misreading. The NORB softplus exception to the second bullet's generalisation is real: 19.19 is greater than 17.68. NORB tanh does improve with pre-training (17.66 vs 19.29).

Second check: another model agreed

• p. 1 Introduction

  linear by part activa-

• p. 4 Section 3.1

  thus linear by parts

Judge's note: Both forms are visible in the crops. This is a minor inconsistency of wording only.

Second check: none (judge only)

• p. 8 Conclusion

  hypothesized to have 95% to 99% sparsity

• p. 2 Section 2.1

  same time to be between 1 and 4% (Lennie, 2003)

Judge's note: This is a small numerical inconsistency. The conclusion phrases it as a hypothesis, so an approximate range is plausible.

Second check: none (judge only)

• p. 9 References

  Dayan, P. and Abott, L. (2001).

• p. 2 Section 2.1

  (Dayan and Abott, 2001)

• p. 9 References

  Dayan, P. and Abott, L. (2001)

• p. 9 References

  Dayan, P. and Abott, L. (2001). Theoretical neuroscience.

Judge's note: The spelling is inconsistent within the paper, with the same L. Abbott spelled with "bb" elsewhere. Verified in the text and reference crops.

Second check: none (judge only)

• p. 5 Section 3.2

  recontruction function as:

• p. 5 Section 3.2

  recontruction function as

• p. 5 Section 3.2, before the definition of f(x,θ)

  define the linear recontruction function as

Judge's note: Checked the page 5 image (Section 3.2, left column). The word is printed 'recontruction'. The same section spells 'reconstruction' correctly elsewhere.

Second check: none (judge only)

• p. 5 Footnote 2

  can still flow through the active (non-zero), possibly

• p. 5 Footnote 2

  the active (non-zero), possibly helping

• p. 5 Footnote 2

  gradients can still flow through the active (non-zero), possibly helping

• p. 5

  through the active (non-zero), possibly helping

Judge's note: Checked footnote 2 in the page 5 image. The adjective 'active (non-zero)' has no head noun, so a word such as 'units' or 'paths' is missing. Cosmetic only.

Second check: none (judge only)

• p. 7 Section 4.2

  particarly appropriate for sparse input

• p. 7 Section 4.2, first paragraph after Figure 4

  rectifier nets are particarly appropriate for sparse input

Judge's note: Checked the page 7 image, Section 4.2, second paragraph. The word is printed 'particarly'.

Second check: none (judge only)

• p. 8 Section 4.2

  The model are stacked denoising auto-encoders

Judge's note: Checked the page 8 image, left column, second paragraph. The text is printed as quoted.

Second check: none (judge only)