Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
arXiv (Cornell University), 2015 · paper · AI and machine learning
Introduces batch normalization, which normalizes layer inputs for each training mini-batch to reduce "internal covariate shift". It allows higher learning rates, matched an image classifier's accuracy with 14 times fewer training steps, and an ensemble reached 4.9% top-5 ImageNet validation error.
Checked: PDF from arxiv.org, fetched 4 Oct 2026. Page numbers are those of that PDF.
4 findings
Substance Sec. 4.2 says the 5×5 convolutions "are replaced by two consecutive layers of 3 × 3 convolutions with up to 128 filters", but Figure 5's 'double #3×3' column lists 160, 192, 256 and 224.
- p. 6 Sec. 4.2, para 1
ers of 3 × 3 convolutions with up to 128 filters
- p. 11 Figure 5, double #3×3 column
256
Only the authors can say what is right here.
Judge's note: The page 6 text says "up to 128 filters". The Figure 5 table on page 11 gives double #3×3 widths of 96, 96, 96, 128, 128, 160, 192, 256, 224 and 224, and the double #3×3 reduce column reaches 192. Neither column is capped at 128. Only the authors can say whether the text or the table is intended.
Second check: another model agreed
Report a problem with this finding
Substance Figure 5, inception (4c): the branches #1×1=160, #3×3=160, double #3×3=160 and 'avg + 128' sum to 608, but the output size is printed as "14×14×576".
- p. 11 Figure 5, row inception (4c)
14×14×576
Only the authors can say what is right here.
Judge's note: Checked the table on the page 11 image. For every other non-pass-through module, output depth equals #1×1 + #3×3 + double #3×3 + pool projection: 3a 256, 3b 320, 4a 576, 4b 576, 5a 1024, 5b 1024. For 3c, 128+128+64+96 = 416 ≠ 576. For 4e, 192+256 plus a pass-through of 576 = 1024 is consistent with 4d outputting 576 (the 3c and 4e checks were added for this cluster only). Row 4c gives 608, so either the 576 or one of the branch widths (likely the pool projection) is wrong. Only the authors can say which.
Second check: another model agreed
Report a problem with this finding
Substance Figure 5, inception (4d): the branches #1×1=96, #3×3=192, double #3×3=192 and 'avg + 128' sum to 608, but the output size is printed as "14×14×576".
- p. 11 Figure 5, row inception (4d)
14×14×576
Only the authors can say what is right here.
Judge's note: Checked on the page 11 image. Same summation convention as rows 3a, 3b, 4a, 4b, 5a and 5b. The 4e row (192+256 plus pass-through = 1024) implies that 4d outputs 576, so a branch width in 4d is probably misprinted. This is a separate row from C023 and needs its own fix.
Second check: another model agreed
Report a problem with this finding
Wording Sec. 1 mini-batch gradient printed as "1/m ∂ℓ(x_i,Θ)/∂Θ" with no Σ_{i=1}^m, so the index i is left free.
- p. 1 Sec. 1, display equation after 'by computing'
1/m ∂ℓ(xi, Θ)/∂Θ
Judge's note: The page 1 image shows the displayed fraction 1/m ∂ℓ(x_i,Θ)/∂Θ with no summation sign. The loss just above (1/N Σ) and the Θ2 update in the right column (α/m Σ_{i=1}^m) both include the sum, so the omission here is a typesetting slip. The intended meaning is clear from context, so this is rated S3.
Second check: another model agreed
Report a problem with this finding
17 minor (typos and the like; most checked by the judge only)
Wording Text says Figure 3 shows "the maximum validation accuracy reached by the network and the number of steps to reach it", but Figure 3 has only 'Steps to 72.2%' and 'Max accuracy' columns.
- p. 7 Sec. 4.2.2, para after model list
racy reached by the network and the number of steps to
- p. 7 Figure 3 table
Steps to 72.2%
- p. 7 Sec. 4.2.2, paragraph describing Figure 3
racy reached by the network and the number of steps to reach it
- p. 7 Sec 4.2.2
as well as the maximum validation accuracy reached by the network and the number of steps to reach it
Only the authors can say what is right here.
Judge's note: The Figure 3 table on page 7 has the columns Model, Steps to 72.2% and Max accuracy, and its caption lists only those. There is no steps-to-maximum column. The 6·10^6 figure for BN-x30 appears only in the text. The text describes content that the figure does not contain.
Second check: another model agreed
Report a problem with this finding
Wording "set new state-of-the-art by a healthy margin" / "better than the best known system on ImageNet, by a significant margin", while the text names He et al. (4.94% top-5) as the current best against the ensemble's 4.82%, and Fig. 4 omits He et al.
- p. 7 Sec. 4.2.3
lows us to set new state-of-the-art by a healthy margin
- p. 7 Sec. 4.2.3
The latter reports the top-5 error of 4.94%
- p. 8 Conclusion, 2nd paragraph
than the best known system on ImageNet, by a significant margin.
- p. 8 Figure 4 caption
comparison with previous state of the art
Only the authors can say what is right here.
Judge's note: Sec. 4.2.3 on page 7 names He et al. as reporting 4.94% top-5 test error and gives the BN ensemble's test error as 4.82%. Figure 4 on page 8 compares only with the GoogLeNet and Deep Image entries (best 5.98%) and does not list He et al. The 'healthy/significant margin' wording therefore rests on a table that leaves out the comparator the text itself identifies as the best. The authors should clarify which comparison the margin wording refers to.
Second check: another model agreed
Report a problem with this finding
Wording Figure 5: the stride-2 modules inception (3c) "28×28×576" and (4e) "14×14×1024" are listed at their pre-stride spatial size, while the following modules are 14×14 and 7×7.
- p. 11 Figure 5, rows inception (3c) and (4e)
28×28×576
- p. 10 Appendix bullet
stride-2 con-
Only the authors can say what is right here.
Judge's note: Checked on the page 11 image. The appendix (page 10) says stride-2 convolution/pooling is applied before the filter concatenation in 3c and 4e, so these modules output 14×14 and 7×7. The other strided rows (conv 7×7/2, max pool 3×3/2) list the post-stride size, so the 'output size' column is internally inconsistent. The data are not otherwise affected.
Second check: another model agreed
Report a problem with this finding
Wording Figure 5: the stem row 'convolution 3×3/1 56×56×192' lists a #3×3 reduce of 64 and a #3×3 of 192 (two weight layers) but gives depth "1".
- p. 11 Figure 5, row 'convolution 3×3/1'
56×56×192
Judge's note: Checked on the page 11 image. Inception rows count the reduce layer in their depth (depth 3 = reduce + two 3×3). Under that convention, a row with a 64-channel reduce plus a 3×3 convolution has depth 2. The appendix defers interpretation to Szegedy et al. 2014, which is not in the files. Minor table-consistency point.
Second check: another model agreed
Report a problem with this finding
Typo Stray article: "We want to a preserve the information in the network".
- p. 3 Sec. 2, last paragraph
We want to a preserve the information
Judge's note: The crop of page 3 shows "We want to a preserve". The 'a' is extraneous.
Second check: none (judge only)
Report a problem with this finding
Typo Subject–verb disagreement: "The distributions of values of any x̂ has the expected value of 0".
- p. 4 Sec 3, paragraph after Algorithm 1
The distri- butions of values of any bx has the expected value of 0
Judge's note: The crop of page 4 shows plural 'distributions' with singular 'has'. It should read 'distribution ... has' or 'distributions ... have'.
Second check: none (judge only)
Report a problem with this finding
Typo Ungrammatical clause in Sec. 3.4: "and the training network no longer producing deterministic values".
- p. 5 Sec. 3.4
no longer produc-
- p. 5 Sec 3.4
the training network no longer produc- ing deterministic values
Judge's note: The crop of page 5 confirms the wording. The coordinated clause has no finite verb; it should read 'no longer produces'.
Second check: none (judge only)
Report a problem with this finding
Typo Garbled phrase in Sec. 3.3: "larger and suboptimal changes in activations in gradients" (should be 'activations and gradients').
- p. 5 Sec. 3.3, first paragraph
timal changes in activations in gradients
Judge's note: The crop of page 5 confirms "changes in activations in gradients". A conjunction appears to be missing.
Second check: none (judge only)
Report a problem with this finding
Typo Run-in heading lacks a period: "Remove Local Response Normalization While Inception...".
- p. 6 Sec. 4.2.1
Remove Local Response Normalization While
- p. 6 Sec 4.2.1
Remove Local Response Normalization While Incep-
Judge's note: On page 6, every other italic run-in heading in Sec. 4.2.1 ends with a period. This one does not.
Second check: none (judge only)
Report a problem with this finding
Typo Sec. 4.1 writes the hidden-layer output as "y = g(Wu+b)", whereas Sec. 3.2 uses z = g(Wu+b) and y denotes the BN output.
- p. 6 Sec. 4.1
Each hidden layer computes y = g(Wu+b)
Judge's note: Page 6 shows y = g(Wu+b). Page 5 shows z = g(Wu + b), and y is used for the BN transform output. This is a minor notation inconsistency; the meaning is clear from context.
Second check: none (judge only)
Report a problem with this finding
Typo Sigmoid written "g(t) = 1/(1+exp(−x))": the argument is t on the left and x on the right.
- p. 7 Sec. 4.2.2, BN-x5-Sigmoid
g(t) =
Judge's note: Page 7 (Sec. 4.2.2, BN-x5-Sigmoid) shows g(t) on the left and exp(−x) in the denominator. This is a dummy-variable mismatch.
Second check: none (judge only)
Report a problem with this finding
Typo "trained on the LSVRC2012 training data" omits the 'I' of ILSVRC, which the paper uses elsewhere ("ILSVRC server").
- p. 7 Sec. 4.2.2, first sentence
LSVRC2012 training data
Judge's note: Page 7 shows "LSVRC2012" in Sec. 4.2.2, while Sec. 4.2.3 on the same page twice writes "ILSVRC server".
Second check: none (judge only)
Report a problem with this finding
Typo Sigmoid defined as "g(t) = 1/(1+exp(−x))": the argument is t on the left and x on the right.
- p. 7 Sec 4.2.2, BN-x5-Sigmoid
g(t) =
Judge's note: The page 7 image (Sec 4.2.2, BN-x5-Sigmoid) shows g(t) on the left-hand side and exp(−x) in the denominator. This is a notation slip; it should be exp(−t), or g(x).
Second check: none (judge only)
Report a problem with this finding
Typo Figure 4: the Deep Image high-res top-1 error is printed as "24.88" with no % sign, unlike every other error entry.
- p. 8 Figure 4, Deep Image high-res row
24.88
Judge's note: The page 8 image confirms that 24.88 lacks the % sign while all other numeric entries in the Top-1 and Top-5 columns carry one. Cosmetic.
Second check: none (judge only)
Report a problem with this finding
Typo Appendix: "GoogleNet archictecture" contains a misspelling ('architecture'), and the model name is inconsistent with 'GoogLeNet' in the next sentence.
- p. 9 Appendix, first paragraph
GoogleNet archictecture
Judge's note: The page 9 image shows 'GoogleNet archictecture'; the following sentence reads 'GoogLeNet model'.
Second check: none (judge only)
Report a problem with this finding
Typo LeCun et al. 1998b reference: the editors are given as "In Orr, G. and K., Muller (eds.)", which is malformed (the same entry's author list has 'Muller, K.').
- p. 9 References, LeCun 1998b
In Orr, G. and K., Muller (eds.)
Judge's note: The page 9 image confirms the text. The inconsistency is inside the entry itself (author list 'Muller, K.' against editor 'K., Muller'), so no outside knowledge is needed.
Second check: none (judge only)
Report a problem with this finding
Typo Appendix bullet: "The number 28×28 inception modules is increased from 2 to 3" is missing 'of'.
- p. 10 Appendix, second bullet
The number 28×28 inception modules is increased
Judge's note: The page 10 image confirms the wording.
Second check: none (judge only)