Adam: A Method for Stochastic Optimization

• p. 2 Section 2.1

  The effective stepsize has two upper bounds

• p. 3 Section 2.1 continuation

  otherwise

Only the authors can say what is right here.

Judge's note: Reworked by hand. Take β1=0.9 and β2=0.5, which satisfy 1−β1 = 0.1 ≤ √0.5 ≈ 0.707. Let g1=1 and g2..g10=0. Then m10 = 0.1·0.9^9 ≈ 0.03874 and m̂10 = 0.03874/(1−0.9^10) ≈ 0.05948. Also v10 = 0.5^10 ≈ 0.000977 and v̂10 ≈ 0.000978. So (m̂/√v̂)² ≈ 3.62, giving |Δ10| ≈ 1.90α > α. The ratio keeps growing for larger t. The paper states the bound unconditionally for β ∈ [0,1) and gives no proof or extra condition in Section 2.1. The bound is false as printed, though the paper's later theorem results are separate from it.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 3 Section 3, last paragraph

  many gradients by chosing a small value of β2; however it is exactly this case of small β2

• p. 5 Section 5, RMSProp

  a value of β2 close to 1 (required in case of sparse gradients)

• p. 8 Section 6.4

  Values of β2 close to 1, required for robust-ness to sparse gradients

• p. 3 Sec 3

  by chosing a small value of β2

Judge's note: Checked on the page 3, 5 and 8 images. Averaging over many gradients, and the large initialisation bias (1−β2^t small), both happen when β2 is close to 1, i.e. small 1−β2. Sections 5 and 6.4 say exactly this. Section 3 as printed says the opposite. The misspellings bundled here duplicate other clusters: "chosing"/"Initalization" (C011), "theoratical" (C014) and "low" for "slow" (C015). One additional cosmetic typo is confirmed on the page 7 image: the Figure 3 panel title reads "First 3 Epoches".

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 15 End of proof of Theorem 10.5, final bound

  2αβ1(1 −λ)2

• p. 15 Preceding step of proof

  (1 −β1)(1 −λ)2

• p. 4 Theorem 4.1

  2α(1 −β1)(1 −λ)2

• p. 15 First display, step from D∞²/(2α)ΣΣ β1,t/(1−β1,t)√(t v̂_{t,i}) to D∞²G∞√(1−β2)/(2α)ΣΣ β1,t/(1−β1,t)√t

  D2∞G∞

Only the authors can say what is right here.

Judge's note: I checked page 15: the last display has β1 where the line before, and Theorems 4.1 (p.4) and 10.5 (p.14), have (1−β1). Substituting the series bound gives 2α(1−β1)(1−λ)². I also checked the reviewer's second point. v̂_t = Σ(1−β2)β2^{t−j}g_j²/(1−β2^t) is a weighted average whose weights sum to 1. For constant gradients G∞, v̂_t = G∞², so √v̂_t = G∞ > G∞√(1−β2). The replacement in the first display on p.15 therefore points the wrong way, and the third term of the theorem (with √(1−β2)) is not established by this step.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 5 Section 5, AdaGrad paragraph

  limβ2→1 bvt = t−1 · Pt i=1 g2 t

• p. 5 Section 5, AdaGrad paragraph

  θt+1 = θt −α · gt/

Judge's note: The page 5 image shows the subscript t, not i, inside each sum. As printed, each sum equals t·g_t², which is not AdaGrad. Eq. (1) correctly uses g_i².

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 9 Section 7.1, after eq. (12)

  AdaMax than Adam, namely: |∆t| ≤α.

Only the authors can say what is right here.

Judge's note: My own working from Algorithm 2 (p9): Δ_t = α/(1−β1^t)·m_t/u_t, with u_t = max(β2 u_{t−1}, |g_t|). The bound |g_i| ≤ u_t/β2^{t−i} gives |m_t| ≤ (1−β1)Σ(β1/β2)^{t−i} u_t, which can exceed (1−β1^t)u_t. Counterexample at the defaults β1=0.9, β2=0.999 with g1=1/0.999, g2=1: u1=1.001001, u2=max(0.999·1.001001, 1)=1, m2=0.9·0.1·1.001001+0.1=0.190090, Δ2/α=0.190090/0.19≈1.00047>1. With β2=0.5, g1=2, g2=1: m2=0.28, u2=1, Δ2/α≈1.47. By contrast, the Adam bounds in Sec 2.1 are hedged ('|∆t| ⪅ α').

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 12 Proof of Lemma 10.3, final inequality

  G∞ q ∥g1:T,i∥2 2 −g2 T + s g2 T,i T ≤2G∞∥g1:T,i∥2

• p. 12 Lemma 10.3 statement and base case

  ≤ 2G∞∥g1:T,i∥2

Only the authors can say what is right here.

Judge's note: Page 12 image. The lemma's only assumptions are ‖g_t‖_2 ≤ G and ‖g_t‖∞ ≤ G∞. Counterexample: T=1, G∞=0.1, g_{1,i}=0.1 gives LHS 0.1 and RHS 2·0.1·0.1=0.02, so the inequality fails. The base case holds only if 2G∞ ≥ 1. The left side is degree 1 in the gradients and the right side is degree 2. Separately, the inductive step yields 2G∞sqrt(‖g‖²−g²_T)+sqrt(g²_{T,i}/T), but the final line substitutes G∞ rather than 2G∞. With 2G∞, using the printed intermediate bound gives 2G∞‖g‖ − g²_{T,i}/√T + |g_{T,i}|/√T, which is ≤ 2G∞‖g‖ only if |g_{T,i}| ≥ 1. Lemma 10.4 and Theorem 4.1 rely on this lemma.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 12 Proof of Lemma 10.3, final display

  Rearrange the inequality and substitute the

Only the authors can say what is right here.

Judge's note: I checked page 12. The line before the final display has coefficient 2G∞, and the final display has G∞. I reworked the step with 2G∞ kept, using √(‖g‖²−g_T²) ≤ ‖g‖ − g_T²/(2√T G∞). This gives 2G∞‖g‖ + (|g_T| − g_T²)/√T, which exceeds 2G∞‖g‖ whenever 0<|g_T|<1. For example g_T=0.5 gives an excess of 0.25/√T. The lemma statement fails as printed when T=1, g=0.1 and G∞=0.1: the left side is 0.1 and the right side is 0.02. Nothing on the page states an assumption that excludes this case (for example G∞ ≥ 1/2).

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 13 Lemma 10.4 statement

  ≤ 2 / 1 −γ 1 / √1 −β2 ∥g1:T,i∥2

• p. 13 Last line of proof of Lemma 10.4

  2G∞ / (1 −γ)2√1 −β2

• p. 13 Lemma 10.4 statement vs last display of proof

  2G∞

• p. 13 Lemma 10.4 statement

  1 −γ

Only the authors can say what is right here.

Judge's note: I checked page 13. The lemma statement has no G∞ and the first power of (1−γ). The last display of the proof has G∞ and (1−γ)². Since 1/(1−γ) < 1/(1−γ)² for γ in (0,1), and G∞ is free, the proved bound does not imply the stated one. As printed, the stated lemma is left without a proof. Theorem 10.5 (p.14–15) uses the (1−γ)² and G∞ form.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 13 Proof of Lemma 10.4

  using the upper bound on the arithmetic-geometric series

• p. 13 Proof of Lemma 10.4, display after 'For γ < 1, using the upper bound on the arithmetic-geometric series'

  For γ < 1, using the upper bound on the arithmetic-geometric series

Only the authors can say what is right here.

Judge's note: I checked page 13. Inside Σ_t ‖g_t,i‖2/√(t(1−β2)) Σ_{j=0}^T tγ^j, the inner sum equals t(1−γ^{T+1})/(1−γ), which grows with t. I checked a counterexample: β1=0.5 and β2=0.25 give γ=0.5 (so γ<1 holds); take T=4, g_4=1 and all other g zero. The printed left side is 4·1.9375/(2√0.75) ≈ 4.47, and the printed right side is (1/(0.25√0.75))·(1/2) ≈ 2.31, so the displayed inequality is false. This leaves Lemma 10.4 without a valid proof at this step.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 14 Proof of Theorem 10.5, expression for g_{t,i}(θ_{t,i}−θ*_{,i})

  (θt,i −θ∗ ,t)2

• p. 14 Proof of Theorem 10.5, passage from the equality for g_{t,i}(θ_{t,i}−θ*_i) to the '≤' line

  2αt(1 −β1)

Only the authors can say what is right here.

Judge's note: I checked page 14 and the crop. The typo θ*_{,t} appears in both lines. I reworked the substantive step: 1/(1−β1) ≥ (1−β1^t)/(1−β_{1,t}), so enlarging the coefficient is valid only if (θ_t−θ*)² − (θ_{t+1}−θ*)² ≥ 0. I recomputed the reviewer's counterexample: d=1, β1=0.9, λ=0.99, β2=0.999, g1=g2=1, t=2, θ*−θ2=0.1α. This gives m̂2=1.0426, v̂1=v̂2=1 and θ3−θ*=−0.8372α. The printed equality sums to −0.0999α ≈ g2(θ2−θ*) = −0.1α. The printed upper bound sums to about −0.957α, which is below the left side, so the per-step inequality is false as printed. The theorem's regret bound is therefore without a valid proof at this step.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 14 Summed regret bound, second term

  (θt,i −θ∗,i)2(

• p. 15 'From the assumption ... we have' first line

  From the assumption, ∥θt −θ∗∥2 ≤D

Only the authors can say what is right here.

Judge's note: I checked pages 14–15. Replacing (θ−θ*)² by D² gives an upper bound only where the coefficient √v̂_t/α_t − √v̂_{t−1}/α_{t−1} is ≥ 0. I checked a counterexample: β2=0.5, g1=1, g2=0 gives v̂1=1 and v̂2=1/3, so √v̂2/α2 − √v̂1/α1 = (0.8165−1)/α < 0. With β1=0.5 the assumption β1²/√β2 ≈ 0.354 < 1 holds. Taking θ2=θ* and |θ1−θ*|=D with T=2, the left side is D²/α and the telescoped bound is 0.8165·D²/α, so the step fails as printed. No assumption on the page excludes this case.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 3 Section 3, text after eq. (4)

  the exponential decay rate β1 can (and should) be chosen

• p. 3 Section 3, after eq. (4)

  the exponential decay rate β1 can (and should) be chosen such that the exponential moving average

• p. 3

  the exponential decay rate β1 can (and should) be chosen such that

Judge's note: Checked on the page 3 image. Eqs. (1)–(4) contain only β2, and ζ depends on β2. The sentence names β1, which is the wrong symbol.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send