Long Short-Term Memory
1997 · paper · AI and machine learning
Introduces long short-term memory (LSTM), a gradient-based recurrent network method using gated units that keep error flow constant. On artificial tasks it bridges time lags beyond 1000 steps, succeeding more often and learning faster than earlier recurrent methods.
Checked: PDF from bioinf.jku.at, fetched 4 Oct 2026. Page numbers are those of that PDF.
11 findings
Substance In Eq. (2) the factor is printed as "f′_{l_m}(net_{l_m}(t − m)) w_{l_m l_{m−1}}"; for q = 1 this gives f′_v w_{vu}, which contradicts Eq. (1)'s "f′_v(net_v(t − 1)) w_{uv}". The subscripts appear transposed; they should read w_{l_{m−1} l_m}.
- p. 4 Equation (2)
f′_{l_m}(net_{l_m}(t − m))w_{l_m l_{m−1}}
- p. 3 Equation (1)
f′_v(net_v(t − 1))w_uv
Only the authors can say what is right here.
Judge's note: Reworked from the definitions on p. 3: w_ij is the weight from j to i, and ϑ_j(t) = f′_j(net_j(t)) Σ_i w_ij ϑ_i(t+1). So ∂ϑ_v(t−1)/∂ϑ_u(t) = f′_v(net_v(t−1)) w_uv, which matches Eq. (1), and the q>1 recursion has w_lv. Unrolling the recursion with l_0 = u and l_q = v gives the product of f′_{l_m}(net_{l_m}(t−m)) w_{l_{m−1} l_m}. Eq. (2) on the p. 4 image prints w_{l_m l_{m−1}}, which is the reverse. The matrix rewrite also uses W_{u^T} with entries w_ui, consistent with Eq. (1). The vanishing-error conclusion is unaffected, but the formula is wrong as printed.
Second check: another model agreed
Report a problem with this finding
Substance Task 2c: the displayed equation "4 + Σ_{k=0}^∞ (1/10)(9/10)^k (q + k) = q + 14" evaluates to q + 13; the derived "(q+10)/p" carries the same offset (expected distractor count is q + 9).
- p. 14 Task 2c, displayed equation for expected length
(q + k) = q + 14
- p. 14 Task 2c, expected occurrences of a_i
in a sequence is (q+10)/p ≈ q/p
- p. 14 Task 2c expected length equation
= q + 14
- p. 14 Task 2c, expected length formula
= q + 14.
Judge's note: Page 14 image. The sum runs from k=0 with weights (1/10)(9/10)^k. These weights sum to 1, and the mean of k is 0.9/0.1 = 9. So the left side is 4 + q + 9 = q + 13, not q + 14. This matches the generation description: a prefix of q+2, then each further element with probability 9/10 or e with probability 1/10, and minimal length q+4. The expected number of distractors is then q + 9, not q + 10. The ≈ q/p conclusion is unaffected.
Second check: another model agreed
Report a problem with this finding
Substance Task 2c training-set definition gives "1 ≤ i1, i2, . . . , iq+k ≤ q", but the distractor symbols are a_1,…,a_p, so the index bound should be p.
- p. 14 Task 2c, definition of the training-sequence sets
1 ≤ i1, i2, . . . , iq+k ≤ q
Judge's note: Page 14: the distractor symbols are a_1..a_p (p+4 input symbols), yet the set definitions bound the indices by q. Table 3 (page 15) uses q=1000 with p=50, 100, 200 and 500, so indices up to q would point to symbols that do not exist. Task 2b's analogous definition (page 13) correctly uses ≤ p−1.
Second check: another model agreed
Report a problem with this finding
Substance Sec. 5.5 says "the average test set error was always below 0.026 (0.013)" for n_seq = 140 (13), but Table 8 reports MSE "0.0139" for n_seq = 13 as a mean of 10 trials.
- p. 19 Section 5.5 Results
the average test set error was always below 0.026 (0.013)
- p. 19 Table 8, n_seq = 13 row, MSE column
0.0139
Only the authors can say what is right here.
Judge's note: Page 19. If every trial's test error were below 0.013, the mean over 10 trials could not be 0.0139. The n_seq=140 pair (0.026 bound, 0.0223 mean) and the wrong-prediction counts (≤170/139, ≤15/14) are consistent, which suggests the same measure is meant. The paper defines no other error measure, so either the bound or the table value needs checking by the authors.
Second check: another model agreed
Report a problem with this finding
Reference or label "Lin et al. (1995) propose variants of time-delay networks called NARX networks" (p. 2) and "Lin et al. 1995" (p. 3) are cited, but the reference list has no Lin et al. entry.
- p. 2 Time-delays paragraph
Lin et al. (1995) propose variants of time-delay networks called NARX networks.
- p. 3 Simple weight guessing paragraph
Lin et al. 1995
- p. 3 Section 2, Simple weight guessing
Miller and Giles 1993, Lin et al. 1995
- p. 31 References (alphabetical, between Lang and Miller)
Lang, K., Waibel, A., and Hinton, G. E. (1990).
Judge's note: Page 2 and page 3 images show both citations. On page 31 the reference list goes directly from Lang, Waibel and Hinton (1990) to Miller and Giles (1993). I also checked pp. 30–31, and a text search finds no Lin entry anywhere in the references.
Second check: another model agreed
Report a problem with this finding
Reference or label The Figure 2 caption reads "by rotating Figure 1 by 90 degrees anti-clockwise, it will match with the corresponding parts of Figure 1"; the second "Figure 1" should evidently be "Figure 2".
- p. 9 Figure 2 caption
by rotating Figure 1 by 90 degrees anti-clockwise, it will match with the corresponding parts of Figure 1
Judge's note: The p. 9 image confirms the wording. The caption is relating cell1/block1 in Figure 2 to Figure 1, so a rotated Figure 1 can only match parts of Figure 2.
Second check: another model agreed
Report a problem with this finding
Reference or label Under output gate bias, the text says "Memory cells with more negative bias automatically get "allocated" later (see Experiments 1, 3, 4, 5, 6 in Section 5)", but Table 10 gives a random output gate bias (ogb = "r") for Experiments 4-1/4-2/4-3, 5, 6a and 6b.
- p. 9 Abuse problem, solution (2) Output gate bias
Memory cells with more negative bias automatically get "allocated" later (see Experiments 1, 3, 4, 5, 6 in Section 5).
- p. 21 Table 10, ogb column for rows 4-1..6b
r
Only the authors can say what is right here.
Judge's note: On the Table 10 image (p. 21), ogb is -1,-2,-3[,-4] for Experiment 1 and -2,-4,-6 for Experiment 3, but "r" for every row of Experiments 4, 5 and 6. Experiments 4 and 6 stagger the input gate biases instead, and Experiment 5 has random values for both gate biases. The authors may have meant input gate biases for Experiments 4 and 6; only they can say.
Second check: another model agreed
Report a problem with this finding
Reference or label Table 10 caption: "“h1” is logistic sigmoid in [−2, 2]" and "“g2” in [−1, 1]" swap the ranges stated in the text and in Eqs. (4)-(5) (h in [−1,1], g in [−2,2]).
- p. 21 Table 10 caption, 13th and 14th columns
"h1" is logistic sigmoid in [−2, 2]. 14th column: the logistic function g, where "g1" is sigmoid in [0, 1], "g2" in [−1, 1]
- p. 24 Appendix A.1, eqs (4)-(5)
The function h (with range [−1, 1]) used in the experiments is
- p. 21 Table 10 caption, 13th and 14th column
“h1” is logistic sigmoid in [−2, 2]
- p. 21 Table 10 caption, 14th column
“g2” in [−1, 1]
Judge's note: Page 21 caption checked against page 24, Eqs. (4)-(5): h(x)=2/(1+exp(−x))−1 with range [−1,1], and g(x)=4/(1+exp(−x))−2 with range [−2,2]. It was also checked against the experiment texts on page 12 (Exp. 1), page 14 (Task 2c) and page 20 (Exp. 6), which say h in [−1,1] and g in [−2,2]. Table rows for these experiments use h1/g2, so the caption's ranges are interchanged.
Second check: another model agreed
Report a problem with this finding
Reference or label Pineda (1987) is cited as 'Physical Review Letters, 19(59):2229–2232', reversing the volume(issue) order used throughout the list.
- p. 31 References
Review Letters, 19(59):2229–2232.
- p. 31 References, Pineda (1987)
Physical Review Letters, 19(59):2229–2232
Only the authors can say what is right here.
Judge's note: Rests on outside knowledge: that Pineda's 1987 PRL paper is in volume 59, issue 19. Within the file, the p31 image shows '19(59)', while other entries use volume(issue), e.g. 5(2):157–166 and 6(5):1212–1228, so the printed order is at least inconsistent.
Second check: another model agreed
Report a problem with this finding
Reference or label NIPS 4 is cited with two editor lists: 'Lippman, D. S., Moody, J. E., and Touretzky, D. S.' (Mozer 1992) vs 'Moody, J. E., Hanson, S. J., and Lippman, R. P.' (Schmidhuber 1992c).
- p. 31 References, Mozer (1992)
In Lippman, D. S., Moody, J. E., and Touretzky, D. S., editors, Advances in Neural Information Processing Systems 4
- p. 32 References, Schmidhuber (1992c)
In Moody, J. E., Hanson, S. J., and Lippman, R. P., editors, Advances in Neural Information Processing Systems 4
Only the authors can say what is right here.
Judge's note: Pages 31-32 images confirm both entries. Internally, the same volume has inconsistent editors and Lippman has inconsistent initials (D. S. vs R. P.). The Mozer list matches the NIPS 3 editors used elsewhere (Lippmann, Moody, Touretzky). Which list is correct rests on outside knowledge (that the NIPS 4 editors were Moody, Hanson and Lippmann).
Second check: another model agreed
Report a problem with this finding
Reference or label Williams and Peng (1990) cited as 'Neural Computation, 4:491–501', although the same list dates Neural Computation volume 4 to 1992.
- p. 32 References
Neural Computation, 4:491–501.
Only the authors can say what is right here.
Judge's note: Rests on outside knowledge: that Neural Computation publishes one volume per year (so a 1990 paper would be in volume 2). Internally on pp30-32: vol 1 = 1989 (Pearlmutter; Cleeremans et al.), vol 3 = 1991 (Baldi & Pineda), vol 4 = 1992 (Schmidhuber 1992a/b; Watrous & Kuhn). The 1990/vol 4 pairing is inconsistent with these; only the authors can say whether the volume or the year is wrong.
Second check: another model agreed
Report a problem with this finding
15 minor (typos and the like; most checked by the judge only)
Reference or label "(Bengio 1994, Bengio and Frasconi 1994, ...)" cites a single-author Bengio 1994, but the only Bengio entries are Bengio and Frasconi (1994) and Bengio, Simard and Frasconi (1994).
- p. 3 Simple weight guessing paragraph
in (Bengio 1994, Bengio and Frasconi 1994
- p. 30 References
Bengio, Y., Simard, P., and Frasconi, P. (1994).
- p. 3 Simple weight guessing
(Bengio 1994, Bengio and Frasconi 1994
Judge's note: The citation is on the p. 3 image, and the p. 30 references list only the two Bengio entries named above. Elsewhere (e.g. p. 2) the paper cites the Bengio, Simard and Frasconi paper as "Bengio et al. (1994)", so this is almost certainly a dropped "et al.".
Second check: another model agreed
Report a problem with this finding
Reference or label "as confirmed by Experiments 4 and 5 in Section 5.4": Experiment 5 is in Section 5.5, and in Experiment 5 all weights, including the input gate biases, are random (Table 10 igb = "r").
- p. 9 Internal state drift paragraph
as confirmed by Experiments 4 and 5 in Section 5.4.
- p. 21 Table 10, row 5, igb column
r
- p. 9 Internal state drift paragraph
as con rmed by Experiments 4 and 5 in Section 5.4
Only the authors can say what is right here.
Judge's note: The p. 9 image shows the sentence. Section headings: 5.4 is Experiment 4 (Adding Problem) and 5.5 is Experiment 5 (Multiplication Problem). On the Table 10 image (p. 21), row 5 has igb = r. The text of Section 5.5 says all weights, including the bias weights, are randomly initialised in [-0.1, 0.1]. The section pointer is wrong for Experiment 5. Whether Experiment 5 supports the claim about the initial input gate bias is for the authors to say.
Second check: another model agreed
Report a problem with this finding
Wording Discussion says '...not by the truncated LSTM algorithm with small weight initializations, such as the 500-step parity problem (see introduction to Section 5)', but the Section 5 introduction gives no sequence length and reports no LSTM result on parity.
- p. 22 Section 6, Limitations, third bullet
such as the 500-step parity problem (see introduction to Section 5)
- p. 10 Section 5 introduction
guessing solved a variant of Bengio and Frasconi's "parity problem" (1994) problem much faster
Judge's note: Page 10 (Section 5 intro, 'Guessing can outperform...') says only that guessing solved a variant of Bengio and Frasconi's 'parity problem' faster than seven methods. It gives no 500-step length and no LSTM run. So the pointer supports the guessing part of the sentence, but not the length or the claim that LSTM fails. Page 22 checked. This is a minor wording/pointer imprecision.
Second check: another model agreed
Report a problem with this finding
Wording The outline says of Experiments 4 and 5 "Again minimal time lags involve hundreds of steps", but Experiment 5 (and Experiment 4 with T = 100) has a minimal lag of 50.
- p. 11 Section 5, Outline of experiments, Experiments 4 and 5 bullet
Again minimal time lags involve hundreds of steps.
- p. 21 Table 10, rows 4-1 and 5, column lag
50
Judge's note: Table 10 (p. 21) gives lag 50 for rows 4-1 and 5, and 250 and 500 for 4-2 and 4-3. Tables 7 and 8 agree. The wording overstates the lags for some settings, including the only Experiment 5 setting.
Second check: another model agreed
Report a problem with this finding
Wording The results text says "Unlike the other methods, LSTM always learns to solve the task", while Table 1 shows 97% success in two LSTM settings and its caption says "Only LSTM almost always learns to solve the task (only two failures out of 150 trials)".
- p. 12 Section 5.1 Results
Unlike the other methods, LSTM always learns to solve the task.
- p. 13 Table 1 and caption
Only LSTM almost always learns to solve the task (only two failures out of 150 trials).
Judge's note: The Table 1 crop (p. 13) shows 97% for 3 blocks/lr 0.2 and 4 blocks/lr 0.5. Each row is 30 trials (p. 12: mean of 30 trials), so 97% means 1 failure per row, 2 of 150 in total, as the caption says. The text's "always" overstates this. One reviewer's figure of "40 trials" is a misreading; the caption says 150.
Second check: another model agreed
Report a problem with this finding
Wording Sec. 5.1 says LSTM uses "3 (4) memory cell blocks" with output gate biases "initialized to -1, -2, and -3, respectively", but Table 10 (rows 1-1, 1-4) gives "-1,-2,-3,-4" for the 4-block nets.
- p. 12 Section 5.1 Architectures
which are initialized to -1, -2, and -3, respectively
- p. 20 Table 10, row 1-1
-1,-2,-3,-4
Judge's note: Page 12: the architecture paragraph covers both the 3-block and the 4-block nets but lists only three bias values. Page 20/21, Table 10: rows 1-1 and 1-4 (b=4) show -1,-2,-3,-4, and rows 1-2, 1-3 and 1-5 (b=3) show -1,-2,-3. The text leaves out the fourth gate's bias. Table 10 resolves it, so this is a minor incompleteness.
Second check: another model agreed
Report a problem with this finding
Wording Appendix A.1 says '(22) needs K(H + C) steps for output units', but output units receive weights from H hidden units and all CS memory cells (W contains KH + KCS), so applying Eq. (22) to them takes K(H + CS) steps; the total '... + KC + ...' inherits this.
- p. 28 Appendix A.1, computational complexity
(22) needs K(H + C) steps for output units
- p. 27 Appendix A.1, weight count
W = KH + KCS + CSI + 2CI + HI
Only the authors can say what is right here.
Judge's note: Pages 27-28. Eq. (22), Δw_lm = α e_l y^m(t-1), is one operation per weight. Per W = KH + KCS + CSI + 2CI + HI, the output-unit weights number KH + KCS, so the count should be K(H+CS), not K(H+C). Summing the listed counts (K, KH, KSC, K(H+C)+HI+CI, KCS, 4×CSI) does reproduce the printed total K+2KH+KC+2KSC+HI+CI+4CSI, so the inconsistency sits in the (22) term. The O(KH+KSC+HI+CSI) = O(W) conclusion still holds, because a KSC term is already present.
Second check: another model agreed
Report a problem with this finding
Typo The author is spelled two ways: "deVries and Principe's above-mentioned approach (1991)" vs "de Vries and Principe 1991" and the reference entry "de Vries, B."
- p. 2 Section 2, Time constants
deVries and Principe's above-mentioned approach (1991)
- p. 2 Section 2, Time-delays
see also de Vries and Principe 1991
Judge's note: On the page 2 image, the Time-constants paragraph reads "deVries" and the Time-delays paragraph reads "de Vries". The reference list on p. 30 has "de Vries, B. and Principe, J. C. (1991)". This is cosmetic only.
Second check: none (judge only)
Report a problem with this finding
Typo Duplicated word: "guessing solved a variant of Bengio and Frasconi's "parity problem" (1994) problem much faster".
- p. 10 Guessing paragraph, Section 5 introduction
"parity problem" (1994) problem much faster
Judge's note: The p. 10 image, in the 'Guessing can outperform...' paragraph, shows "problem" both before and after "(1994)".
Second check: none (judge only)
Report a problem with this finding
Typo Doubled article: "This is a another remarkable property of LSTM".
- p. 15 Scaling paragraph, Task 2c
This is a another remarkable property of LSTM
Judge's note: Seen on the page 15 image, Scaling paragraph.
Second check: none (judge only)
Report a problem with this finding
Typo Eq. (37) denominator printed as '∂y^{in_j}(t))', with an extra closing parenthesis.
- p. 29 Equation (37)
∂s_cj(t) / ∂y^inj(t))
Judge's note: Page 29 image: in Eq. (37) the second fraction's denominator reads ∂y^{in_j}(t)), unlike the analogous Eq. (36), whose ∂g(net_{c_j}(t)) is balanced.
Second check: none (judge only)
Report a problem with this finding
Typo Last line of Eq. (41) prints 'g′(net_{c_j}(t − q)y^{in_j}(t − q)' and 'g(net_{c_j}(t − q)f′_{in_j}(...)', each missing a closing parenthesis.
- p. 29 Equation (41), final case expression
g′(net_cj(t − q)y^inj(t − q)
Judge's note: Page 29 image: in both cases of the final expression of Eq. (41), g′(net_{c_j}(t−q) and g(net_{c_j}(t−q) open two parentheses and close only one.
Second check: none (judge only)
Report a problem with this finding
Typo Reference reads 'Hochreiter, J. (1991). Untersuchungen zu dynamischen neuronalen Netzen.', where the initial should be S. as in the other Hochreiter entries.
- p. 30 References
Hochreiter, J. (1991). Untersuchungen zu dynamischen neuronalen Netzen.
Judge's note: Page 30 image shows 'Hochreiter, J. (1991)'. Every other Hochreiter entry (p31) and the URL (~hochreit) point to Hochreiter, S. The identity is clear, so this is cosmetic.
Second check: none (judge only)
Report a problem with this finding
Typo Duplicated phrase in two references: 'Technical Report CRL Technical Report 8801' (Elman 1988) and 'Technical Report Technical Report NU-CCS-89-27' (Williams 1989).
- p. 30 References, Elman 1988
Technical Report CRL Technical Report 8801
- p. 32 References, Williams 1989
Technical Report Technical Report NU-CCS-89-27
Judge's note: Both are visible in the page images on p30 (Elman) and p32 (Williams 1989).
Second check: none (judge only)
Report a problem with this finding
Typo NIPS 5 editors printed as 'S. J. Hanson, J. D. C. and Giles, C. L.' in Plate (1993), Ring (1993) and Sun et al. (1993); the second editor's name is garbled.
- p. 31 References, Plate (1993)
In S. J. Hanson, J. D. C. and Giles, C. L., editors
- p. 32 References, Sun et al. (1993)
In S. J. Hanson, J. D. C. and Giles, C. L., editors
Judge's note: Page 31 (Plate, Ring) and p32 (Sun) images show 'J. D. C.' with no surname, and the name order is mixed. The entry presumably means Cowan, J. D., who is listed as 'Cowan, J. D.' in the Bengio & Frasconi entry (p30). This is a bibliography formatting glitch.
Second check: none (judge only)