A Neural Probabilistic Language Model
2003 · paper · AI and machine learning
Proposes a neural network language model that learns distributed word representations together with the probability of word sequences, to counter the curse of dimensionality. On two text corpora it significantly improves on state-of-the-art n-gram models and can use longer contexts.
Checked: PDF from jmlr.org, fetched 4 Oct 2026. Page numbers are those of that PDF.
6 findings
Reference or label Both "computing the exponential (step 1(c)ii above)" and "(instead of step 1(c)ii)" point to step ii. In the algorithm the exponential p_j ← e^{y_j} is step 1(c)iii; step ii is the direct-connection update.
- p. 10 Numerical stability paragraph
computing the exponential (step 1(c)ii above)
- p. 10 Numerical stability paragraph
(instead of step 1(c)ii)
- p. 9 Algorithm, forward phase 1(c)
iii. pj ←eyj
- p. 10 numerical problem paragraph
too large for computing the exponential (step 1(c)ii above)
Judge's note: Checked the algorithm on page 9: i. y_j ← b_j + a.U_j; ii. if direct connections, y_j ← y_j + x.W_j; iii. p_j ← e^{y_j}; iv. s_i ← s_i + p_j. Both references on page 10 cite ii.
Second check: another model agreed
Report a problem with this finding
Reference or label The text ("When m is specified for a back-off model in the table, a class-based n-gram is used (m is the number of word classes)") and the caption ("m : ... number of classes for class-based n-grams") say class counts appear under m. Table 1 instead lists them in column c (which the caption also defines as the number of word classes), and m is blank for every back-off row.
- p. 13 Table 1 caption
m : number of word features for MLPs, number of classes for class-based n-grams
- p. 12 Section 4.2
When m is specified for a back-off model in the table, a class-based n-gram is used (m is the number of word classes)
- p. 13 Table 1 caption
c : number of word classes in class-based n-grams
Judge's note: Checked the Table 1 image on page 13. The class counts 150/200/500/1000/2000/500/500 are in column c, and the m column is empty for all back-off rows. The m label in the text and caption therefore does not match the table.
Second check: another model agreed
Report a problem with this finding
Reference or label Two different reports carry the same number: Brown and Hinton 'Products of hidden markov models. Technical Report GCNU TR 2000-004' and Hinton 'Training products of experts by minimizing contrastive divergence. Technical Report GCNU TR 2000-004'.
- p. 17 References, Brown and Hinton
Technical Report GCNU TR 2000-004
- p. 18 References, Hinton 2000
GCNU TR 2000-004, Gatsby Unit
- p. 17 References, Brown and Hinton
Products of hidden markov models. Technical Report GCNU TR 2000-004
- p. 18 References, Hinton 2000
Training products of experts by minimizing contrastive divergence. Technical Report GCNU TR 2000-004
Only the authors can say what is right here.
Judge's note: Rests on outside knowledge: which of the two reports actually has number GCNU TR 2000-004 (reviewers say it is the contrastive-divergence report). The page images (p17 and p18) show both entries with the identical number, even though their titles and authors differ, so at least one number is wrong.
Second check: another model agreed
Report a problem with this finding
Reference or label The Stolcke (2002, Denver) and Xu and Rudnicky (2000, Beijing) entries give the venue as 'International Conference on Statistical Language Processing'. ICSLP, which the Jensen and Riis entry cites under that abbreviation, is the International Conference on Spoken Language Processing.
- p. 19 References, Stolcke 2002
Conference on Statistical Language Processing, Denver, Colorado, 2002
- p. 19 References, Xu and Rudnicky 2000
Conference on Statistical Language Processing, pages M1–13, Beijing, China
- p. 19 References, Xu and Rudnicky 2000
Conference on Statistical Language Processing, pages M1–13, Beijing, China, 2000
Only the authors can say what is right here.
Judge's note: Rests on outside knowledge: that the Denver 2002 and Beijing 2000 conferences were ICSLP, the International Conference on Spoken Language Processing. The page 19 image shows 'Statistical Language Processing' in both entries.
Second check: another model agreed
Report a problem with this finding
Wording The text says "the maximum of the exponential’s argument is 1". With p_j ← e^{y_j−Q} and Q = max y_j, the maximum argument is 0; it is the maximum of the exponential itself that is 1.
- p. 10 Paragraph after the parallel algorithm (numerical stability)
the maximum of the exponential’s argument is 1
Judge's note: Checked page 10. Q is the overall maximum of the y_j, so max(y_j − Q) = 0 and e^0 = 1. The sentence as printed is wrong, but the intended meaning is clear.
Second check: another model agreed
Report a problem with this finding
Wording In the out-of-vocabulary paragraph the new word is j ("a new word j ∉ V", "C(j) ← Σ_{i∈V} C(i)P̂(i|w)"), but the text goes on to say "except for word i, which requires a pass through the neural network). This feature vector C(i) can then be used ... words that follow word i"; j is meant.
- p. 15 Section 5.1, Out-of-vocabulary words
except for word i, which requires a pass through the neural network). This feature vector C(i) can then be used
- p. 15 Section 5.1, Out-of-vocabulary words
except for word i, which requires a pass through the neural network
- p. 15 Out-of-vocabulary words
This feature vector C(i) can then be used
- p. 15
for all the words, except for word i, which requires a pass through the neural network
Judge's note: Checked the crop and page 15. In the same paragraph i is the summation index over V and j is the new word whose vector is initialised. 'This feature vector' can only refer to the newly initialised C(j), and the word that needs a network pass after it is added to V is also j. So the three later uses of i should read j.
Second check: another model agreed
Report a problem with this finding
9 minor (typos and the like; most checked by the judge only)
Wording The operation count on p. 1144 is printed as "|V|(1+nm+h)+h(1+nm)+nm", but the fraction derived from it on p. 1145 uses "|V|(1+(n−1)m+h)+h(1+(n−1)m)+(n−1)m". The input layer has (n−1)m units.
- p. 8 Section 3.2, last paragraph
|V|(1+nm+h)+h(1+nm)+nm
- p. 9 Section 3.2, fraction
|V|(1+(n−1)m+h)
- p. 9 fraction formula
|V|(1+(n−1)m+h)+h(1+(n−1)m)+(n−1)m
Judge's note: Checked the page 8 and page 9 images. The model defines x as the concatenation of n−1 feature vectors (step 1(a) on p. 1145), so (n−1)m is the consistent term. Both forms give 99.7%, so only the notation differs.
Second check: another model agreed
Report a problem with this finding
Wording "The energy function E(w_{t−n+1},…,w_t) can be interpreted as an unnormalized log-probability for the joint occurrence": the following formula gives P̂ ∝ e^{−E}, so it is −E, not E, that is the unnormalized log-probability.
- p. 14 Section 5.1
can be interpreted as an unnormalized log-probability for the joint occurrence
Judge's note: Checked page 14. E is described as low for likely sub-sequences, and the softmax uses e^{−E}. The sign in the sentence is reversed relative to both. This is a wording slip; the formula itself is consistent.
Second check: another model agreed
Report a problem with this finding
Typo Garbled phrase on p. 1140 (PDF p. 4): "the same g_i is used across time – that is, and across input words at different positions"; 'that is, and' looks like an editing leftover.
- p. 4 Section 1.2
the same gi is used across time – that is, and across input words at different positions
Judge's note: Checked the crop. The phrase reads exactly as quoted and is not grammatical.
Second check: none (judge only)
Report a problem with this finding
Typo Stray period in the in-text citation "Chen and Goodman., 1999".
- p. 12 Section 4.1
Chen and Goodman., 1999
Judge's note: The stray period is visible in the crop of page 12.
Second check: none (judge only)
Report a problem with this finding
Typo Unbalanced parenthesis in "x = (C(w_t),C(w_{t−1}),C(w_{t−2}),··· ,C(w_{t−n+1}).": the outer opening parenthesis is never closed.
- p. 14 Section 5.1, definition of x
x = (C(wt),C(wt−1),C(wt−2),··· ,C(wt−n+1).
Judge's note: In the crop image, the last term C(w_{t−n+1}) is followed by a single ')' and then a period. That ')' closes C(, so the tuple's opening parenthesis is left unclosed.
Second check: none (judge only)
Report a problem with this finding
Typo In the Dongarra et al. reference, 'University of Tenessee' should be 'University of Tennessee'.
- p. 17 References, Dongarra et al.
University of Tenessee
Judge's note: The misspelling is visible in the page 17 image.
Second check: none (judge only)
Report a problem with this finding
Typo The Goodman (2001) reference prints the author as 'J .Goodman' instead of 'J. Goodman'.
- p. 18 References, Goodman 2001
J .Goodman
Judge's note: The page 18 image shows the space before the period and no space after it.
Second check: none (judge only)
Report a problem with this finding
Typo The Hinton (1986) reference prints the year twice: 'pages 1–12, Amherst 1986, 1986.'
- p. 18 References, Hinton 1986
Amherst 1986, 1986
- p. 18
Society, pages 1–12, Amherst 1986, 1986. Lawrence Erlbaum,
Judge's note: Visible in the page 18 image; the repeated year is redundant.
Second check: code check
Report a problem with this finding
Typo The Schutze (1993) reference has a doubled page marker: 'pages pp. 895–902'.
- p. 19 References, Schutze 1993
pages pp. 895–902
- p. 18 References, Hinton 1986
Amherst 1986, 1986.
- p. 18 References, Goodman
J .Goodman.
Judge's note: Visible in the page 19 image. The other items in this cluster ('Amherst 1986, 1986' and 'J .Goodman') are already covered by C022 and C021, so this verdict covers only the 'pages pp.' defect.
Second check: none (judge only)