Efficient Estimation of Word Representations in Vector Space
· paper · AI and machine learning
Proposes two model architectures for learning continuous vector representations of words from very large datasets. They report large accuracy gains at lower computational cost, learning high-quality vectors from 1.6 billion words in under a day.
Checked: PDF from arxiv.org, fetched 4 Oct 2026. Page numbers are those of that PDF.
1 finding
Substance Table 4: the Total for both Mikolov RNNLM rows ("80 320M 4.9 18.4 12.7" and "640 320M 8.6 36.5 24.6") cannot be reached as the question-weighted mean of the Semantic and Syntactic columns. The test set has "8869 semantic and 10675 syntactic questions", and every other row in Tables 4 and 5 is consistent with that weighting.
- p. 8 Table 4, Mikolov RNNLM rows
Mikolov RNNLM 640 320M 8.6 36.5 24.6
- p. 6 Section 4.1
Overall, there are 8869 semantic and 10675 syntactic questions.
- p. 2 Section 1.1, first lines
with a modest dimensionality of the word vectors between 50 - 100
- p. 8 Table 4, Mikolov RNNLM rows
Mikolov RNNLM 80 320M 4.9 18.4 12.7
Only the authors can say what is right here.
Judge's note: I checked Table 4 on the page-8 image and the crop, and the question counts in Section 4.1 on the page-6 crop. Using weights 8869/19544 and 10675/19544, the RNNLM rows give 12.27 (printed 12.7) and 23.84 (printed 24.6). Even if each printed component is pushed to its upper rounding limit (4.95/18.45 and 8.65/36.55), the maxima are about 12.32 and 23.89, so rounding of the components cannot explain the gap. All other Table 4 rows match the weighted mean to within rounding: Collobert 10.94 vs 11.0, Turian 2.06/1.84, Mnih 5.79/8.71, Huang 12.37, the NNLM rows 20.27/43.14/50.75, CBOW 36.04, Skip-gram 53.22. I searched the text for a different question subset or coverage rule for the RNNLM vectors and found none. The Table 4 caption says "Full vocabularies are used", and Section 4.3 describes the 82K vocabulary only for the Table 3 setup. Only the authors can say which of the three numbers in each row is off. A secondary point in this cluster, the Section 1.1 phrase "modest dimensionality of the word vectors between 50 - 100" set against the 200- and 640-dimension prior vectors in Table 4, is at most loose wording. The sentence is about prior architectures trained on more than a few hundred million words, so it is not counted as a defect here.
Second check: another model agreed
Report a problem with this finding
5 minor (typos and the like; most checked by the judge only)
Reference or label Reference [3] gives the venue as "...Empirical Methods in Natural Language Processing and Computational Language Learning, 2007.", which omits "Natural" from the CoNLL name.
- p. 11 Reference [3]
Processing and Computational Language Learning, 2007.
- p. 11 Reference [3]
Processing and Computational Language Learning, 2007
Only the authors can say what is right here.
Judge's note: Rests on outside knowledge: that the joint conference is EMNLP-CoNLL, i.e. Computational Natural Language Learning. The page prints the quoted text, and the reference remains identifiable.
Second check: another model agreed
Report a problem with this finding
Wording Section 1.1 says "none of the previously proposed architectures has been successfully trained on more than a few hundred of millions of words", but Table 4 lists earlier vectors trained on 660M (Collobert-Weston) and 990M (Huang) words; the phrase "hundred of millions" is also ungrammatical.
- p. 2 Section 1.1, first line
than a few hundred of millions of words, with a modest dimensionality of the word vectors between
- p. 8 Table 4, Huang row
Huang NNLM 50 990M 13.3 11.6 12.3
- p. 1 Section 1.1 Goals of the Paper
none of the previously proposed architectures has been successfully trained on more
- p. 2 Section 1.1, continuation
than a few hundred of millions of words
Only the authors can say what is right here.
Judge's note: Checked pages 1-2 and the Table 4 rows on page 8. The 990M Huang entry is close to one billion, which strains "a few hundred millions". The sentence is hedged with "as far as we know", so this is a wording overstatement rather than a wrong number. The authors should say whether they meant "about a billion" or "a few hundred million".
Second check: another model agreed
Report a problem with this finding
Typo Page 9 reads "The result are reported in Table 6."; it should read "The results are reported".
- p. 9 Section 4.4, end
The result are reported in Table 6.
Judge's note: The agreement error is visible in the page image.
Second check: none (judge only)
Report a problem with this finding
Typo Reference [14] prints "language models for higly inflective languages"; "higly" should be "highly".
- p. 11 Reference [14]
guage models for higly inflective languages
- p. 11 Reference [14]
for higly inflective languages
Judge's note: The misspelling is visible on page 11.
Second check: none (judge only)
Report a problem with this finding
Typo Reference [26] prints "Nature, 323:533.536, 1986.", with a period in place of the page-range dash (533-536).
- p. 12 Reference [26]
Nature, 323:533.536, 1986.
- p. 12 Reference [26]
Nature, 323:533.536, 1986
Judge's note: The page image shows a period, so this is not an extraction artefact. It is a punctuation slip in a page range.
Second check: none (judge only)