Neural Machine Translation by Jointly Learning to Align and Translate
arXiv (Cornell University), 2014 · paper · AI and machine learning
Extends neural machine translation so the model can softly search for the source-sentence parts relevant to each target word, avoiding a fixed-length vector bottleneck. It reaches English-to-French translation performance comparable to a phrase-based system, with sensible learned alignments.
Checked: PDF from arxiv.org, fetched 4 Oct 2026. Page numbers are those of that PDF.
3 findings
Reference or label The reference list has two entries for one paper: "Pascanu, R., Mikolov, T., and Bengio, Y. (2013a). On the difficulty of training recurrent neural networks. In ICML'2013." and "Pascanu, R., Mikolov, T., and Bengio, Y. (2013b). On the difficulty of training recurrent neural networks. In Proceedings of the 30th International Conference on Machine Learning (ICML 2013)."
- p. 11 References
Pascanu, R., Mikolov, T., and Bengio, Y. (2013a). On the difficulty of training recurrent neural networks. In ICML’2013.
- p. 11 References
Pascanu, R., Mikolov, T., and Bengio, Y. (2013b). On the difficulty of training recurrent neural
- p. 11 References
Pascanu, R., Mikolov, T., and Bengio, Y. (2013b).
- p. 11 References
Pascanu, R., Mikolov, T., and Bengio, Y. (2013b). On the difficulty of training recurrent neural networks.
Judge's note: Viewed page 11. Both entries have identical authors, title and venue (ICML 2013), differing only in how the venue is written. The text layer shows they are cited separately as 2013a (Appendix A.1.1) and 2013b (Appendix B.2).
Second check: another model agreed
Report a problem with this finding
Wording "can be observed with sentences of any length" and "regardless of the sentence length" are not borne out at the left edge of Figure 2, where the RNNsearch-50 curve starts below both RNNenc curves.
- p. 2 Introduction, second paragraph on page
The improvement is more apparent with longer sentences, but can be observed with sentences of any length.
- p. 9 Conclusion, third paragraph
outperforms the conventional encoder–decoder model (RNNencdec) significantly, regardless of the sentence length
- p. 5 Figure 2, left edge of plot (sentence length below about 5)
RNNsearch-50
Only the authors can say what is right here.
Judge's note: Viewed the Figure 2 crop. At length 0 the solid RNNsearch-50 line starts near 12.5 BLEU, while the dashed RNNenc-50 and dash-dot RNNenc-30 lines start near 15-16. The solid line crosses above them at about length 2-3 and clearly leads from about length 5. RNNsearch-30 (dotted) is above the baselines throughout. So the overstatement is confined to the very shortest sentences for RNNsearch-50; this is a read from a small plot and that region may be noisy.
Second check: another model agreed
Report a problem with this finding
Wording A.2.2 writes the output probability as "p(yi|si, yi−1, ci)" and Eq. (4) as "g(yi−1, si, ci)", but the introducing prose says "With the decoder state si−1" and the formula is "t̃i = Uo si−1 + Vo E yi−1 + Co ci", so the state index differs between s_i and s_{i-1}.
- p. 14 Appendix A.2.2, sentence before the output-probability equation
With the decoder state si−1, the context ci and the last generated word yi−1, we define the probability
- p. 14 Appendix A.2.2, output-probability equation
p(yi|si, yi−1, ci)
- p. 14 Appendix A.2.2, definition of t̃_i
t̃i =Uosi−1 + VoEyi−1 + Coci.
- p. 3 Sec. 3.1, Eq. (4)
p(yi|y1, . . . , yi−1, x) = g(yi−1, si, ci)
Only the authors can say what is right here.
Judge's note: Read on the page 14 and page 3 images. The left-hand side of the appendix equation and Eq. (4) condition on s_i, while the sentence before the equation and the definition of t̃_i use s_{i-1}. The paper does not say which is intended, so only the authors can settle whether the output layer uses s_i or s_{i-1}.
Second check: another model agreed
Report a problem with this finding
15 minor (typos and the like; most checked by the judge only)
Wording Conclusion says "the whole family of neural machine translation, has only been proposed as recently as this year", while the introduction credits "Kalchbrenner and Blunsom (2013)" alongside the 2014 works.
- p. 9 Conclusion, fourth paragraph
the whole family of neural machine translation, has only been proposed as recently as this year
- p. 1 Introduction, first sentence
recently proposed by Kalchbrenner and Blunsom (2013), Sutskever et al. (2014) and Cho et al. (2014b)
Judge's note: Viewed page 9 for the conclusion sentence. The introduction attribution was confirmed in the text layer (main.txt line 29) and the same 2013 citation appears on page 2. "This year" (2014 at the time of writing) is loose rhetoric that does not cover the 2013 work the paper itself cites as a proposer; a minor wording inconsistency.
Second check: another model agreed
Report a problem with this finding
Wording Table 2 caption says "NLL is the average conditional log-probabilities of the sentences", but the NLL columns hold positive values (e.g. 28.1, 53.0), so the caption omits "negative".
- p. 14 Table 2 caption
NLL is the average conditional log-probabilities of the sentences
- p. 14 Table 2, Train NLL / Dev. NLL columns
28.1
Judge's note: Caption and table read on the page 14 image. All Train NLL and Dev. NLL entries are positive (26.7 to 53.0), whereas log-probabilities cannot exceed zero. The column heading NLL makes the intended meaning recoverable, so this is a wording slip in the caption.
Second check: another model agreed
Report a problem with this finding
Typo Sec. 2.1 defines "c = q({h1, · · · , hTx})" but the example in the same paragraph reads "q({h1, · · · , hT}) = hT", using an undefined T in place of T_x.
- p. 2 Sec. 2.1, unnumbered equation after Eq. (1)
c = q ({h1, · · · , hTx})
- p. 2 Sec. 2.1, last line of the paragraph
q ({h1, · · · , hT }) = hT
Judge's note: Viewed page 2. The displayed equation uses subscript T_x; the last line of the paragraph uses plain T. Notational slip, meaning is clear.
Second check: none (judge only)
Report a problem with this finding
Typo Eq. (2) takes the product from t=1 to an undefined "T" while the next line defines "y = (y1, · · · , yTy)"; the preceding sentence indexes the word as y_{t′} but the equation uses t.
- p. 3 Sec. 2.1, Eq. (2) and the following line
where y = (y1, · · · , yTy)
Judge's note: Viewed page 3. The product's upper limit is plain T, the line below uses T_y, and the lead-in sentence uses y_{t'} and y_{t'-1}. Notational inconsistency only.
Second check: none (judge only)
Report a problem with this finding
Typo Sec. 4.2 and Table 1 name the baseline "RNNencdec-30"/"RNNencdec-50", but the Figure 2 legend, Table 2 and Table 3 (and its caption) use the undefined labels "RNNenc-30"/"RNNenc-50".
- p. 5 Sec. 4.2
(RNNencdec-30, RNNsearch-30)
- p. 5 Figure 2 legend
RNNenc-50
- p. 14 Table 2, first column
RNNenc-30
- p. 15 Table 3 caption
The translations generated by RNNenc-50 and RNNsearch-50
Judge's note: Viewed page 5 (Sec. 4.2 and the Figure 2 legend) and page 7 (Table 1). The Table 2 and Table 3 occurrences were confirmed in the text layer only (main.txt lines 909, 916, 1022, 1076), not on the page images. Evidently the same model; cosmetic label inconsistency.
Second check: none (judge only)
Report a problem with this finding
Typo Sec. 4.2: "with the sentences of length up to 50 word" (should be "words"); Appendix B.2: "In Tables 2 we present" (should be "Table 2").
- p. 5 Sec. 4.2, first paragraph
with the sentences of length up to 50 word
- p. 5 4.2
length up to 50 word
- p. 15 B.2
In Tables 2 we present
Judge's note: Viewed page 5: "up to 30 words ... up to 50 word" is on the image. "In Tables 2" was confirmed in the text layer only (main.txt line 1012), not on the page 15 image.
Second check: none (judge only)
Report a problem with this finding
Typo Sec. 5.2.1 quotes the model output as "[zone économique européen]" and two sentences later as "[zone économique européenne]"; Figure 3(a) shows "européenne".
- p. 7 Sec. 5.2.1, second paragraph
[zone économique européen]
- p. 7 Sec. 5.2.1, second paragraph
[zone économique européenne]
- p. 6 Figure 3(a), y-axis labels
européenne
Judge's note: Viewed page 7: the first occurrence is "européen]" with the closing bracket, not a line-break artefact; the second is "européenne]". The Figure 3(a) crop shows the y-axis label "européenne". The first quotation is a misspelling.
Second check: none (judge only)
Report a problem with this finding
Typo Sec. 5.2.2 quotes the source phrase as "[a medical center]" while the source sentence on the previous page reads "a medical centre".
- p. 8 Sec. 5.2.2, first line of page
until [a medical center]
- p. 7 Sec. 5.2.2, source sentence
a medical centre
- p. 7 Section 5.2.2, quoted source sentence
a medical centre to carry out a diagnosis
Judge's note: Viewed page 7 (the source sentence has "a medical centre") and the page 8 crop ("until [a medical center]"). Spelling variant in a bracketed quotation; cosmetic.
Second check: none (judge only)
Report a problem with this finding
Typo Conclusion: "One of challenges left for the future" is missing the article ("One of the challenges").
- p. 9 Sec. 7, last paragraph
One of challenges left for the future
Judge's note: Viewed page 9; the last paragraph reads as quoted.
Second check: none (judge only)
Report a problem with this finding
Typo Reference Cho et al. (2014a): "Proceedings of the Empiricial Methods in Natural Language Processing (EMNLP 2014)" — "Empiricial" should be "Empirical".
- p. 10 References, Cho et al. (2014a)
In Proceedings of the Empiricial Methods in Natural Language Processing (EMNLP 2014)
- p. 10 References, Cho et al. 2014a
Proceedings of the Empiricial Methods in Natural Language Processing
Judge's note: Viewed the crop of the page 10 reference list; the misspelling "Empiricial" is on the image.
Second check: none (judge only)
Report a problem with this finding
Typo Reference Schwenk (2012): "Proceedings of the 24th International Conference on Computational Linguistics (COLIN)" (acronym should be COLING); reference Schwenk et al. (2006): author printed as "Dchelotte".
- p. 11 References, Schwenk (2012)
Computational Linguistics (COLIN)
- p. 11 References, Schwenk et al. 2006
Dchelotte
Only the authors can say what is right here.
Judge's note: Rests on outside knowledge: that the conference acronym is COLING and that the author's surname is Déchelotte. Viewed page 11: "(COLIN)" and "Dchelotte, D." are printed as quoted. The adjacent entry writes "COLING/ACL", which supports the acronym reading from within the paper; the surname (apparently a dropped accented character) cannot be checked from the files.
Second check: none (judge only)
Report a problem with this finding
Typo Reference list prints the second author's surname as "Schwenk, H., Dchelotte, D., and Gauvain, J.-L. (2006).", apparently with an accented letter dropped.
- p. 11 References, Schwenk et al. (2006)
Schwenk, H., Dchelotte, D., and Gauvain, J.-L. (2006).
Only the authors can say what is right here.
Judge's note: Rests on outside knowledge: the cited author's surname is Déchelotte. The page 11 image does print "Dchelotte", so this is not an extraction artefact. Other accented names on the same page (e.g. "van Merriënboer") render correctly, which supports a dropped character.
Second check: none (judge only)
Report a problem with this finding
Typo Duplicated word in A.2.1: "We concatenate the forward and backward states to to obtain the annotations".
- p. 13 Appendix A.2.1, before Eq. (7)
states to to obtain the annotations
- p. 13 A.2.1
concatenate the forward and backward states to to obtain
Judge's note: The page 13 image shows "to to" in the sentence before Eq. (7).
Second check: none (judge only)
Report a problem with this finding
Typo B.1 says "All the elements of Va and all the bias vectors were initialized to zero" with an upper-case V_a, while the alignment vector is lower-case v_a in A.1.2 and A.2.2.
- p. 14 Appendix B.1
All the elements of Va and all the bias vectors were ini-
- p. 14 Appendix A.2.2
va ∈ Rn′
Judge's note: The page 14 image shows upper-case V_a in B.1 and lower-case v_a in the e_ij equation and the dimension statement on the same page. No upper-case V_a is defined on the appendix pages I read (12 to 15); the only other V is V_o. The evident fix is v_a.
Second check: none (judge only)
Report a problem with this finding
Typo B.2 reads "In Tables 2 we present the statistics related to training all the models used in the experiments", using a plural for a single table.
- p. 15 Appendix B.2, last sentence
In Tables 2 we present the statistics
- p. 15 B.2 last sentence
In Tables 2 we present the statistics related to training all the models
Judge's note: The page 15 image shows "In Tables 2". Only Table 2 (page 14) holds training statistics, so it should read "In Table 2".
Second check: none (judge only)