Neural Machine Translation of Rare Words with Subword Units

• p. 10 References, Liang 1983

  Ph.D. thesis, Stanford University, De- partment of Linguistics, Stanford, CA.

• p. 10 References, Liang 1983

  partment of Linguistics, Stanford, CA.

Only the authors can say what is right here.

Judge's note: Rests on outside knowledge: that Franklin M. Liang's thesis 'Word Hy-phen-a-tion by Com-put-er' was submitted to Stanford's Department of Computer Science (report STAN-CS-83-977). The crop of the page 10 reference list confirms the entry prints "Department of Linguistics". The true department cannot be checked from the files, so this is a query for the authors.

Second check: another model agreed

• p. 1 Abstract, last sentence

  by up to 1.1 and 1.3 BLEU, respectively

• p. 8 Table 3, BLEU single, rows WDict and C2-50k

  20.9

• p. 1 Abstract, last sentence

  by up to 1.1 and 1.3 BLEU, respectively.

Judge's note: Read Tables 2 and 3 on the page images. Ensemble gains are 25.3-24.2 = 1.1 (EN→DE) and 24.1-22.8 = 1.3 (EN→RU), and Section 4.2 says 'the subword ensembles outperform the WDict baseline by 0.3–1.3 BLEU', so the abstract quotes ensemble figures. The numbers are correct; only the missing 'ensemble' qualifier beside 'up to' is at issue, since the single-model EN→RU gain is 1.8.

Second check: another model agreed

• p. 4 Algorithm 1, vocab definition

  vocab = {'l o w </w>' : 5, 'l o w e r </w>' : 2,

• p. 4 Figure 1 caption

  tionary {‘low’, ‘lowest’, ‘newer’, ‘wider’}

• p. 4 Section 3.2, left column, last paragraph

  In our example, the OOV ‘lower’ would be segmented into ‘low er·’.

• p. 4 Figure 1 caption

  learned from dictionary {‘low’, ‘lowest’, ‘newer’, ‘wider’}

Judge's note: Checked on the page 4 image. Figure 1 is internally consistent with its own caption: on the unweighted dictionary low, lowest, newer, wider the pairs (r,·), (l,o), (o,w), (e,r) each occur twice, so the merges r·, lo, low, er· are a valid output, and 'lower' is out of vocabulary and segments as 'low er·'. So the figure and the OOV sentence are not wrong, and reviewer claims that the merges fit the code's words or that the caption is the mismatched item are incorrect. What remains is that Algorithm 1's own vocabulary would give a different first merge ((e,s), count 6+3 = 9, against (r,</w>) at 2) and contains 'lower', and the text moves from Algorithm 1 to 'our example' without saying the dictionaries differ. Possible reader confusion, not an error.

Second check: another model agreed

• p. 8 Section 5.2, first paragraph

  tem fails for all of the examples, either by delet- ing content (health)

• p. 8 Section 5.2, first paragraph

  search→Fo|rs|ch|un|g

• p. 9 Table 4, row C2-50k (first example)

  Fo|rs|ch|un|gs|in|st|it|ut|io|ne|n

Judge's note: Read Section 5.2 on page 8 and Table 4 on page 9. The C2-50k row reads Fo|rs|ch|un|gs|in|st|it|ut|io|ne|n, with no 'Gesundheits', so it drops the same content as WDict (Forschungsinstitute); only the two BPE systems translate 'health'. The text does not say outright that the C2-50k output is fully correct: its parenthesis claims only that research→Forschung is learned despite oversplitting, which the table supports. So the contrast is left imprecise, not contradicted. The 'g' against 'gs' difference is consistent with the text quoting only the stem 'Forschung', but it cuts through a bigram unit shown in the table. Both points are minor.

Second check: another model agreed

• p. 8 Section 5.2

  encodes Mirza-yeva inconsistently for the two language pairs

Judge's note: Read the paragraph on the page 8 image and Table 5 on page 9. The example contrasts the English source segmentation Mirz|ayeva with the Russian target segmentation Мир|за|ева within the single English→Russian pair; the preceding sentence speaks of "non-consistent segmentations between source and target text". The paper's two language pairs are English→German and English→Russian, and the example does not involve German. A wording slip with an evident fix; the meaning is recoverable from context.

Second check: another model agreed

• p. 1 Introduction, right column, first paragraph

  Abwasser|behandlungs|anlange

• p. 3 Section 3.1, right column, top

  Abwasserbehand- lungsanlange

• p. 3 Section 3.1, right column

  lungsanlange

• p. 3 Section 3.1

  Abwasserbehand-

Judge's note: Both page images show the extra 'n' ('anlange'), so it is not an extraction artefact. The paper's own gloss 'sewage water treatment plant' fixes the intended word as ...anlage. Spelling slip only, so S4.

Second check: none (judge only)

• p. 1 Introduction

  morphological changes and transliteration is often required

Judge's note: Read on the page 1 image, right column. Compound plural subject with a singular verb.

Second check: none (judge only)

• p. 5 Section 4, settings

  τ = 30000

• p. 5 Section 4, settings

  K = 30000

• p. 1 Introduction

  30 000–50 000

Judge's note: Confirmed on the page 1 and page 5 images. The two ungrouped values are in math mode; the table captions also print 'n = 44085' and 'n = 55654' ungrouped in math mode, so this looks like a math-mode typesetting habit. Cosmetic style inconsistency; either form is legitimate.

Second check: none (judge only)

• p. 3 Section 3.2, second paragraph

  word symbol ‘·’, which allows us to restore the

• p. 4 Algorithm 1, vocab definition

  'n e w e s t </w>':6, 'w i d e s t </w>':3}

Judge's note: Checked the page 3 and page 4 images. The prose and Figure 1 use '·', the code uses '</w>'. The meaning is recoverable and either form is legitimate (a typeset shorthand beside code), so this is a notation consistency point only.

Second check: none (judge only)

• p. 5 Section 4, left column, training paragraph

  for each models, once with cut-off for gradient

• p. 5 Section 4

  for each models

Judge's note: Read on the page 5 image, left column, training paragraph.

Second check: none (judge only)

• p. 8 Table 3, row BPE-60k, column unigram F1 rare

  55.3 29.7 15.6

• p. 8 Table 3, row BPE-J90k, column unigram F1 rare

  55.8 29.7 18.3

• p. 8 Table 3 caption

  Best NMT system in bold.

• p. 8 Table 3, unigram F1 rare column

  29.7

Judge's note: Checked on the page 8 image and the enlarged crops. In the rare column 29.7 is plain for BPE-60k and bold for BPE-J90k; in the BLEU ens-8 column 24.1 is bold in both tied rows. The two 29.7 values may differ before rounding, which would justify a single bold, and the paper does not say either way, so this is a minor consistency point and not a data error.

Second check: none (judge only)

• p. 9 Table 4 caption

  Table 4: English→German translation example.

• p. 8 Section 5.2, first sentence

  Table 4 shows two translation examples

• p. 8 Section 5.2, first sentence

  Table 4 shows two translation examples for

Judge's note: Verified on the page 9 image: Table 4 has two blocks (health research institutes; asinine situation) and its caption uses the singular, while the Table 5 caption directly below reads "translation examples". The page 8 text says "two translation examples". Cosmetic.

Second check: none (judge only)

• p. 9 Table 4, last row

  BPE-J90K

Judge's note: Verified on the Table 4 crop and the page 9 image: the first block has BPE-J90k and the second block BPE-J90K. Table 5 on the same page and Table 3 on page 8 use the lower-case k. Cosmetic.

Second check: none (judge only)

• p. 10 References, Ling et al. 2015a

  Ramon Fermandez

Only the authors can say what is right here.

Judge's note: Rests on outside knowledge: that the fifth author of 'Finding Function in Form' (EMNLP 2015) is Ramón Fernandez Astudillo. The enlarged crop of the page 10 reference list prints "Ramon Fermandez" with an 'm'. The correct spelling cannot be verified from the files; the spelling 'Fermandez' is an unusual form and likely a typo, possibly inherited from the bibliographic record.

Second check: none (judge only)