Deep Contextualized Word Representations

• p. 5 Table 1 caption

  Due to the small test sizes for NER and SST-5, we report the mean

• p. 5 Table 1, SNLI row

  88.7 ± 0.17

• p. 1 Introduction, last paragraph (also first paragraph and abstract)

  alone significantly improves the state of the art

• p. 5 Table 1, SNLI row, Previous SOTA

  88.6

Judge's note: Pages 1 and 5 show these values as quoted. "Significantly" is probably meant informally, and the other five tasks show clear gains, so this is a wording overstatement for one row only. The reviewer's second point, about the caption's coverage of the ± entries, is handled under C011.

Second check: another model agreed

• p. 4 Section 3.3, last paragraph

  weights by adding λ‖w‖

• p. 3 Section 3.2, after Eq. (1)

  are softmax-normalized weights and

Judge's note: Page 4 shows λ‖w‖₂² in Section 3.3, and the page 3 crop shows Eq. (1) with s^task. The symbol w is not introduced anywhere, so it is unclear whether the penalty applies to the normalised weights or to the pre-softmax parameters. This is a notational ambiguity only; the intended meaning (pre-softmax weights) can be inferred. I did not verify the reviewer's separate point about Θ_e versus Θ^task.

Second check: another model agreed

• p. 5 Table 1, NER row

  2.06 / 21%

Judge's note: Read on the page 5 image: baseline 90.15, ELMo + baseline 92.22 ± 0.10, increase 2.06. The 0.01 gap is consistent with subtracting unrounded means, so either form could be legitimate.

Second check: another model agreed

• p. 5 Table 1 caption

  Due to the small test sizes for NER and SST-5, we report the mean

• p. 5 Table 1, SNLI row

  88.7 ± 0.17

• p. 5 Section 4, Textual entailment

  0.7% across five random seeds

Judge's note: Read on the page 5 image. The SNLI row carries a ± value that the caption does not mention, although the body text does explain that SNLI was averaged over five seeds. The caption is incomplete, not contradictory.

Second check: another model agreed

• p. 6 Table 2, SQuAD row, All layers λ=0.001

  85.2

• p. 6 Table 3, SQuAD row, Input & Output

  85.6

• p. 9 Table 7, SQuAD row, ELMo + GloVe

  85.6

Only the authors can say what is right here.

Judge's note: Read on the page 6 and page 9 images. The baseline is 80.8 in both Table 2 and Table 7, and SNLI is 89.5 in all three tables, yet the SQuAD figure in Table 2 matches neither placement in Table 3. I found no statement in the text saying Table 2 used a different configuration or a separate run. A separate run is a plausible explanation, so this is a question for the authors.

Second check: another model agreed

• p. 6 Table 2, SRL row, All layers λ=0.001

  84.8

• p. 6 Table 3, SRL row, Input Only

  84.7

• p. 9 Table 7, SRL row, ELMo + GloVe

  84.7

Judge's note: Read on the page 6 and page 9 images. The baseline is 81.6 in both Table 2 and Table 7. The 0.1 difference is unexplained in the text but is consistent with a separate run or seed, so either value could be legitimate.

Second check: another model agreed

• p. 8 Figure 2 caption

  malized weights less then 1/3 are hatched with hori-

• p. 8 Figure 2 caption

  Nor-malized weights less then 1/3 are hatched

• p. 8 Figure 2 caption

  those greater then 2/3 are speckled

Judge's note: Viewed the page 8 image and the crop. The hatched cells (e.g. the Token row and LSTM 2 for SRL) and the lower part of the colour bar carry slanted lines, not horizontal ones. The caption misdescribes the pattern; the figure's meaning is unaffected. The "then" for "than" slips in the same caption are handled under C017.

Second check: another model agreed

• p. 1 Introduction, right column

  ficial, allowing the learned models select the types

• p. 1 Introduction, third paragraph

  allowing the learned models select the types

Judge's note: Read on the page 1 image, right column, second paragraph. The word "to" is missing as printed.

Second check: none (judge only)

• p. 2 Section 2 Related work, right column

  tags then second layer. Finally, the top layer of an

Judge's note: Read on the crop of page 2. The text prints "then" for "than" and omits the article.

Second check: none (judge only)

• p. 8 Figure 2 caption

  zontal lines and those greater then 2/3 are speckled

• p. 4 Section 4, Question answering

  larger then the 1.8% improvement

• p. 6 Section 5, overview

  tently provides richer representations then CoVe

• p. 8 Section 5.3, WSD

  WSD then the first layer

Judge's note: Each instance was read on the page image or crop: p2 Related work, p4 Question answering, p6 Section 5 overview, p8 WSD paragraph and the Figure 2 caption. All are comparatives that need "than". The p2 instance also lacks "the" before "second layer". Cosmetic.

Second check: none (judge only)

• p. 4 Section 4, Question answering paragraph

  larger then the 1.8% improvement from adding

• p. 4 Question answering

  significantly larger then the 1.8% improvement

Judge's note: Read on the page 4 image, right column, last lines of the paragraph.

Second check: none (judge only)

• p. 4 Section 4, Question answering paragraph

  A 11 member ensem-

Judge's note: Read on the page 4 image. The article "A" is printed before "11".

Second check: none (judge only)

• p. 4 Section 4 Evaluation, first paragraph

  verse set model architectures and language under-

Judge's note: Read on the page 4 image, Section 4, first paragraph. The first sentence of the same paragraph has the correct form, "diverse set of six benchmark NLP tasks".

Second check: none (judge only)

• p. 6 Section 5 Analysis, intro paragraph

  mation is captured a higher layers, consistent with

• p. 6 Section 5, overview paragraph

  mation is captured a higher layers

• p. 6 Sec 5 intro

  semantic information is captured a higher layers

Judge's note: Read on the page 6 image, Section 5 introductory paragraph. The same sentence has the correct form, "at lower layers".

Second check: none (judge only)

• p. 6 Section 5 Analysis, intro paragraph

  tently provides richer representations then CoVe.

• p. 6 Sec 5 intro

  richer representations then CoVe

Judge's note: Read on the page 6 image, Section 5 introductory paragraph.

Second check: none (judge only)

• p. 8 Section 5.5

  coreference and SQuAD, the this is strongly

• p. 8 Sec 5.5

  For coreference and SQuAD, the this is strongly

• p. 8 Sec. 5.5

  the this is strongly

• p. 8

  For coreference and SQuAD, the this is strongly favored

Judge's note: Read on the page 8 image, right column, Section 5.5. The words "the this" are printed as quoted; it is not an extraction artefact.

Second check: none (judge only)

• p. 8 Section 5.3, POS tagging paragraph

  of model capacity, this is direct test of the biLM

• p. 8 Section 5.3, POS tagging

  this is direct test of the biLM

• p. 8 Section 5.4

  formance of baselines models with and without

• p. 8 POS tagging

  this is direct test of the biLM’s

Judge's note: Both phrases were read on the page 8 image (POS tagging paragraph in the left column; Section 5.4 continuation in the right column). Cosmetic grammar slips.

Second check: none (judge only)

• p. 9 Sec 5.7

  results presented in Sec.4 include

Judge's note: The crop of page 9 shows "Sec.4" with no space. A text search shows every other section reference is printed as "Sec. N". Purely typographic.

Second check: none (judge only)

• p. 10 References, Kumar et al. 2016

  Iyyer, Ishaan Gulrajani James Bradbury, Victor

• p. 10 References, Kumar et al.

  Ishaan Gulrajani James Bradbury

Only the authors can say what is right here.

Judge's note: Rests on outside knowledge: that Ishaan Gulrajani and James Bradbury are two separate authors of the cited work. The crop shows the text as quoted, with no comma between the two names while every other author in the entry is comma-separated, so a missing separator is very likely.

Second check: none (judge only)

• p. 10 References, Kim et al.

  models. In AAAI 2016.

Judge's note: The crop shows the entry as quoted, and it is cited in the text as Kim et al. (2015). The two years disagree within the entry. Either could be legitimate (a preprint year beside the conference year), so this is a consistency point only.

Second check: none (judge only)

• p. 11 References

  Squad: 100, 000+ questions

Judge's note: The crop of page 11 shows "Squad: 100, 000+" as quoted. The article body uses "SQuAD" throughout, so the lower-casing looks like a bibliography-style artefact, as with "lstm" in other entries. Cosmetic.

Second check: none (judge only)