ROUGE: A Package for Automatic Evaluation of Summaries

• p. 2 Section 3, first sentence

  A sequence Z = [z1, z2, ..., zn] is a subsequence of

• p. 2 Section 3, first sentence

  such that for all j = 1, 2, ..., k, we have xij = zj

Judge's note: Read the first sentence of Section 3 on the page 2 image: the last element of Z is z_n, the index sequence ends at i_k and the condition is for j = 1..k. The definition only covers all of Z if Z has k elements; n and k are never equated, and n is used again a few lines later for the length of Y. The mismatch is internal to the sentence and does not need the cited textbook to establish. Notational slip, so minor.

Second check: another model agreed

• p. 4 Section 4, right column

  when k >= 0, and

• p. 4 Section 4, dynamic programming procedure

  c(i,j) = c(i-1,j-1) + f(k+1) – f(k)

• p. 4

  ( 1 n f Y X WLCS f (14) Fwlcs wlcs wlcs wlcs wlcs P R P R 2

Only the authors can say what is right here.

Judge's note: Read the procedure and the example on the page 4 image and reworked it. k = w(i-1,j-1) is 0 at the start of a run (w is initialised to 0 and reset to 0 on a non-match). With f(k) = αk − β for all k >= 0: f(k+1) − f(k) = [α(k+1) − β] − [αk − β] = α for every k including 0. So WLCS = α × (plain LCS length) and consecutive matches earn nothing extra, contrary to the sentence about a −β gap penalty. The example behaves as described only if f(0) = 0 (first match of a run adds α − β, later ones α), i.e. the linear form for k > 0. The stated property f(x+y) > f(x) + f(y) is for positive integers and is satisfied; the fault is the printed domain 'k >= 0'. The measure and the k² function actually used are unaffected, hence S3. The 'repeated wlcs' note in this cluster is an extraction artefact of the subscripts in Equation 15.

Second check: another model agreed, code check

• p. 4 Section 4, procedure step (1)

  (1) For (i = 0; i <=m; i++)

• p. 4 Section 4, step (1) of the dynamic programming procedure

  For (i = 0; i <=m; i++)

Judge's note: Read the pseudo-code on the page 4 image. Step (1) has a single loop over i and assigns c(i,j) and w(i,j) with no loop or value for j; step (2) reads c(i-1,j-1), c(i-1,j), c(i,j-1) and w(i-1,j-1) for i, j >= 1, so both row 0 and column 0 need initialising. The comments ('initialize c-table') make the intent clear, so this is an incompleteness in the pseudo-code and not a wrong result.

Second check: another model agreed

• p. 4 Section 5, first paragraph

  bigrams between a candidate translation and a set of

• p. 5 Section 5, before Equation 16

  Given translations X of

Judge's note: Read Section 5 on the page 4 and page 5 images; both passages are printed as quoted. Sections 2-4 use 'candidate summary' and 'reference summary' for the same X and Y, and the example sentences S1-S4 are those of Section 3.1. The wording is machine-translation terminology in a summarisation paper; the meaning is recoverable, so this is a consistency point.

Second check: another model agreed

• p. 7 Section 6, left column, first paragraph (Table 2 discussion)

  clusion of stopwords improved performance in gen-

• p. 6 Table 2, rows R-6 to R-9, CASE vs STOP columns

  0.55

Judge's note: Read Table 2 on the crop; the column order is CASE, STEM, STOP, each split into 1 REF and 4 REFS. R-2 to R-5 (1 REF) and all S/SU rows do improve with STOP, and R-L and R-W are unchanged, so 'in general' is defensible. But ROUGE-1 is named as the exception while four other rows drop further. The same sentence already describes ROUGE-N with N > 1 as performing significantly worse, which softens this. Wording only.

Second check: another model agreed

• p. 7 Section 7 Conclusions, finding (5)

  were increased by using multiple references.

• p. 7 Section 6, left column, first paragraph

  erences did not improve correlations.

• p. 6 Table 2, 1 REF vs 4 REFS columns

  1 REF 4REFS 1 REF 4 REFS 1 REF 4 REFS

Judge's note: Read conclusion (5) and the Table 2 paragraph on page 7, and Table 2 on the crop. The paper itself states the exception for the very short summary task and gives a reason (624 samples), and multiple references do help in Tables 1 (DUC 2001) and 3 (A1-A3). The conclusion is a summary that omits the exception the results text reports. Wording only.

Second check: another model agreed

• p. 1 Equation 1, numerator summation index

  ReferemceSummaries

• p. 1 Equation (1), numerator subscript

  S∈{ReferemceSummaries}

• p. 1

  Referemce

Judge's note: Read Equation 1 on the page 1 image: the numerator subscript is spelled 'ReferemceSummaries' and the denominator 'ReferenceSummaries'. It is in the rendered equation, not an extraction artefact.

Second check: none (judge only)

• p. 3 Section 3.1

  ROUGE-2 would prefer S4 than S3.

Judge's note: Read the end of Section 3.1 on the page 3 image; the sentence is printed as quoted. Grammar slip only.

Second check: none (judge only)

• p. 5 Section 6, first paragraph

  Acquiring human judgments are usually very expensive

Judge's note: Read the first paragraph of Section 6 on the page 5 image; the sentence is printed as quoted (hyphenated 'hu-man' across a line break). Subject-verb disagreement.

Second check: none (judge only)

• p. 6 Section 6, right column, paragraph on Table 1

  data. This might due to the double sample size in

• p. 6 Results discussion of Table 1

  This might due to the double sample size

Judge's note: Read on the page 6 image, right column. The word 'be' is absent on the page, not only in the text layer.

Second check: none (judge only)

• p. 6 Section 6, example paragraph

  the 12 systems participated in the DUC 2001 single document summarization evaluation

Judge's note: Read on the page 6 image, right column, top. Ungrammatical as printed; cosmetic.

Second check: none (judge only)

• p. 7 Table 3, lower block, first panel header

  (C) DUC02 10

• p. 7 Section 6, left column

  Table 3 C, D1, D2, E1, E2, and F show the corre-

Judge's note: Read the panel headers on the Table 3 crop and the sentence on page 7. The lettering skips B in both table and text, so the two agree with each other and nothing is mis-pointed. The gap is unexplained; cosmetic.

Second check: none (judge only)

• p. 7 Table 3, panel (A1), first sub-header

  1 RFF

Judge's note: Viewed the enlarged crop: the header above the first three A1 columns reads '1 RFF', while the matching headers in A2 and A3 read '1 REF'. It is on the page image, not an extraction artefact.

Second check: none (judge only)

• p. 7 Section 6 final paragraph

  The only tasks that had over 30 samples was from DUC 2002

Judge's note: Read the crop of page 7, right column below Table 3. The sentence is printed as quoted: "The only tasks that had over 30 samples was from DUC 2002". Either "task ... was" or "tasks ... were" is needed. Purely grammatical.

Second check: none (judge only)

• p. 8 References, first entry

  Cormen, T. R., C. E. Leiserson, and R. L. Rivest.

• p. 8 References, Radev et al.

  Radev, D. S. Teufel, H. Saggion, W. Lam, J. Blitzer, A. Gelebi

Only the authors can say what is right here.

Judge's note: Rests on outside knowledge: the first author of Introduction to Algorithms (MIT Press) is Thomas H. Cormen, so the initials would be "T. H.". The page image shows "Cormen, T. R., C. E. Leiserson, and R. L. Rivest. 1989. Introduction to Algorithms. The MIT Press." exactly as quoted. Nothing in the paper itself contradicts the printed initials. The other points in this cluster (Radev entry's missing comma and "Gelebi", the Melamed year in parentheses, "Saggion H.") are judged under C033, C034 and C035; this verdict covers the Cormen initials only.

Second check: none (judge only)

• p. 8 References, Radev et al. 2002

  zer, A. Gelebi, H. Qi, E. Drabek, and D. Liu.

• p. 8 References, Radev et al. 2002

  Radev, D.

• p. 8 References, Radev et al. 2002

  Radev, D. S. Teufel, H. Saggion, W. Lam, J. Blitzer, A. Gelebi

Only the authors can say what is right here.

Judge's note: Rests on outside knowledge: the sixth author of this Johns Hopkins workshop report is Arda Çelebi (Celebi), which is what makes "Gelebi" a misspelling. The missing comma needs no outside fact: the page image shows "Radev, D. S. Teufel" with a wide gap and no comma, where every other multi-author entry separates authors with commas; that part has an evident fix. Both points are visible on the page 8 image, right column.

Second check: none (judge only)

• p. 8 References, Melamed, Green and Turian

  Melamed, I. D., R. Green and J. P. Turian (2003).

Judge's note: Read the full page 8 image. All other entries follow the pattern "Authors. Year. Title" with a period after the author list and a bare year; this entry alone has no period after the authors and the year in parentheses. It is a formatting inconsistency only; the citation is unambiguous.

Second check: none (judge only)

• p. 8 References

  Saggion H., D. Radev, S. Teufel, and W. Lam.

Judge's note: Read the page 8 image and the crop of the right column. The entry is printed "Saggion H., D. Radev, S. Teufel, and W. Lam. 2002." Every other entry in the list has "Surname, Initials" for the first author. Cosmetic punctuation slip with an evident fix ("Saggion, H.").

Second check: none (judge only)