Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks

• p. 1 Abstract, second paragraph

  while maintaining the ac-

• p. 5 Table 2, NLI + STSb block

  88.33 ± 0.19

• p. 5 Section 4.3

  serve a performance drop of SBERT by about 7

• p. 5 Section 4.3, paragraph after Table 3

  we observe a performance drop of SBERT by about 7 points Spearman correlation

Only the authors can say what is right here.

Judge's note: Tables 2 and 3 read on the page 5 and 6 images. Two wording points. (1) The abstract's unqualified claim holds for STSb-only base training (84.67 vs 84.30) and roughly for AFS 10-fold, but not for the best STSb setting (about 3 points lower) or AFS cross-topic (about 7 lower); the paper's own section 4.3 says so. (2) "Drop of SBERT by about 7 points" follows a sentence comparing SBERT with BERT, so it most likely means relative to BERT (6.58 and 7.24), but as written it can be read as SBERT's own drop, which is about 23 points. Overstatement and ambiguity, not a wrong number.

Second check: another model agreed

• p. 2 Section 1, third paragraph on the page

  SBERT achieves an improvement of 11.7 points compared to InferSent and 5.5 points compared to Universal Sentence Encoder

• p. 4 Table 1, Avg. column

  76.55

Only the authors can say what is right here.

Judge's note: Table 1 read on the page 4 image: InferSent 65.01, USE 71.22, SBERT-NLI-base 74.89, SBERT-NLI-large 76.55, SRoBERTa-NLI-large 76.68. The differences are recomputed below; rounding does not turn 11.54 into 11.7 or 5.33 into 5.5. The quoted gains therefore belong to the SRoBERTa row, although the paper names SBERT and SRoBERTa separately elsewhere. No statement in the text says "SBERT" here covers the RoBERTa variant. The discrepancy is 0.2 points and does not change the direction or the size of the claim, hence minor; the authors can say which model was meant.

Second check: another model agreed

• p. 2 Introduction, paragraph 'SBERT can be adapted to a specific task'

  sets new state-of-the-art performance on a chal-

• p. 6 Table 3, 10-fold and cross-topic blocks

  SBERT-AFS-large

• p. 5 Section 4.3, results paragraph

  a performance that is nearly on-par with BERT

Only the authors can say what is right here.

Judge's note: Table 3 read on the page 6 image; section 4.3 on page 5 itself says SBERT is "nearly on-par with BERT" in 10-fold and about 7 points lower in cross-topic. The claim may be meant against earlier published results (SVR, r = 63.33) or against sentence-embedding methods only, but the sentence does not say so and no passage in the paper restricts it. Within the paper's own table the best result on this dataset is BERT's, so the wording overstates. The same sentence's claim for the Wikipedia triplet dataset is supported by Table 4.

Second check: another model agreed

• p. 5 Table 2 caption

  SBERT-NLI was pretrained on the NLI datasets, then fine-tuned on the STSb dataset.

Judge's note: Read the caption and table on the page 5 image. The name SBERT-NLI in the caption matches the label of the first-block rows, which were not fine-tuned on STSb, so the caption is loosely worded; the intended meaning is clear from the block headings. The reviewer's second point (10 seeds and 4 epochs stated for BERT only) is not a contradiction with the text, which says all systems used 10 seeds, and is not confirmed.

Second check: another model agreed

• p. 8 Section 7, after Table 7

  On CPU, InferSent is about 65% faster than SBERT.

• p. 8 Table 7

  InferSent 137 1876 ... SBERT-base 44 1378

• p. 8 Table 7

  SBERT-base 44 1378

• p. 8 Section 7, paragraph after Table 7

  On CPU, InferSent is about 65% faster than

Judge's note: Read Table 7 and the paragraph below it on the page 8 image. 137/83 = 1.65, so the 65% is the comparison with the smart-batching row; against the row labelled SBERT-base (44) InferSent is about 3.1 times as fast. The sentence says only 'SBERT', while the GPU sentence two lines later says 'SBERT with smart batching'. The other percentages in the paragraph match the table (2042/1876 = 1.09, 2042/1318 = 1.55, 83/44 = 1.89, 2042/1378 = 1.48). The number is not wrong; the comparator is left unstated. One reviewer rated this an error on the 137 vs 44 comparison; that reading is not the one the figure comes from.

Second check: another model agreed

• p. 8 Section 8, Conclusion

  On a GPU, it is about 9% faster than InferSent and about 55% faster than Universal Sentence Encoder.

• p. 8 Table 7

  SBERT-base 44 1378

Judge's note: Read Table 7 and the conclusion on the page 8 image. 2042/1876 = 1.09 and 2042/1318 = 1.55 reproduce the stated figures; 1378/1876 = 0.73 and 1378/1318 = 1.05 for the row without smart batching. Section 7 states the qualifier ('SBERT with smart batching is about 9% faster ...') and describes smart batching as part of the authors' implementation, so the conclusion is a shortened restatement that drops the qualifier. Minor wording point; a different sentence and location from C034.

Second check: another model agreed

• p. 1 Section 1, right column

  Manhatten / Euclidean distance

• p. 4 Section 4, first paragraph

  tive Manhatten and negative Euclidean distances

• p. 7 Section 5, last paragraph before Section 6

  similarity or with Manhatten / Euclidean distance

• p. 1 Introduction, right column, paragraph 'To alleviate this issue'

  similarity or Manhatten / Euclidean distance

Judge's note: Read on the page images for pages 1 and 4; the page 7 occurrence was checked in the extracted text only. Consistent misspelling, not an extraction artefact. The cluster also quotes "has set" and "can further improved", which are handled in C002 and C009.

Second check: none (judge only)

• p. 1 Abstract, first sentence

  has set a new state-of-the-art

• p. 1 Abstract

  that use siamese and triplet net-

Judge's note: Both read on the page 1 image. Plain subject-verb disagreement in each.

Second check: none (judge only)

• p. 1 Introduction

  Similar, finding which of the over 40 mil-

Judge's note: Read on the page 1 image, right column. Adjective used where an adverb is needed.

Second check: none (judge only)

• p. 1 Introduction

  similarity measures can be performed extremely

Judge's note: Read on the page 1 image, right column, last paragraph.

Second check: none (judge only)

• p. 1 Introduction

  n = 10 000 sentences

• p. 1 Introduction

  collection of 10,000

Judge's note: Read on the page 1, 2 and 4 images. The spaced form appears only in the math-mode expressions, so it is probably a typesetting choice for formulas; either form is legitimate, the mix is a minor style inconsistency.

Second check: none (judge only)

• p. 2 Section 2, right column, top

  Textual Semilarity (STS) benchmark

Judge's note: Read on the page 2 image, top of the right column. Printed as "Semilarity".

Second check: none (judge only)

• p. 2 Section 2, right column

  the performance of BERT can further improved by

• p. 2 Section 2

  of BERT can further improved by

• p. 2 Section 2

  can further improved by

• p. 4 Section 4.2

  provides is a popular dataset

Judge's note: The page 2 and page 4 items were read on the page images. The page 8 item ("MAX strategy perform") was checked in the extracted text only. All are plain copy-edit errors.

Second check: none (judge only)

• p. 2 Section 2, right column, second paragraph

  this limitations

• p. 2 Section 2

  bypass this limitations

Judge's note: Read on the page 2 image, right column. The words fall across a line break but both are printed as quoted.

Second check: none (judge only)

• p. 2 Paragraph on paper structure

  some design aspect of SBERT

Judge's note: Read on the page 2 image, left column, structure paragraph.

Second check: none (judge only)

• p. 2 Paper structure paragraph

  Section 3 presents SBERT, section 4 evaluates

Judge's note: Read on the page 2 image. The capitalised forms are both sentence-initial, so the paper may simply use lower-case "section" mid-sentence throughout; that is a consistent rule, though it departs from the usual capitalised cross-reference. Style point only.

Second check: none (judge only)

• p. 6 Section 5, bullet MR

  snippets on a five start scale

• p. 2 Section 2, Related Work

  Textual Semilarity (STS) benchmark

• p. 4 Section 3.1

  eintailment

• p. 6 Section 5, MR task description

  on a five start scale

Judge's note: Read on the page 2 image, right column, top: "Semantic / Textual Semilarity (STS) benchmark". "Semantic" is present on the preceding line, so only the misspelling stands. The other two misspellings bundled into this cluster are judged under C017 ("eintailment") and C028 ("five start scale"); this verdict covers "Semilarity" only.

Second check: none (judge only)

• p. 7 Section 6

  SNLI and the Multi-NLI dataset

• p. 4 Section 3.1

  MultiNLI contains 430,000 sentence pairs

• p. 2 Related Work

  methods like SkipThought

Judge's note: Checked on the images of pages 2, 4 and 7. All the forms appear as quoted. Stylistic inconsistency in naming; no ambiguity about what is meant.

Second check: none (judge only)

• p. 4 Section 3.1, continued under Table 1

  eintailment

• p. 4 Section 3.1, SNLI description

  contradiction, eintailment, and neu-

Judge's note: The page 4 image shows the italic word spelled "eintailment", so this is in the print and not an extraction artefact.

Second check: none (judge only)

• p. 4 Section 4.2, first sentence

  vides is a popular dataset

• p. 4 Section 4.2

  provides is a popular dataset

• p. 4

  provides is a popular dataset to evaluate supervised

Judge's note: Read on the page 4 image: "pro-/vides is a popular dataset". One of the two verbs is left over.

Second check: none (judge only)

• p. 4 Section 4.1, right column

  The results shows that directly using the output

Judge's note: Read on the page 4 image, right column.

Second check: none (judge only)

• p. 4 Table 1, fourth row

  InferSent - Glove

• p. 5 Table 2

  InferSent - GloVe

Judge's note: Checked the row labels on the images of pages 4, 5, 6 and 7. Only Table 1 has the lower-case v. Same model (STSb 68.03 in Tables 1 and 2). Cosmetic.

Second check: none (judge only)

• p. 4 Table 1 caption

  for various Textual Similarity (STS) tasks

Judge's note: Caption read on the page 4 image. Section 1 and Section 4 on the same page give "Semantic Textual Similarity (STS)". A missing word in the expansion of the abbreviation.

Second check: none (judge only)

• p. 5 Section 4.2, paragraph after Table 2

  the later strategy leads to a

Judge's note: Read on the page 5 image. The preceding sentence lists two setups, and the sentence refers to the second.

Second check: none (judge only)

• p. 6 Section 5

  The purpose of SBERT sentence embeddings

Judge's note: Read on the page 6 image, right column under Table 4. Grammar slip; the meaning is clear.

Second check: none (judge only)

• p. 6 Section 5, MR task description

  on a five start scale

• p. 6 Section 5, MR task bullet

  five start scale

• p. 6

  snippets on a five start scale

Judge's note: Read on the page 6 image: "snippets on a five start scale". In the print, not an extraction artefact.

Second check: none (judge only)

• p. 7 Table 5, TREC column

  Avg. GloVe embeddings

• p. 6 Table 4

  mean-vectors

Judge's note: Both points are as described on the images of pages 6 and 7. The baselines in Table 4 are taken from Dor et al. and may carry that source's precision, and one-decimal TREC values may also be legitimate. A presentation inconsistency only; no value is shown to be wrong.

Second check: none (judge only)

• p. 8 Section 6, last paragraph

  strategy perform

• p. 8 Section 6

  the MAX strategy perform

Judge's note: Read on the page 8 image, left column, last paragraph of Section 6. Singular subject 'the MAX strategy' with 'perform'. The similar slip the reviewer mentions on page 6 ('The purpose of SBERT sentence embeddings are not to be used') is also in the text.

Second check: none (judge only)

• p. 8 Section 7

  computed for Millions of sentences

• p. 8 Conclusion

  as around 50 Million sentence

Judge's note: Read both page 8 instances on the page image; the page 6 instance and the lower-case form in the abstract were found in the text layer (the reviewer's 'over 40 million' is not the wording; the abstract has 'about 50 million'). Cosmetic inconsistency in capitalisation.

Second check: none (judge only)

• p. 9 References, Cer et al. 2017

  Iigo Lopez-

• p. 9 References, Agirre et al. 2015

  Inigo Lopez-Gazpio

• p. 10 Humeau et al. reference

  arXiv:1905.01969,

Judge's note: Checked on the crops of page 9: the printed page shows 'Iigo Lopez-Gazpio' in the Cer et al. 2017 entry and 'Inigo Lopez-Gazpio' in the Agirre et al. 2015 entry, so this is not a text-layer loss and the list is internally inconsistent; a character has been dropped in 'Iigo'. That the intended spelling carries an ñ is outside knowledge and is not needed for the finding. The second reviewer note's point about duplicated arXiv identifiers is the subject of C038 and is judged there.

Second check: none (judge only)

• p. 10 References, Humeau et al. 2019

  arXiv preprint arXiv:1905.01969, abs/1905.01969.

• p. 10 References, Reimers and Gurevych 2018

  abs/1803.09578.

• p. 10 References, Schroff et al. 2015

  arXiv:1503.03832, abs/1503.03832.

• p. 11 References, Yang et al. 2019

  arXiv:1906.08237, abs/1906.08237.

Judge's note: Read the three entries on the page 10 image and the Yang et al. entry on the page 11 crop. Each carries a trailing 'abs/<id>' after the arXiv number; the other arXiv entries in the list (e.g. Johnson et al. 2017, Liu et al. 2019, May et al. 2019, Qiao et al. 2019, Zhang et al. 2019) do not. Cosmetic bibliography artefact; the references remain findable.

Second check: none (judge only)