KLEE: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs

• p. 2 Introduction

  In total, we ran KLEE on more than 452 programs

• p. 1 Abstract

  applying it to 452 applications

• p. 15 Conclusion

  we used KLEE to check 452 applications

Judge's note: I viewed p2. 89+279+84=452 accounts for exactly 452. The HiStar kernel or other runs could justify "more than", but the wording is inconsistent with the other two statements.

Second check: another model agreed

• p. 2 Introduction, contribution item 5

  all COREUTILS bugs were confirmed and fixed within two days

• p. 10 Section 5.2.3

  While one bug (in seq) had been fixed in the develop- ers’ unreleased version, the other bugs were confirmed and fixed within two days of our report.

Judge's note: I viewed p10 §5.2.3 and p2 item 5. The Introduction's "all" slightly overstates §5.2.3, which excludes seq because it had already been fixed. This is a minor overstatement.

Second check: another model agreed

• p. 6 Section 3.3, paragraph after Figure 2

  the combined optimizations reduce this by almost 300%

Only the authors can say what is right here.

Judge's note: I viewed p6. A share drop from 92% to 41% is about 55% relative (a factor of 2.2). Table 1 STP time goes from 281 s to 10 s, about 96% lower (a factor of 28). Neither gives 300%. The authors should say what quantity and ratio is meant.

Second check: another model agreed

• p. 9 Table 2 caption

  Number of COREUTILS tools which achieve line coverage in the given ranges

Judge's note: The p.9 image shows Table 2 with COREUTILS and BUSYBOX column groups, and the caption names only COREUTILS. The 'last rows shows' grammar point is handled in C019.

Second check: another model agreed

• p. 12 Section 5.3

  over 90.5% total line coverage

• p. 9 Table 2, Overall cov., BUSYBOX KLEE

  90.5%

Judge's note: The p.12 crop shows 'over 90.5%' and, later in the same paragraph, '90.5% total line coverage'. Table 2 (p.9) gives Overall cov. 90.5% for BUSYBOX KLEE tests, and the intro also says 90.5%. 'Over' may be a leftover from 'over 90%'. The unrounded value could exceed 90.5%, so this is only a consistency point.

Second check: another model agreed

• p. 10 Section 5.2.3

  is positive, width will be less than T since 0 ≤ input position mod T < T

Only the authors can say what is right here.

Judge's note: Reworked from the p.10 image. width = T − m with 0 ≤ m < T gives T ≥ width > 0. When m = 0 (input_position a multiple of T), width = T, so the strict 'less than T' is false as printed. The point being made (no out-of-bounds write for non-negative input_position) still stands with ≤ T, so the stated result is unaffected.

Second check: another model agreed

• p. 5 Section 3.3, Counter-example Cache

  structure of Hoffmann and Hoehler [28]

• p. 16 Reference [28]

  HOFFMANN, J., AND KOEHLER, J.

• p. 16 Reference [28]

  HOFFMANN, J., AND KOEHLER, J. A new method to index and query sets.

• p. 5 §3.3 Counter-example Cache

  Hoffmann and Hoehler [28]

Judge's note: The p5 image shows "Hoehler"; the p16 reference list shows "KOEHLER". The internal inconsistency is a misspelling in the text.

Second check: none (judge only)

• p. 6 Section 3.4, end of Random Path Selection paragraph

  Note that the simply selecting a state at random

Judge's note: The p6 image shows "Note that the simply selecting"; it should read "Note that simply selecting".

Second check: none (judge only)

• p. 7 Section 4.1 code examples

  int fd = open("/etc/fstab", O_RDNLY);

• p. 7 4.1 code examples

  open("/etc/fstab", O_RDNLY)

• p. 7 4.1 code examples

  open(argv[1], O_RDNLY)

Judge's note: The crops of p.7 show O_RDNLY in both code lines. This is a typeset misspelling, not an extraction artefact.

Second check: none (judge only)

• p. 8 Section 5.1, item 2

  can only hit a small a fraction

Judge's note: The p.8 crop shows "a small a fraction" as printed.

Second check: none (judge only)

• p. 9 5.1 end

  only emitting tests cases

• p. 9 Table 2 caption

  The last rows shows the aggregate

Judge's note: The p.9 crop shows both phrases as printed.

Second check: none (judge only)

• p. 10 5.2.2

  the output pro-duces by a different implementation

Judge's note: The p.10 crop shows 'pro-duces by'. The hyphen is a line break, but the verb form is wrong.

Second check: none (judge only)

• p. 11 5.2.4

  made paths ran 7x to 220x more slowly

Judge's note: The p.11 crop shows 'made paths ran' as printed.

Second check: none (judge only)

• p. 13 Section 5.5, crosschecking infrastructure

  a test case that can be run to natively to confirm the difference

• p. 13 5.5

  can be run to natively to confirm the difference

Judge's note: The p.13 crop shows 'can be run to natively' as printed.

Second check: none (judge only)

• p. 15 Acknowledgements

  We thank our shepard Terence Kelly

• p. 15 Acknowledgements

  our shepard Terence Kelly

Judge's note: The p.15 crop shows 'shepard' as printed.

Second check: none (judge only)

• p. 15 Section 7 Conclusion, first paragraph

  ”out of the box.”

Judge's note: The p.15 crop shows a reversed opening quote. Section 5.2.1 (p.9) typesets it correctly as “out of the box”.

Second check: none (judge only)

• p. 15 Section 7 Conclusion, second paragraph

  its reasoning allow it to prove properties of paths

• p. 15 7 Conclusion

  its reasoning allow it to prove properties

• p. 15

  its reasoning allow

Judge's note: Checked the crop of p.15, Conclusion, second paragraph. The printed text reads "because KLEE's constraints have no approximations, its reasoning allow it to prove properties of paths". The subject 'reasoning' is singular, so the verb should be 'allows'.

Second check: none (judge only)

• p. 15 Section 7 Conclusion, first paragraph

  Our system KLEE, automatically generated tests

Judge's note: Checked the crop of p.15, Conclusion, first paragraph. The printed text reads "Our system KLEE, automatically generated tests that, on average, covered...". There is a single comma between subject and verb and no closing comma for an appositive. Punctuation slip.

Second check: none (judge only)

• p. 16 Reference [36]

  In In 5th joint meeting of the European

• p. 16 Reference [36]

  In In 5th joint meeting of the European Software Engineering Conference

• p. 16 Reference [36]

  In In 5th joint meeting

Judge's note: Checked the page-16 crop. The image shows "In" in roman type followed by "In 5th joint meeting" in italics. The word really is printed twice, so this is not an extraction artefact.

Second check: none (judge only)

• p. 16 Reference [20]

  Proceedings of the 21th ACM Symposium on Operating Systems Principles (SOSP 2007)

• p. 16 Reference [20]

  21th ACM Symposium on Operating Systems Principles

Judge's note: Checked the page-16 crop. Reference [20] prints "21th". Whatever the true edition number is, '21th' is not a valid ordinal, so the fix is '21st'. Nearby references use valid ordinals ("20th" in [21], "29th" in [34]).

Second check: none (judge only)