TurKit: Human Computation Algorithms on Mechanical Turk

• p. 2 Introduction, contributions list (Performance Evaluation bullet)

  performance drawn from a corpus of 20 scripts posting almost 30,000 tasks over the past year

• p. 8 Performance Evaluation, para 2

  29,731 assignments across 3,829 HITs

Only the authors can say what is right here.

Judge's note: Checked p2 and p8. 29,731 matches "almost 30,000" but is the assignment count; the HIT count is 3,829. The paper elsewhere uses "tasks" for HITs (e.g. "1-cent tasks" for the HITs in Fig 8). This is a wording looseness, not a data error.

Second check: another model agreed

• p. 3 Crash-and-Rerun Programming, para 1

  Every line that is successfully run is stored in a database.

• p. 9 Crash-and-Rerun Benefits, retroactive debugging

  only expensive or non-deterministic operations are recorded

• p. 5 Implementation

  The basic idea is to record a trace of once calls in an array.

Judge's note: All three crops confirm the wording. The same p3 paragraph goes on to say the programmer controls via once whether a line is retrieved or re-evaluated, so the sentence reads as an informal overstatement. Read literally, it still misdescribes the mechanism.

Second check: another model agreed

• p. 7 Figure 4

  == a[i]) { right.push(a[i])

• p. 6 Sorting / Figure 1 comparator

  return v == a ? ‐1 : 1

Judge's note: Checked the Fig 4 code (p7) and the Fig 1 comparator recalled on p6. On p6/7 the text says the serial sort has a problem and "in TurKit we can implement a parallel quicksort, as shown in Figure 4." The two orderings are opposite. The code is illustrative, so the ordering choice is minor.

Second check: another model agreed

• p. 8 Performance Evaluation, para 2

  About 0.1% complete within 10 seconds.

• p. 8 Figure 8 caption

  pleted for 2648 HITs with 1 cent reward. Five com-pleted within 10 seconds.

• p. 8 Figure 8 caption

  Five completed within 10 seconds.

Judge's note: Both refer to the same 1-cent HITs in Figure 8 and the same 10 s threshold. The figures do not match; 0.19% rounds to 0.2%. The size of the discrepancy is small.

Second check: another model agreed

• p. 8 Performance Evaluation, para 2

  Figure 8 shows the round-trip time-to-completion for 1-cent tasks

• p. 8 Figure 8 caption

  Time until the first assignment is com-pleted

• p. 8 Figure 8 caption

  Time until the first assignment is com-pleted for 2648 HITs

Only the authors can say what is right here.

Judge's note: Checked on page 8 (text, column 2, and the Figure 8 caption) and page 9. The same page says there are on average 7.8 assignments per HIT, so the time to the first assignment and the time for a task to complete are not obviously the same thing. The paper never says these are one quantity, so the authors should say which one is plotted. This is a wording ambiguity, not a wrong number.

Second check: another model agreed

• p. 9 Performance Evaluation, continuation

  script 10x slower would only be faster than 70% of hit-completion times

• p. 8 Figure 8

  1001 HITs

• p. 9 Performance Evaluation, last para

  faster than 99% of our hit-completion times

Only the authors can say what is right here.

Judge's note: The slowest script takes 10.6 s (page 8), so 10x slower is 106 s. On the page 8 image, the Figure 8 curve is at or below the 100 s line at x=1001. That suggests a 106 s script would be faster than only about 60% of times, not 70%. The plot is small and log-scaled, so I read it only roughly. The 99% claim is consistent (five HITs finished within 10 s). This needs the authors' check. The lower-case 'hit' for the acronym HIT is a minor style point and is included here.

Second check: another model agreed

• p. 1 Introduction

  (as short as 10 second)

Judge's note: The crop of p1 shows "(as short as 10 second)" with no hyphen and no noun after it, so the plural is needed.

Second check: none (judge only)

• p. 2 Related Work, Human Computation para 2

  coordinate worker’s efforts

Judge's note: The crop of p2 confirms the text. The preceding sentence talks about "many humans" working on each article, so the plural is meant.

Second check: none (judge only)

• p. 3 TurKit Script section, para 2

  Turkit Script introduces the crash-and-rerun

• p. 3 TurKit Script section, para 2

  Turkit Script introduces the crash-and-rerun programming

Judge's note: The crop of p3 shows "Turkit Script". The same page has "TurKit Script" a few lines earlier.

Second check: none (judge only)

• p. 5 Online Web Interface, para 2

  polling Mechancial Turk

• p. 5 Online Web Interface, para 2

  effectively polling Mechancial Turk

Judge's note: Visible in the crop and on the full p5 image, so it is not an extraction artefact.

Second check: none (judge only)

• p. 5 Online Web Interface, para 4

  Reseting the database is also useful

• p. 5 Online Web Interface

  Reseting the database

• p. 7 Creating Interfaces for Turkers

  accidently fill out

Judge's note: The crop of p5 shows "Reseting" next to an earlier "Resetting" in the same paragraph. The "accidently" part of reviewer note 4 is judged under C011.

Second check: none (judge only)

• p. 6 Using TurKit, first code line

  print(mturk.prompt(“Where is UIST 2010?”))

Judge's note: The crop of p6 shows curly quotes in this listing and straight quotes in mturk.prompt("What is your favorite color?", 100).

Second check: none (judge only)

• p. 7 Creating Interfaces for Turkers, continuation

  turkers may accidently fill out the form

Judge's note: Visible in the crop of p7.

Second check: none (judge only)

• p. 7 Figure 5 code

  vote(“Which describes " + X + " better?",

Judge's note: The crop of p7 shows the opening curly quote; every other string in the listing uses straight quotes.

Second check: none (judge only)

• p. 10 Exploring New Algorithms

  TurKit can aide in exploring

• p. 10 Exploring New Algorithms

  TurKit can aide in exploring human computation algorithms

• p. 10 Exploring New Algorithms, last sentence

  TurKit can be a fundamental aide in that research

• p. 10 Exploring New Algorithms

  a fundamental aide in that research

Judge's note: Both instances appear on the page 10 image in the Exploring New Algorithms paragraph. 'Aide' means an assistant (a person); the verb and the noun needed here are 'aid'.

Second check: none (judge only)

• p. 10 Reference 20

  Sorokin, A. and D. Forsyth

Judge's note: Checked the reference list on the page 10 image. Ref 20 is the only entry that inverts the second author's name.

Second check: none (judge only)