Crowdsourcing User Studies with Mechanical Turk
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 2008 · paper · Computer science
Investigates using Amazon's Mechanical Turk micro-task market to collect user measurements quickly and cheaply. The authors found it promising but report that tasks need careful formulation to work well, and discuss design considerations.
Checked: PDF from pensivepuffin.com, fetched 5 Oct 2026. Page numbers are those of that PDF.
2 findings
Reference or label Reference 6 "Viégas, F.B., Wattenberg, M., and McKeon, M. 2007. The Hidden Order of Wikipedia" is listed but never cited in the text.
- p. 4 References, entry 6
Viégas, F.B., Wattenberg, M., and McKeon, M. 2007.
Judge's note: Searched the full text layer and checked pages 1–4. In-text citations are [5], [3], [2], [1], [4] and [7]. There is no citation of [6] anywhere, including the Wikipedia-quality passage on page 2, which cites [4] and [7] only.
Second check: another model agreed
Report a problem with this finding
Reference or label Reference 6 venue printed as "In Proc. of HCI Interactional Conference."
- p. 4 References, entry 6
In Proc. of HCI Interactional Conference.
- p. 4 References, entry 6
In Proc. of HCI Interactional Conference
- p. 4 References, item 6
Proc. of HCI Interactional Conference
Only the authors can say what is right here.
Judge's note: Rests on outside knowledge: that the venue is HCI International (2007). The page 4 image confirms the printed word is 'Interactional', so this is not an extraction artefact. 'Interactional' looks like a garbling of 'International'. The entry also gives no pages or publisher.
Second check: another model agreed
Report a problem with this finding
9 minor (typos and the like; most checked by the judge only)
Wording Exp 2 methods say users completed "four questions that had verifiable, quantitative answers", but the questions listed are three counts (references, images, sections) plus a 4-6 keyword summary, and the Discussion says "Not all of these questions need to be quantitative", singling out the keyword question.
- p. 3 Experiment 2, method paragraph
required to complete four questions that had verifiable, quantitative answers before rating the quality of the article
- p. 3 Experiment 2, method paragraph
These questions required users to input how many references, images, and sections the article had.
- p. 3 Design Recommendations
Not all of these questions need to be quantitative
- p. 3 Experiment 2 methods
required to complete four questions that had verifiable, quantitative answers
Only the authors can say what is right here.
Judge's note: Read the page 3 Exp 2 methods paragraph and the Design Recommendations (pages 3–4). The methods name three count questions and then, 'In addition', the keyword question. The Design Recommendations treat the keyword question as one of 'the first four questions' and describe it as non-quantitative. So the methods' description of all four as 'quantitative' does not match the questions listed. This is a wording inconsistency about what was asked.
Second check: another model agreed
Report a problem with this finding
Wording Completion-time comparison reports "(4:06 vs. 1:30; t(486) = 5.14, p < .001)"; with 210 Exp 1 and 277 Exp 2 ratings, a pooled two-sample t test would have 485 df, so the df and the declared samples do not correspond.
- p. 3 Experiment 2 Results
t(486) = 5.14, p < .001
- p. 3 Experiment 2 Results
t(486) = 5.14
Only the authors can say what is right here.
Judge's note: Checked page 3 in the image. The sample sizes are 210 ratings (Exp 1, page 2) and 277 ratings (Exp 2). The ratings-per-user test t(180) matches 58+124-2. A Welch df cannot exceed 485. The paper does not say which test or data subset gives 486 df, and the text also mentions medians beside a t test. Only the authors can say which test was run. Off by one; the P value is not at issue.
Second check: another model agreed
Report a problem with this finding
Typo Abstract: "However, the costs involved in engaging users often requires practitioners" – the plural subject 'costs' needs 'require'.
- p. 1 Abstract
the costs involved in engaging users often requires practitioners
Judge's note: Checked the page 1 crop. The text reads 'the costs involved in engaging users often requires practitioners to trade off'. This is a subject–verb agreement error.
Second check: none (judge only)
Report a problem with this finding
Typo ACM Classification Keywords format the two codes differently: "H5.2 Information interfaces and presentation" vs "H.5.3 [Information Interfaces]".
- p. 1 ACM Classification Keywords
H5.2 Information interfaces and presentation
Judge's note: Checked the page 1 crop. The first code is printed 'H5.2' and the second 'H.5.3', so the format is inconsistent within the same field. This is cosmetic only.
Second check: none (judge only)
Report a problem with this finding
Typo Exp 1 results: "less time than likely needed for reading the article, let along rating it" – should be 'let alone'.
- p. 3 Experiment 1 Results, first paragraph
let along rating it
- p. 3
less time than likely needed for reading the article, let along rating it
Judge's note: The page 3 image (left column, first paragraph) shows 'let along rating it'. This is a misspelled idiom.
Second check: none (judge only)
Report a problem with this finding
Typo "In experiment 2, we tried" uses lowercase, whereas the paper elsewhere writes 'Experiment 2'.
- p. 3 Experiment 2 section
In experiment 2, we tried
Judge's note: The page 3 image shows a lowercase 'experiment 2' at the start of the second paragraph of the Experiment 2 section. Everywhere else the term is capitalised. Cosmetic capitalisation inconsistency.
Second check: none (judge only)
Report a problem with this finding
Typo Design Recommendations: "comments that are repeated verbatim across multiple tasks are indicators of suspect edits." – the paragraph is about suspect responses, not edits.
- p. 4 Design Recommendations, third point
are indicators of suspect edits
- p. 4 Design Recommendations, third point
are indicators of suspect edits.
Judge's note: Page 4 image, left column, third recommendation. Every other sentence in the paragraph refers to 'responses'. Participants rated articles and made no edits, so 'edits' is a terminology slip.
Second check: none (judge only)
Report a problem with this finding
Typo Reference punctuation slips: ref 1 "Coase's penguin,or Linux" (missing space after the comma) and ref 4 "Chi, E.H.. 2007" (doubled period).
- p. 4 References, entry 1
Coase's penguin,or Linux
- p. 4 References, entry 4
Chi, E.H.. 2007.
- p. 4 References 1-2
1. Benkler, Y. 2002. Coase's penguin,or Linux
Judge's note: The page 4 image shows both: 'penguin,or' with no space, and 'E.H..' with a doubled period. The alphabetical-order point raised in this cluster is adjudicated separately as C013.
Second check: none (judge only)
Report a problem with this finding
Typo Reference list order: "1. Benkler, Y. 2002." precedes "2. Andreasen, M. S.", although entries 2–7 are alphabetical.
- p. 4 References 1–2
1. Benkler, Y. 2002.
Judge's note: The page 4 image confirms the order Benkler, Andreasen, Fogg, Kittur, Spool, Viégas, Wikipedia. The list is not in citation order either: [5] is cited first, on page 1. So Benkler is out of place in an otherwise alphabetical list. Whether the CHI format requires alphabetical order is outside knowledge; treated as cosmetic.
Second check: none (judge only)