False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant
PsycEXTRA Dataset, 2011 · paper · Psychology
Shows through simulations and two experiments that flexibility in data collection, analysis and reporting greatly raises false-positive rates, making it easy to present false hypotheses as significant. Proposes disclosure requirements for authors and guidelines for reviewers as a remedy.
Checked: PDF from pdf.stafforini.com, fetched 5 Oct 2026. Page numbers are those of that PDF.
1 finding
Reference or label Study 1's outcome is a five-option rating ("very young, young, neither young nor old, old, and very old"), yet its adjusted means are given in years: "(adjusted M = 2.54 years) than after listening to the control song (adjusted M = 2.06 years)". These are scale points, not years.
- p. 2 Study 1, method
by choosing among five options (very young, young, neither young nor old, old, and very old)
- p. 2 Study 1, results
(adjusted M = 2.54 years)
- p. 2 Study 1, results
(adjusted M = 2.06 years)
- p. 2 Study 1, ANCOVA result
(adjusted M = 2.54 years) than after listening to the control song (adjusted M = 2.06 years)
Judge's note: I checked p.2 in the crop C001-d and the extracted text. The method describes a five-option ordinal answer to "How old do you feel right now?" and says nothing about converting it to years. Means of 2.06 and 2.54 fit a 1-5 coding. Study 2's "years" is correct because its outcome is age from birth date. The paper may have reused Study 2's wording, and a deliberate joke can't be ruled out, but as printed the unit is wrong.
Second check: another model agreed
Report a problem with this finding
2 minor (typos and the like; most checked by the judge only)
Typo The band is written "performed by The Wiggles" on p.2 but "“Hot Potato” by the Wiggles" in Table 3 (p.6), where the same sentence has "by The Beatles".
- p. 2 Study 1, method
performed by The Wiggles
- p. 6 Table 3, gray text
“Hot Potato” by the Wiggles
Judge's note: The Table 3 crop shows lowercase "the Wiggles" next to "The Beatles". Page 2's text has "The Wiggles". This is only an inconsistency in capitalising a proper name.
Second check: none (judge only)
Report a problem with this finding
Typo "Á la Bonferroni" has an acute accent. The French phrase takes a grave accent: "À la".
- p. 7 Nonsolutions, Correcting alpha levels, first sentence
Á la Bonferroni
- p. 7 Nonsolutions, 'Correcting alpha levels' paragraph
Á la Bonferroni, one may consider
Judge's note: Crop C004-a on p.7 clearly shows an acute accent on the capital A, so this is not an extraction artefact. It is a cosmetic diacritic error.
Second check: none (judge only)