Measuring the Prevalence of Questionable Research Practices with Incentives for Truth Telling
2012 · paper · Psychology
Surveys over 2,000 psychologists anonymously, with incentives for honest answers, about questionable research practices. Admitted rates were surprisingly high, and estimates suggest some practices may be the prevailing research norm.
Checked: PDF from cmu.edu, fetched 5 Oct 2026. Page numbers are those of that PDF.
2 findings
Reference or label "nearly 1 in 10 research psychologists has introduced false data into the scientific record (Items 5 and 10)" cites Item 5 (rounding off p values, Fig. 1 geometric mean 39%) alongside Item 10 (falsifying data, 9%); only Item 10 fits 'nearly 1 in 10'.
- p. 3 Prevalence estimates, last paragraph
nearly 1 in 10 research psychologists has introduced false data into the scientific record (Items 5 and 10)
- p. 4 Fig. 1 labels
39%
- p. 3 Prevalence estimates, last paragraph
has introduced false data into the scientific record (Items 5 and 10)
- p. 4 Fig. 1
Rounding Down p Values
Only the authors can say what is right here.
Judge's note: I checked Fig. 1 on p.4. The geometric means are 39% above 'Rounding Down p Values' (Item 5) and 9% above 'Falsifying Data' (Item 10). Table 1 (p.2) confirms that Item 5 is 'rounding off' a p value. The 'nearly 1 in 10' figure corresponds only to Item 10's 9%. If rounding a p value counts as introducing false data, Item 5's 39% does not support a 1-in-10 estimate, so the pointer to Item 5 does not match the number given. Only the authors can say whether the pointer was meant to be Item 10 alone or whether the sentence should be reworded.
Second check: another model agreed
Report a problem with this finding
Wording Text says the items were "not judged to be more defensible by" experimental researchers, but Table 2 prints the Experimental defensibility rating as "0.66* (0.29)" (*p < .05).
- p. 6 Subgroup differences, paragraph on follow-up results
the items were particularly applicable to (but not judged to be more defensible by) researchers who conduct behavioral, experimental, and laboratory research.
- p. 7 Table 2, Research type: Experimental
0.66* (0.29)
- p. 6 Subgroup differences, follow-up results
(but not judged to be more defensible by) researchers who conduct behavioral, experimental, and laboratory research
Only the authors can say what is right here.
Judge's note: The p.7 Table 2 image shows Behavioral 0.63 and Laboratory 0.66 with no star, but Experimental 0.66 with one star, defined in the table note as p < .05 under random-effects ordered probit. The p.6 sentence names experimental research among the groups with no elevated defensibility. Either the text or the star is wrong as printed, and only the authors can say which.
Second check: another model agreed
Report a problem with this finding
5 minor (typos and the like; most checked by the judge only)
Wording Fig. 1 geometric-mean labels '78%' (Item 1) and '62%' (Item 7) do not follow from the plotted bars with the derived estimate capped at 100%, whereas the other eight labels do.
- p. 4 Fig. 1, Failing to Report All Dependent Measures and Excluding Data bars
78%
- p. 3 Prevalence estimates
they exceeded 100% by a small margin for a few items
Only the authors can say what is right here.
Judge's note: Bar heights were read from page 4. Item 1 is about 66.5, 61 and 100, giving a geometric mean near 74. Item 7 is about 43.4, 45 and 100, near 58. Items 2, 3, 4, 5, 6, 8, 9 and 10 reproduce their labels. The labels may come from uncapped derived values, but those would need to be about 117% and 122%, which is hard to reconcile with 'exceeded 100% by a small margin'. Only the authors' data can settle it. Products are recorded below; cube roots of 405650 and 195300 are about 74.0 and 58.0.
Second check: another model agreed
Report a problem with this finding
Wording Effects reported as "subfield (p < .05, ηp2 = .21)" and methodology "(p < .05, ηp2 = .27)" are described as "highly significant" predictors.
- p. 6 ANOVA paragraph
subfield (p < .05, ηp2 = .21) were highly signifi-
- p. 7 top of page
a highly significant predictor of self-admission rates (p < .05,
- p. 6 ANOVA paragraph
subfield (p < .05, ηp2 = .21) were highly signifi-cant predictors
- p. 7 methodology sentence
a highly significant predictor of self-admission rates (p < .05
Judge's note: P.6 and p.7 images: both effects are reported only at p < .05, yet they are called 'highly significant'. Table 2 uses ** and *** for p < .01 and p < .0005. The wording overstates the reported significance level. The effect sizes are moderate to large, so the authors may have meant the effects were strong. This is a wording point only.
Second check: another model agreed
Report a problem with this finding
Typo Stray closing parenthesis: "(M = 6.02 out of 25, SD = 3.70), F(1, 129) = 17.0, p < .0005). This"
- p. 5 Frequency of engagement, last paragraph
F(1, 129) = 17.0, p < .0005). This
- p. 5 Frequency of engagement
F(1, 129) = 17.0, p < .0005).
Judge's note: P.5 image: both parentheticals, (M = 3.77 ... SD = 2.27) and (M = 6.02 ... SD = 3.70), are closed before the F statistic, so the ')' after '.0005' has no opening partner. This is cosmetic only.
Second check: none (judge only)
Report a problem with this finding
Typo "64% (i.e., .153/ (.151 + .062 + .023))": the numerator .153 differs from the 'once or twice' term .151 in the denominator; 64% follows from .151.
- p. 6 Frequency of engagement, first paragraph
64% (i.e., .153/ (.151 + .062 + .023))
Judge's note: The p.6 image confirms that '.153/' is printed in the numerator and '.151' is the first term of the denominator. These are the same category. .151/.236 gives 64%, matching the printed value, while .153/.236 gives 64.8%. The 26% and 10% shares follow from the same denominator. This is a transcription typo with negligible effect.
Second check: none (judge only)
Report a problem with this finding
Typo Comma instead of semicolon between citations: 'Doyen, Klein, Pichon, & Cleeremans, 2012, Enserink, 1999;'.
- p. 8 Discussion, replication paragraph
Doyen, Klein, Pichon, & Cleeremans, 2012, Enserink, 1999
- p. 8 Discussion, replication paragraph
Doyen, Klein, Pichon, & Cleeremans, 2012, Enserink, 1999;
Judge's note: Page 8 image shows a comma, while every other separator in the list is a semicolon.
Second check: none (judge only)