The Natural Selection of Bad Science
Royal Society Open Science, 2016 · paper · Psychology
Argues, using a 60-year meta-analysis of statistical power and a model of competing labs, that incentives favoring publication select for poor methods. Power has not improved, high output drives up false discovery rates, and replication only slows this decline.
Checked: PDF from europepmc.org, fetched 5 Oct 2026. Page numbers are those of that PDF.
3 findings
Substance Section 5.3 prints "High effort labs had effort eH = 0.75, corresponding to a false-positive rate of α = 0.05 and an investigation rate of h = 0.625. Low effort labs had effort eL = 0.15, corresponding to a false-positive rate of α = 0.2 and an investigation rate of h = 0.765"; the stated rates require e = 75 and e = 15.
- p. 13 Section 5.3, second paragraph
High effort labs had effort eH = 0.75, corresponding to a false-positive rate of α = 0.05 and an investigation rate of h = 0.625
- p. 7 Section 4.1, after Eq. 4.1
ei is bounded between 1 and 100 for all labs
- p. 8 Figure 2 caption
e0 = 75 (shown in orange), such that α = 0.05 when power is 0.8
- p. 13 Section 5.3, second paragraph
Low effort labs had effort eL = 0.15, corresponding to a false-positive rate of α = 0.2 and an investigation rate of h = 0.765
Already corrected: the journal's 2023 correction gives eH = 75 and eL = 15 (R. Soc. Open Sci. 10: 231026, 6 Sep 2023, doi:10.1098/rsos.231026).
Judge's note: Viewed pages 13 and 7; the page 13 image prints eH = 0.75 and eL = 0.15. With power 0.8, alpha = W/(1+(1-W)e) gives 0.05 at e = 75 and 0.2 at e = 15, and h = 1 - 0.2*log10(e) gives 0.625 (log10 75 = 1.875) and 0.765 (log10 15 = 1.176). The printed values give alpha of about 0.70 and 0.78 and h above 1. Page 7 states 'ei is bounded between 1 and 100 for all labs', and the Figure 2 caption and Table 1 use e0 = 75 for alpha = 0.05 at power 0.8 (checked in the text layer, not the images of pages 8-9). No rescaling of effort is described for the simplified model, which is said only to omit evolution. Both values are off by a factor of 100.
Second check: another model agreed
Report a problem with this finding
Substance Table 1 lists the tested initial replication rates as "{0, 0.01, 0.2, 0.5}", but Figure 6 reports runs at "zero, 25% or 50% of all studies performed are replications" (legend: "25% replication"); 0.25 is not in the table and 0.2 is used nowhere else.
- p. 9 Table 1, row r0
{0, 0.01, 0.2, 0.5}
- p. 12 Figure 6 legend and caption
The evolution of effort when zero, 25% or 50% of all studies performed are replications.
- p. 12 Figure 6 caption
zero, 25% or 50% of all studies performed are replications
- p. 13 Figure 7 panel (c)
r = 0.05
Only the authors can say what is right here.
Judge's note: Page 9 image shows r0 values {0, 0.01, 0.2, 0.5}. Page 12 image shows the Figure 6 legend "no replication / 25% replication / 50% replication" and the matching caption. Section 4.3 (page 10) says "Table 1 describes all the model parameters and their default and range values", and Section 5.2 describes the Figure 6 runs as the replication rate "initialized to very high levels" with mu_r = 0, so these are r0 values. I searched the text for another explanation and found none. Either the table or the figure is wrong; only the authors can say which. The r = 0.05 in Figure 7 belongs to the separate non-evolutionary model and is not counted.
Second check: another model agreed
Report a problem with this finding
Wording Section 5.2 ends "institutional incentives for publication quality will still select for increasingly low-quality methods"; "quality" should read "quantity".
- p. 12 Section 5.2, last sentence
institutional incentives for publication quality will still select for increasingly low-quality methods
- p. 10 Section 5.1, last sentence
institutional incentives for publication quantity select for the continued degradation
- p. 12 Section 5.2, last sentence
for publication quality will still select for increasingly low-quality methods
Judge's note: Viewed pages 10, 12 and 13. The page 12 image prints 'publication quality'. The parallel sentence closing Section 5.1 (page 10) reads 'institutional incentives for publication quantity select for the continued degradation of scientific practices', and the Discussion (page 13) uses 'publication quantity' twice. The model rewards the number of publications and has no incentive for quality. A one-word slip that inverts the sentence, though the intended meaning is clear from context.
Second check: another model agreed
Report a problem with this finding
11 minor (typos and the like; most checked by the judge only)
Wording Abstract calls the power analysis "a 60-year meta-analysis" and Section 3 says "no sign of increase over six decades", but the Figure 1 caption gives the span as "between 1960 and 2011" (51 years) and the introduction says "over 50 years".
- p. 1 Abstract
a 60-year meta-analysis of statistical power
- p. 3 Introduction, top of page
despite over 50 years of reviews of low statistical power
- p. 6 Figure 1 caption
between 1960 and 2011
- p. 1 Abstract
60-year meta-analysis of statistical power
Judge's note: Checked pages 1, 3, 5 and 6. All four phrases are printed as quoted, and the Figure 1 points run from about 1960 to about 2011. "60-year" overstates a 51-year span; "six decades" is defensible as touching six calendar decades. The second strand (reviews "published between 1992 and 2014" against the caption's 2011) is not a defect: the caption dates the papers reviewed, the text dates the reviews, and a 2014 review can cover papers up to 2011.
Second check: another model agreed
Report a problem with this finding
Wording "newly hired biologists now have almost twice as many publications as they did 10 years ago (22 in 2013 versus 12.5 in 2005)": the figures quoted are 8 years apart, and the study period is given as 2005 to 2013.
- p. 3 Section 2, first paragraph
as they did 10 years ago (22 in 2013 versus 12.5 in 2005)
Judge's note: Page 3 image confirms the sentence and the preceding "between 2005 and 2013". The interval is 8 years, not 10; "10 years ago" is a loose rounding. "Almost twice" for 22/12.5 = 1.76 is acceptable.
Second check: another model agreed
Report a problem with this finding
Typo Second author is abbreviated "RME" in the ORCID line ("PES, 0000-0002-7133-5620; RME, 0000-0002-0387-5377") but "R.M." in the contributions statement ("P.E.S. and R.M. conceived and designed the research").
- p. 1 ORCID line under affiliations
RME, 0000-0002-0387-5377
- p. 15 Authors' contributions
P.E.S. and R.M. conceived and designed the research
Judge's note: Checked page 1 and page 15 images. Both forms are printed as quoted; the cite line on page 1 also gives "McElreath R.". The first author is PES / P.E.S. consistently. An inconsistency in initials only; either form could be a house abbreviation.
Second check: none (judge only)
Report a problem with this finding
Typo "not only in medicine, by also in psychology and other fields" should read "but also".
- p. 2 Introduction, second paragraph
not only in medicine, by also in psychology
- p. 2 Introduction, para 2
not only in medicine, by also in psychology and other
Judge's note: Page 2 image shows "by also" in the second paragraph of the Introduction. Not an extraction artefact.
Second check: none (judge only)
Report a problem with this finding
Typo "how scientific theories evolve by variation and selection retention" should read "variation and selective retention", as in the title of reference 19 cited in the same sentence.
- p. 2 Introduction, third paragraph
evolve by variation and selection retention
- p. 16 Reference 19
Variation and selective retention
Judge's note: Page 2 image shows "variation and selection retention". The reference 19 crop shows "Campbell DT. 1965 Variation and selective retention in socio-cultural evolution". The printed phrase is ungrammatical on either reading.
Second check: none (judge only)
Report a problem with this finding
Typo "some researchers place emphasis on metrics such the h-index" is missing "as" ("such as the h-index").
- p. 3 Section 2, last paragraph
metrics such the h-index
- p. 3 Section 2, fourth paragraph
such the h-index
Judge's note: Page 3 image, Section 2 fourth paragraph: the line break falls after "metrics" and the next line begins "such the h-index". The word is absent in print.
Second check: none (judge only)
Report a problem with this finding
Typo Footnote 3: "especially if hypothesis selection did not improved at a corresponding rate" should read "did not improve".
- p. 5 Footnote 3
especially if hypothesis selection did not improved at a corresponding rate
- p. 5 Footnote 3
if hypothesis selection did not improved at a corresponding rate
- p. 5 Footnote 3
did not improved at a corresponding rate
- p. 5 footnote 3
hypothesis selection did not improved at a corresponding rate
Judge's note: Page 5 image, footnote 3, shows "did not improved".
Second check: none (judge only)
Report a problem with this finding
Typo Power is written with capital W in the text and equations ("Each lab i has a characteristic power, Wi"; Eq. 4.2; Eq. 5.1) but lower-case w in Table 1 ("w0 initial power for all labs", "probability of w mutation", "standard deviation of w mutation magnitude").
- p. 8 Section 4.1
Each lab i has a characteristic power, Wi
- p. 9 Table 1
probability of w mutation
Judge's note: Page 8 image shows W_i in the text and Eq. 4.2; page 10 shows W in Eq. 5.1; page 9 Table 1 shows w0 and "w mutation". Effort (e) and replication rate (r) use one letter throughout. A notational inconsistency with no risk of misreading the quantity.
Second check: none (judge only)
Report a problem with this finding
Typo Section 5.1 repeats a sentence: "insofar as at least some aspects of power are directly measurable. At least some aspects of experimental power, such as statistical power, are directly measurable."
- p. 10 Section 5.1, third paragraph
insofar as at least some aspects of power are directly measurable. At least some aspects of experimental power, such as statistical power, are directly measurable.
Judge's note: Viewed page 10. The two consecutive sentences make the same statement in near-identical words; the second only adds 'such as statistical power'. It reads as an editing leftover.
Second check: none (judge only)
Report a problem with this finding
Typo Reference 39 prints the first author as "Vinkers V, Tijdink J, Otte W. 2015"; the initial is believed to be C (Christiaan H. Vinkers).
- p. 16 Reference 39
Vinkers V, Tijdink J, Otte W. 2015
Only the authors can say what is right here.
Judge's note: Rests on outside knowledge: the first author's given name on the cited BMJ paper (351, h6467) is Christiaan H. Vinkers. Viewed page 16: the entry prints 'Vinkers V'. A wrong initial only; the reference is still identifiable. Verify against doi:10.1136/bmj.h6467.
Second check: none (judge only)
Report a problem with this finding
Typo Reference 93 title prints "Editors’ JIF-boosting strategems"; the standard spelling, and believed to be the published title, is "stratagems".
- p. 17 Reference 93
Editors’ JIF-boosting strategems
Only the authors can say what is right here.
Judge's note: Rests on outside knowledge: the published title of Martin BR 2016, Res. Policy 45, 1-7 uses 'stratagems'. Viewed page 17: the entry prints 'strategems'. The standard spelling is 'stratagems', though 'strategem' occurs as a variant, so check against doi:10.1016/j.respol.2015.09.001.
Second check: none (judge only)