Many Labs 2: Investigating Variation in Replicability across Samples and Settings

• p. 35 Table 4, Affect and risk

  −0.08 [−0.13, −0.02]

• p. 28 Table 2, Affect and risk

  −0.08 [−0.13, −0.03]

• p. 15 Effect 5 text

  d = −0.08, 95% CI = [−0.13, −0.03]

Judge's note: I checked all three crops. Table 4's note says its Global column is the overall effect size ignoring position, so it is the same quantity as Table 2 and the text. Two of the three places give −0.03, so Table 4 is probably a rounding or transcription slip. The difference is small.

Second check: another model agreed

• p. 33 Figure 3

  Histogram of the WEIRDness (Western, educated, industrialized, rich, democratic) scores of the samples.

• p. 30 Results, WEIRDness classification

  (Slate 1: n = 42; Slate 2: n = 44)

• p. 41 Discussion

  Figure 3 illustrates the highly skewed distribution.

Only the authors can say what is right here.

Judge's note: Read from the p. 33 image: 8, 15, 1, 2, 12, 1, 0, 1, 3, 17, 70, 14. The upper three bars total about 101 against 86 WEIRD samples. Page 7 states 125 total samples, with 64 for Slate 1 and 61 for Slate 2 already counted separately, so slate double-counting does not explain the gap. No other unit of count is stated.

Second check: another model agreed

• p. 46 References, Dreber et al.

  (2016). Using predic-tion markets to estimate the reproducibility of scientific research. Proceedings of the National Academy of Sci-ences, USA, 112, 15343–15347.

• p. 36 Discussion

  Dreber et al., 2016

Only the authors can say what is right here.

Judge's note: Rests on outside knowledge: PNAS volume 112 / the Dreber et al. prediction-markets article appeared in 2015. The reference list itself supports this: it dates PNAS 113 to 2016 (Van Bavel et al.) and PNAS 115 to 2018 (Nosek et al.), which makes volume 112 a 2015 volume. The DOI suffix also ends in 112. The crop of p.46 shows the entry with (2016) and volume 112. The text cites "Dreber et al., 2016" on p.36 (crop C061-b) and again at text-layer line 4837. The volume and the year disagree, but only the authors can say which one is right, so this is a query.

Second check: another model agreed

• p. 46 References, Crisp et al. (2014)

  45, 303–304. doi:10.1027/1864-9335/a000202

• p. 46 References, Ferguson et al. (2014)

  Psychology, 45, 301–302. doi:10.1027/1864-9335/a000202

• p. 49 References, Schwarz & Strack (2014)

  305–306. doi:10.1027/1864-9335/a000202

• p. 46 References

  Crisp, R. J., Miles, E., & Husnu, S. (2014). Support for the rep-licability of imagined contact effects. Social Psychology, 45, 303–304. doi:10.1027/1864-9335/a000202

Only the authors can say what is right here.

Judge's note: Rests on outside knowledge: whether Social Psychology published these commentaries under one shared DOI or under separate ones. The page images confirm that all three entries print a000202 with different titles and page ranges. If the journal used a single DOI for a combined commentaries item, the entries would be legitimate; otherwise at least two DOIs are wrong. The authors or the publisher would need to confirm.

Second check: another model agreed

• p. 28 Table 2, Construing actions as choices, WEIRD samples

  0.08 [−0.33, 0.50]

• p. 21 Effect 19 replication discussion

  from the original null effect (d = 0.04) among Ameri

• p. 36 Discussion, para 1

  Seven (25%) of the replications had effect sizes (Cohen’s d or q) larger than the original, and 21 (75%) had effect sizes smaller

• p. 5 Abstract

  Seven (25%) of the replications yielded effect sizes larger than the original ones

Only the authors can say what is right here.

Judge's note: The text on p. 20 gives β = −0.13, OR = 0.88, d = 0.08 across cultures and d = 0.04 among Americans; Table 2 uses 0.08 for the WEIRD row. From Table 2, only six replications exceed the original in the same direction: Knobe, Ross supermarket, Gray, Ross traffic, Norenzayan WEIRD and Hsee. Savani is a seventh only on absolute magnitude. The wording is ambiguous rather than plainly wrong, so this goes to the authors as a query.

Second check: another model agreed

• p. 14 Effect 3 original study

  t(39) = 2.01, p = .051, d = 0.63

• p. 14 Effect 3 replication summary

  vs. original d = 0.64

• p. 14 Effect 3, original study

  d = 0.63, 95% CI = [−0.004, 1.25]

• p. 28 Table 2

  0.63 [−0.004, 1.25]

Judge's note: p.14 and Table 2 (p.28) images checked. One quantity is printed two ways. A plain 2t/sqrt(df) approximation gives 0.644, but the paper's own calculation (Table 2 and the original-study paragraph) prints 0.63. The difference is small.

Second check: another model agreed

• p. 14 Effect 3 replication, English in-lab subsample

  t(2578) = −0.79, p = .43, d = −0.03, 95% CI = [−0.08, 0.01]

• p. 14 Effect 3 follow-up, aggregate

  t(6933) = −1.37, p = .17, d = −0.03, 95% CI = [−0.08, 0.01]

Only the authors can say what is right here.

Judge's note: p.14 image checked. For N = 2,580 the approximate 95% half-width of d is about 0.077, against a printed half-width of 0.045. The printed interval fits N = 6,935 (half-width about 0.047) and matches the Table 2 global effect [−0.08, 0.01], which Table 5 ties to N = 6,935. The secondary analysis on the same subsample has a half-width of about 0.075 ([−0.12, 0.03]), consistent with N = 2,580. Authors should confirm which interval belongs to the focal subsample test.

Second check: another model agreed

• p. 16 Effect 8 replication

  z = 2.62, p = .02, q = 0.05

• p. 16 Effect 8, replication

  z = 2.62, p = .02, q = 0.05, 95% CI = [0.01, 0.10]

• p. 38 Table 5, Inbar row

  .024

Only the authors can say what is right here.

Judge's note: p.16 image checked. A two-sided normal p for z = 2.62 is about .009. From r = .12 and r = .07 with N = 7,117, q/SE is about 2.1–2.3, which fits p ≈ .02 and Table 5's .024. The follow-up on p.17 pairs the same q and CI with z = 2.28, p = .02. Both readings fall below .05.

Second check: another model agreed

• p. 16 Effect 7 original study

  t(23) = 1.84, p = .069

Only the authors can say what is right here.

Judge's note: p.16 image checked. The critical t for df 23 is 1.714 at two-sided .10 and 2.069 at .05. Log-interpolating, t = 1.84 gives a two-sided p of about .078–.079 (one-sided about .04), not .069. Both readings are above .05, and this is an original-study value.

Second check: another model agreed

• p. 17 Effect 9 original study

  Gilovich’s (2008) Study 2, 207 participants

• p. 38 Table 5, Influence of incidental anchors, original sample size

  200

• p. 17 Effect 9

  207 participants predicted the relative popularity

Only the authors can say what is right here.

Judge's note: p.17 and Table 5 (p.38) images checked. No note explains the difference, unlike Graham's footnote 4. The Welch df of 197.5 is compatible with either an analysed subset or the full sample. Table 5 computes its original-sample-size criterion from this N.

Second check: another model agreed

• p. 17 Effect 10 original study

  In Study 3, 631 participants reported

• p. 38 Table 5, Social value orientation, original sample size

  536

• p. 17 Effect 10

  631 participants reported how many

Only the authors can say what is right here.

Judge's note: p.17 and Table 5 (p.38) images checked. Three different Ns appear for the same original analysis, and no exclusion or explanatory note was found in the text.

Second check: another model agreed

• p. 18 Effect 11 follow-up

  Among the 3,069 participants reporting that they

• p. 18 Effect 11 replication

  6,842 after removing participants who responded in less

• p. 18 Effect 11

  N = 6,842 after removing

• p. 18 Effect 11 replication and follow-up

  N = 6,842 after removing participants who responded in less than 4 s

Only the authors can say what is right here.

Judge's note: Page 18 image checked: the replication N is 6,842 after the <4-s exclusion, and the follow-up gives 3,069 and 4,107. Their sum (7,176) is larger than the analysed sample. I searched the text for 'familiar' and '4 s' and found no statement that the follow-up used a different exclusion rule or analysis set. For Effect 17 the subgroups sum to 7,855, which is below N = 7,923, so missing familiarity responses could explain that case but not this one. Only the authors can say which N applies.

Second check: another model agreed

• p. 19 Effect 16 original study

  z = 5.14, p = 7.4e−7

Only the authors can say what is right here.

Judge's note: Page 19 image confirms 'z = 5.14, p = 7.4e−7'. Under a standard normal reference, two-sided p for 5.14 is about 2.7e−7 and one-sided about 1.4e−7; p = 7.4e−7 would need z of about 4.95. No continuity correction or other test is stated. The value describes the original study and is not in the abstract.

Second check: another model agreed

• p. 20 Effect 19 original study

  t(88) = 0.39, p = .65, d = 0.04

Only the authors can say what is right here.

Judge's note: Page 20 image confirms the values. On 88 df, t = 0.39 gives a two-sided p of about .70. The printed .65 is close to 1 minus the one-sided p (about .65), but no directional test is declared. The value is an original-study figure and is not in the abstract.

Second check: another model agreed

• p. 21 Effect 19 replication, India subsample

  3.79, p = 1.0e−4, d = 0.19

• p. 21 Effect 19 replication

  β = 0.35, SE = 0.09, z = 3.79, p = 1.0e−4

Only the authors can say what is right here.

Judge's note: Page 21 image confirms the values. For z = 3.79 the two-sided normal p is about 1.5e−4. p = 1.0e−4 fits z of about 3.89, which is β/SE computed from the rounded values. Because β and SE are rounded, the printed z may be the accurate one. Both readings are far below .05, and the result is not in the abstract.

Second check: another model agreed

• p. 24 Effect 26 original, belief vs control

  t(193) = 2.23, p = .03, d = 0.37

• p. 25 Effect 26 original, concern vs control

  t(193) = 2.23, p = .02, d = 0.31

Only the authors can say what is right here.

Judge's note: Page 24 and 25 images confirm both strings. t(193) = 2.23 gives a two-sided p of about .027, which rounds to .03, not .02. With roughly equal groups, d ≈ 2t/sqrt(df) = 0.32, which matches the concern d = 0.31 but not the belief d = 0.37 (whose CI [0.14, 0.59] is centred near 0.37). One of the two t values may have been copied from the other. Only the authors or the original article can say which values are correct.

Second check: another model agreed

• p. 39 Table 5, Effect of choosing versus rejecting

  5.47e−10

• p. 24 Effect 25 replication

  z = −6.10, p = 1.1e−9

Only the authors can say what is right here.

Judge's note: Table 5 footnote a says its p values were calculated from the observed global effect size, so the table could in principle use a different computation. Elsewhere, though, the table repeats the text p values. A two-sided p for |z| = 6.10 is about 1.06e−9. The table value is about half that, which is what a one-sided test would give, but neither place says so. The authors should say which test and sidedness lie behind each number. Both values are below .0001, so the classification is unaffected.

Second check: another model agreed

• p. 25 Effect 26 replication, English-only subsample

  t(2937) = −0.18, p = .24, d = −0.04

• p. 25 Effect 26 replication

  t(2937) = −0.18, p = .24, d = −0.04, 95% CI = [−0.12, 0.03]

Only the authors can say what is right here.

Judge's note: Page 25 image confirms the values. On 2,937 df, |t| = 0.18 gives p ≈ .86, while p = .24 needs |t| ≈ 1.18. For N = 2,939, d ≈ 2t/sqrt(df): t = −0.18 gives −0.007, but t = −1.18 gives −0.044, which matches the printed d = −0.04. A dropped digit is likely, but only the authors can confirm it. The result is not in the abstract.

Second check: another model agreed

• p. 25 Effect 28 original

  In their Study 2, 144 participants

• p. 26 Effect 28 original results

  t(153) = 2.99

Only the authors can say what is right here.

Judge's note: Pages 25–26 images and the Table 5 text row (original N = 144) checked. The design is described as a between-participants manipulation, and with 144 participants the df cannot exceed 142 (Welch df is no larger). The paper names no other unit of analysis. The original article would settle which number is right.

Second check: another model agreed

• p. 26 Effect 28 original study

  2.92, p = .001, d = 0.64

• p. 26 Effect 28 original

  t(20) = 2.92, p = .001, d = 0.64

Only the authors can say what is right here.

Judge's note: Page 26 image confirms the values. On 20 df, t = 2.92 gives a two-sided p of about .0085 and a one-sided p of about .004, so neither is .001. The similar 't(20) = 2.72' in the next paragraph is reported as p < .01. This is an original-study value and is not in the abstract.

Second check: another model agreed

• p. 26 Effect 28 original

  t(20) = 2.92, p = .001

Only the authors can say what is right here.

Judge's note: Checked the crop and p. 26. Under a t distribution with 20 df, t = 2.92 lies between the two-sided .01 critical value (2.845) and the .005 critical value (3.153), so two-sided p is about .008 and one-sided p about .004. Neither rounds to .001. This is the paper's report of the original study's statistics, so the authors should check which value was meant.

Second check: another model agreed

• p. 28 Results text after Table 2

  for the other 6, 11% to 46% of the individual samples had significant results

• p. 27 Table 2, Cardinal direction WEIRD samples

  65.22

• p. 28 Results, first paragraph after Table 2

  for the other 6, 11% to 46% of the individual samples

Only the authors can say what is right here.

Judge's note: Table 2 (pp. 27–28) shows eight supported effects at 89–100%. The remaining six are framing 45.45, moral foundations 25.00, Trolley 2 18.33, tempt fate 25.42, consumerism 11.11 and cardinal direction (65.22 WEIRD, 11.11 less WEIRD). The stated range fits only if Huang is represented by its less-WEIRD value, and the paper does not say so.

Second check: another model agreed

• p. 28 Table 2, Effect of choosing versus rejecting

  18.97

• p. 32 Table 3, Shafir row

  51.67

Only the authors can say what is right here.

Judge's note: Checked Table 2 (p. 28) and Table 3 (p. 32 upright). The percentages are exactly 11/58, 46/58 and 1/58. For most other effects the Table 3 df plus 1 matches the Table 2 denominator: framing 25/55 with df 54, Alter 1/66 with df 65, Anderson 3/59 with df 58. Zaval also differs (5.36% = 3/56 against df 46). I found no text explaining why samples would be dropped from the meta-analysis for this effect.

Second check: another model agreed

• p. 30 Results, WEIRDness scores

  countries with WEIRDness scores above the mean across

• p. 41 Discussion

  Countries with scores above 0.70 were categorized as WEIRD

• p. 30 Results, WEIRDness score

  WEIRDness scores above the mean across samples were categorized as WEIRD

Only the authors can say what is right here.

Judge's note: Checked the images of p. 30 (right column) and p. 41 (left column). The paper does not report the mean WEIRDness score, so a reader cannot tell whether the two rules are the same.

Second check: another model agreed

• p. 32 Table 3, Moral violations and desire for cleansing, Lab versus online

  9.15

• p. 32 Table 3 (Continued), Zhong & Liljenquist, lab versus online

  9.15 < .001

Only the authors can say what is right here.

Judge's note: The note on the p. 32 upright image says df for Q is 1. χ²(1) = 9.15 corresponds to z ≈ 3.02, giving p ≈ .0025, which is not < .001. It is still below the stated Bonferroni alpha of .003, so whether the result counts as significant does not change. The other moderation pairs check out (6.19 → .01; 13.01 → < .001).

Second check: another model agreed

• p. 10 Participants

  data collections was

• p. 10 Participants, para 2

  data collections was entirely Web based

Judge's note: Confirmed on the p.10 crop.

Second check: none (judge only)

• p. 14 Effect 3 replication, secondary analysis

  95% CI = [−0.12, 0.03]).

Judge's note: p.14 image: every earlier parenthesis in the sentence, e.g. (M = 0.84, SD = 0.81), is already closed, so the ')' after the CI is stray.

Second check: none (judge only)

• p. 16 Effect 8 original

  describing more generally as encouraging kissing

• p. 16 Effect 8, original study description

  than when he was describing more generally as encouraging kissing

• p. 16

  when he was describing more generally as encouraging kissing

Judge's note: p.16 image checked. The director is the one being described, so the passive is needed.

Second check: none (judge only)

• p. 17 Effect 10 original

  d = 0.19, 95% CI = [< 0.01, 0.47]

• p. 28 Table 2, Social value orientation

  0.19 [< 0.001, 0.47]

Judge's note: p.17 and p.28 images checked. Both are upper-bound statements about the lower limit and do not contradict each other (a value below .001 is also below .01), but the same bound is printed inconsistently.

Second check: none (judge only)

• p. 19 Effect 15

  In Giessner and Shubert’s (2007) Study

• p. 19 Effect 15 text

  Giessner and Shubert’s (2007) Study

• p. 47 Reference list

  Giessner, S. R., & Schubert, T. W. (2007)

Judge's note: Page 19 image shows 'Shubert's' in the body text, while the heading on the same page reads 'Giessner & Schubert, 2007'. The reference list on p. 47 reads 'Schubert, T. W.'.

Second check: none (judge only)

• p. 22 Effect 20 replication

  (d = 0.67, 95% CI = [0.52, 0.81).

• p. 22 Effect 20 Replication, first paragraph

  d = 0.67, 95% CI = [0.52, 0.81).

• p. 22

  0.67, 95% CI = [0.52, 0.81). Not

Judge's note: Page 22 image (left column, below Fig. 1) shows the interval opening with '[' and closing with ')'. Everywhere else the paper uses square brackets for CIs.

Second check: code check

• p. 23 Effect 22 follow-up

  the adult man (M = 4.66 SD = 1.25)

Judge's note: Page 23 image, top of the right column, shows no comma between M and SD. The adjacent pair '(M = 5.44, SD = 1.25)' has one.

Second check: none (judge only)

• p. 27 Table 2, effect name

  intuitive Reasoning (Norenzayan, Smith, Kim,

Judge's note: The p. 27 image shows the capital R at the start of the second line. In the sentence-case table names elsewhere (Table 1, Table 4, the text, the appendix) it is lower case. The figures use title case throughout, so they are consistent with themselves.

Second check: none (judge only)

• p. 29 Text below Fig. 2

  is available in at https://osf.io/5yzn8/

Judge's note: Seen on the p. 29 image, in the last line of the right column.

Second check: none (judge only)

• p. 43 Table A1, effect 21

  Hsee (1998. Study 1)

Judge's note: The page image shows '(1998. Study 1)'. Every other row uses the form '(year, Study n)'.

Second check: none (judge only)

• p. 44 Author Contributions

  L. A.Vaughn

Judge's note: The page image shows 'L. A.Vaughn' in the 'designed the study' list. Later in the same section it is printed 'L. A. Vaughn'.

Second check: none (judge only)

• p. 46 References, Coppock (in press)

  Political Science Research Methods.

Only the authors can say what is right here.

Judge's note: Rests on outside knowledge: the journal is titled 'Political Science Research and Methods'. The crop of p.46 shows "Political Science Research Methods." in italics. Nothing in the files settles the journal's title, so this is a query to the authors.

Second check: none (judge only)