How Much Should We Trust Differences-in-Differences Estimates?
2004 · paper · Economics
Examines standard-error bias in differences-in-differences studies caused by serial correlation. Using placebo laws in state wage data, it finds significant false effects for up to 45% of laws, and proposes several corrective techniques.
Checked: PDF from nber.org, fetched 5 Oct 2026. Page numbers are those of that PDF.
11 findings
Substance Footnote 18: "Thus while OLS overestimates standard errors, the estimated coefficients are unbiased." The paper's argument is that OLS under-states standard errors.
- p. 10 Footnote 18
Thus while OLS overestimates standard errors, the estimated coefficients are unbiased.
- p. 10 Footnote 18
Thus while OLS overestimates standard errors
- p. 6 Section 2.1
positive serial correlation in the error term (ρ > 0) will cause an under-statement of the standard error
Judge's note: The page-10 image confirms the wording. Page 6 says positive serial correlation "will cause an under-statement of the standard error", and page 13 says OLS standard errors "grossly understate the true standard errors". A 67.5% over-rejection rate is consistent only with understated standard errors.
Second check: another model agreed
Report a problem with this finding
Substance "When 12 states or 36 states are affected, the rejection rates are 39% and 43% respectively." Table 2 gives .433 for 12 states and .398 for 36 states, so the two values are swapped.
- p. 12 Section 3.2, para on rows 5 and 6
When 12 states or 36 states are affected, the rejection rates are 39% and 43% respectively.
- p. 28 Table 2, rows 5-6
12 states, one date
- p. 28 Table 2, rows 5-6
12 states, one date OLS .433
- p. 12
the rejection rates are 39% and 43% respectively
Judge's note: Table 2 image (page 28): the "12 states, one date" row shows .433 and the "36 states, one date" row shows .398. Because of "respectively", the text assigns the values in reverse order.
Second check: another model agreed
Report a problem with this finding
Substance Row 7 (staggered dates) is described as "the null of no effect being rejected about 35% of the time", but Table 2 row 7 ("25 states, multiple dates", OLS) shows .48 (.029).
- p. 12 Section 3.2, row 7 discussion
with the null of no effect being rejected about 35% of the time
- p. 28 Table 2, row 7 (25 states, multiple dates)
.48
- p. 28 Table 2, row 7
25 states, multiple dates
- p. 12 Section 3.2, row 7
the null of no effect being rejected about 35% of the time
Judge's note: Page 28 image: row 7 reads .48, with a 2% effect value of .71. No other row in Table 2 matches 35%. The gap is far larger than rounding.
Second check: another model agreed
Report a problem with this finding
Substance Table 4, 'CPS agg, Employment' 2% effect rate .698 is printed with standard error '(.010)'.
- p. 30 Table 4, CPS agg Employment, 2% Effect
(.010)
- p. 30 Table 4, CPS agg Employment row
.698 (.010)
- p. 30 Table 4, CPS agg Employment
.415 .698 (.028) (.010)
- p. 30 Table 4, CPS agg Employment row
(.028) (.010)
Only the authors can say what is right here.
Judge's note: Page 30 image checked. Note 4 says the SEs are computed from the number of simulations, and note 1 says at least 200. A rate of .698 then gives an SE of about .026–.032. Neighbouring cells match this (.700 gives .026; .72 gives .026). Getting .010 would take about 2100 draws. The printed SE does not fit the stated rule.
Second check: another model agreed
Report a problem with this finding
Reference or label Footnote 1 cites "Meyer (1994)", but the only Meyer entry in the reference list is "Journal of Business and Economic Statistics, v. 13 (1995): 151-162."
- p. 3 Footnote 1
See Meyer (1994) for an overview.
- p. 26 References
Economic Statistics, v. 13 (1995): 151-162.
- p. 26 References
v. 13 (1995): 151-162
- p. 26 References
v. 13 (1995): 151-162.
Judge's note: Page 3, footnote 1, reads "See Meyer (1994) for an overview." The reference list on page 26 has a single Meyer entry, dated 1995. The years are inconsistent within the paper; which one is right cannot be settled from the files.
Second check: another model agreed
Report a problem with this finding
Reference or label "Besley and Case (1994)" (footnotes 2 and 3) and "Blanchard and Katz (1992)" (Section 2.2) are cited but missing from the reference list. MaCurdy (1982) and Rosenbaum (1993) are listed but never cited.
- p. 3 Footnote 2
See Besley and Case (1994).
- p. 7 Section 2.2
Blanchard and Katz (1992)
- p. 26 References
MaCurdy, Thomas E., “The Use of Time Series Processes
Judge's note: Page 26 image: the reference list has no Besley/Case or Blanchard/Katz entry. Searching the text finds MaCurdy only as part of "Blundell and MaCurdy (1999)". Rosenbaum is cited only as 1996, so the 1982 and 1993 entries are uncited.
Second check: another model agreed
Report a problem with this finding
Reference or label "Rows 5-9 and 14-18 vary the number of years." Table 5 has only 17 rows; the manufactured T-varying rows are 14-17.
- p. 15 Section 3.5
Rows 5-9 and 14-18 vary the number of years.
- p. 31 Table 5 panel B
B. MANUFACTURED DATA
- p. 31 Table 5
AR(1), ρ=.8
Judge's note: Table 5 image (page 31): 9 real-data rows and 8 manufactured rows. Rows 10-13 vary N and rows 14-17 vary T. There is no row 18.
Second check: another model agreed
Report a problem with this finding
Reference or label "29% in the manufactured data as well (row 3)" and "In row 4, when we allow for 400 states" do not match Table 7: the .285 result is in row 4 and the 400-state block bootstrap (.075) is in row 6.
- p. 17 Section 4.2
29% in the manufactured data as well (row 3). The problem appears to be the small number of blocks or states. In row 4, when we allow for 400 states
- p. 33 Table 7
Block Bootstrap 400
- p. 17 Section 4.2
35% in the CPS data (row 2) 29% in the manufactured data as well (row 3)
- p. 33 Table 7
Block Bootstrap 50 .285 .645
Judge's note: Table 7 image (page 33): row 1 is CPS OLS .435, row 2 CPS BB .35, row 3 manufactured OLS .38, row 4 manufactured BB .285, row 5 OLS N=400 .415, row 6 BB N=400 .075. The text's row 2 citation counts every line, so rows 3 and 4 are off.
Second check: another model agreed
Report a problem with this finding
Wording Section 4.4 describes the regressors in the sandwich formula as 'x_jt is a row vector of dependent variables (including the constant)', but they are independent variables.
- p. 18 Section 4.4
xjt is a row vector of dependent variables (including the constant)
- p. 18
xjt is a row vector of dependent variables
Judge's note: Page 18 (printed p.17) image checked. Two lines earlier the same passage defines 'X is matrix of independent variables'. x_jt is a row of X and includes the constant, so 'dependent' is a wrong word.
Second check: another model agreed
Report a problem with this finding
Wording Section 5 says 'We concentrate on 3 data sets: micro CPS wage data ..., aggregate CPS wage data, and manufactured data', but Table 12 reports only 'CPS Aggregate' and 'AR(1), ρ = .8'. The text's '(.105 and .150)' for 'the aggregate data' does not match Table 12's .106, and .150 is the AR(1) manufactured column.
- p. 24 Section 5
We concentrate on 3 data sets: micro CPS wage data
- p. 24 Section 5, last test
especially in the aggregate data (.105 and .150)
- p. 39 Table 12 header
CPS Aggregate
- p. 24 Section 5
We concentrate on 3 data sets: micro CPS wage data (with clustering of the error term by state-year cell), aggregate CPS wage data, and manufactured data
Only the authors can say what is right here.
Judge's note: Pages 24 and 39 images checked. Table 12 has only two data columns and no micro CPS results. The TREND row shows .106 (CPS Aggregate, no effect) and .150 (AR(1) no effect). The text prints .105, and its 'aggregate data' label covers the manufactured value as well. Only the authors can say whether the micro results were dropped and which value is right.
Second check: another model agreed
Report a problem with this finding
Wording Table 6 Panel B, 'AR(1) correction' imposing ρ=.8: '.06 (.023)' with '.323' and no SE beneath it.
- p. 32 Table 6, Panel B, AR(1) correction imposing rho=.8
.06 .323
Only the authors can say what is right here.
Judge's note: Page 32 image checked. The 2% effect cell .323 has no standard error, unlike every other cell. The single SE (.023) sits under .06, where the simulation-count rule (at least 200) gives about .017 or less. It may belong to the other column. Only the authors can say.
Second check: another model agreed
Report a problem with this finding
23 minor (typos and the like; most checked by the judge only)
Reference or label "As we saw in equation 2.1" points to an equation that does not exist; Section 2.1 (equations 2-3) is presumably meant. Also redundant "at the 5% percent level" on page 4.
- p. 11 Section 3.2
As we saw in equation 2.1
- p. 4 Section 1
at the 5% percent level
Judge's note: The equations are numbered (1)-(5) (pages 3, 10 and 11); 2.1 is a section number. The page-4 crop shows "5% percent", which is cosmetic.
Second check: another model agreed
Report a problem with this finding
Wording The text says the effect is significant "in as much as 45% of the simulations" (abstract: "up to 45%"), yet Table 2 reports .48 for CPS aggregate data with multiple dates.
- p. 4 Introduction
we find a significant effect at the 5% level in as much as 45% of the simulations
- p. 28 Table 2 row 1
.675
Only the authors can say what is right here.
Judge's note: The 67.5% micro OLS figure is reasonably excluded, because footnote 5 assumes grouped errors have been accounted for. The .48 entry is for aggregate state-level data, however, and exceeds the stated ceiling. The text describes that same row as "about 35%". Authors should clarify which specification the 45% refers to.
Second check: another model agreed
Report a problem with this finding
Wording Section 3.5, "focused data with 51 states and 21 time periods", and Table 9 ("# States 51") conflict with "50 times 21 state-year cells" (page 11) and N=50 in Tables 5 and 7 for the same .435/.72 baseline.
- p. 14 Section 3.5
The stylized exercise above focused data with 51 states and 21 time periods.
- p. 11 Section 3.2
This leaves us with 50 times 21 state-year cells
- p. 35 Table 9
CPS agg OLS 51
- p. 8 Section 3
(50*21=1050) state-year cells
Only the authors can say what is right here.
Judge's note: Page 35 image: Table 9 shows 51 for CPS agg OLS .435/.72. Table 5 (page 31) and Table 7 (page 33) show 50 for the identical row. Page 11 says 50 times 21 cells, and page 17 refers to 50 states. The paper never mentions DC, so the discrepancy is unexplained.
Second check: another model agreed
Report a problem with this finding
Wording "the false rejection problem diminishes with the serial correlation in the dependent variable" is ambiguous. Table 4 shows false rejection falling as serial correlation falls, i.e. rising with it.
- p. 14 Section 3.4
the false rejection problem diminishes with the serial correlation in the dependent variable
Judge's note: Table 4 (page 30): log wage has rho1 .509 and rejection .435; hours worked has .151 and .263; changes in log wage has -.046 and 0. In the manufactured data, rejection rises with rho. The phrase can be read as "diminishes along with", but it can also be read the wrong way.
Second check: another model agreed
Report a problem with this finding
Wording The text says that with ρ=.8 imposed "the rejection rate goes down to 12.5%", but Table 6 row 3 prints .12 (.019).
- p. 15 Section 4.1
the rejection rate goes down to 12.5%
- p. 32 Table 6, row 3
.12
- p. 32 Table 6 row 3
imposing ρ=.8 (.019) .12
Judge's note: Page 32 image: "AR(1) correction imposing ρ=.8" shows .12. The difference is small and may be rounding of .125 in the table, but the text and the table print different values for the same cell.
Second check: another model agreed
Report a problem with this finding
Wording The text says "the variance of the white noise is 13 % of the total variance of the residual", while Table 6 gives "noise/signal=.13". A noise-to-signal ratio of .13 corresponds to about 11.5% of total variance.
- p. 16 Section 4.1
the variance of the white noise is 13 % of the total variance of the residual
- p. 32 Table 6 note 2 / row
ρ = .95, noise/signal=.13
Judge's note: Page 16 text and the Table 6 rows and note 2 (page 32) define n/s as the relative variance of the components. The two descriptions are not equivalent; the discrepancy is minor.
Second check: another model agreed
Report a problem with this finding
Wording Manufactured AR(1) ρ=.8 OLS shows '.373 (.028)' with a 2% effect of '.725 (.026)' in Tables 2 and 4, but '.373 (.028)' with '.765 (.024)' in Table 6.
- p. 32 Table 6, manufactured OLS
.765
- p. 28 Table 2 row 8
.725
- p. 31 Table 5 panel B row 1
.638
- p. 32 Table 6, manufactured OLS
.373 .765
Only the authors can say what is right here.
Judge's note: Pages 28, 30, 32 and 31 images checked. The no-effect cell and its SE are identical across Tables 2, 4 and 6, which suggests the same run, yet the 2% effect cell differs (.725 vs .765). Table 5 row 1 for the same design shows .35/.638, so separate runs per table do occur, and the paper does not say which numbers come from which run. Only the authors can resolve this.
Second check: another model agreed
Report a problem with this finding
Typo Abstract: "the bias in the estimated standard errors that serial correlation introduce4s" has a stray digit; it should read "introduces".
- p. 2 Abstract
serial correlation introduce4s
Judge's note: The page-2 image shows "introduce4s" exactly as quoted, so this is in the published text and not an extraction artefact.
Second check: none (judge only)
Report a problem with this finding
Typo Copy-editing slips: "the 5% percent level" (p.3), "procedures are use to correct" (p.5), "follow and AR(1) process" (fn 13).
- p. 4 Intro
the 5% percent level
- p. 6 Section 2.2
whether any procedures are use to correct
- p. 9 Footnote 13
follow and AR(1) process
- p. 4 Introduction, paragraph 3
at the 5% percent level
Judge's note: All three slips are visible in the page images.
Second check: none (judge only)
Report a problem with this finding
Typo "Section 5, discusses implications" has a stray comma between subject and verb.
- p. 5 Roadmap paragraph
Section 5, discusses implications
Judge's note: Visible on page 5.
Second check: none (judge only)
Report a problem with this finding
Typo Footnote 8 reads "When a used several data sets"; the word "paper" is missing.
- p. 7 Footnote 8
When a used several data sets
Judge's note: Visible on page 7.
Second check: none (judge only)
Report a problem with this finding
Typo Equation 4 is printed as "Y_ist = A_s + B_t + c X_it + β T_st + ν_st + ε_it"; the X and ε subscripts drop s, unlike equation 1 (X_ist, ε_ist).
- p. 10 Equation 4
Yist = As + Bt + c Xit + β Tst + νst + ²it
Judge's note: The subscripts are confirmed in the equation image. This is a notational inconsistency only.
Second check: none (judge only)
Report a problem with this finding
Typo "has been described by Donald and Lang and (2001)" has an extra "and"; it should read "Donald and Lang (2001)".
- p. 10 Section 3.2
Donald and Lang and
- p. 10 Section 3.2, paragraph 4
Donald and Lang and (2001)
- p. 10
described by Donald and Lang and (2001)
Judge's note: Page 10 image, Section 3.2, fourth paragraph: the extra "and" is visible.
Second check: none (judge only)
Report a problem with this finding
Typo The placebo equation '¯Yst = αs + γt + γ Pst + εst' (p.20) uses γ both for the year fixed effects and for the placebo coefficient.
- p. 21 Section 4.6 equation
¯Yst = αs + γt + γ Pst + ²st
- p. 10 Equation 4
c Xit
Judge's note: Page 21 image checked: the equation reads Ȳst = αs + γt + γ Pst + εst, and the next sentences refer to γ̂ as the placebo estimate. One symbol therefore stands for two parameters. Context makes the meaning clear, so this is cosmetic. Also, equation 4 (text layer line 378) writes 'c Xit' and 'εit' against Yist, while equation 1 has Xist. This is a minor subscript inconsistency.
Second check: none (judge only)
Report a problem with this finding
Typo "as when use the micro data" is missing the subject "we".
- p. 11 Section 3.2, paragraph on row 3
as when use the micro data
Judge's note: Visible on page 11.
Second check: none (judge only)
Report a problem with this finding
Typo "the element (i, i + j) is the correlation between εi and εi−j" uses an index offset that does not match the element position.
- p. 17 Section 4.3
the element (i, i + j) is the correlation between ²i and ²i−j
Judge's note: The index sign is a minor notation inconsistency. The sentence is otherwise clear because the block is symmetric.
Second check: none (judge only)
Report a problem with this finding
Typo "Suppose that the we make the hypothesis" contains a stray "the".
- p. 20 Section 4.6, flu shot illustration
Suppose that the we make the hypothesis
Judge's note: Visible in the crop.
Second check: none (judge only)
Report a problem with this finding
Typo "this estimates rests on strong assumptions" has a subject-verb agreement error.
- p. 20 Section 4.6
this estimates rests on strong assumptions
Judge's note: Visible in the crop.
Second check: none (judge only)
Report a problem with this finding
Typo "the statistician know that half of the sample" should read "knows".
- p. 22 Section 4.6, paragraph on additional assumption
the statistician know that half of the sample
Judge's note: Visible in the crop.
Second check: none (judge only)
Report a problem with this finding
Typo The benchmarks are listed as '78% ... no auto-correlation and 32% in AR(1)', and the RI results then as '30% and 84 %', in the opposite order (Table 11: AR(1) .3, iid .84).
- p. 22 Section 4.6
was 78% in manufactured data with no auto-correlation and 32% in AR(1) data with ρ = .8. The performance of randomization inference, 30% and 84 %
- p. 38 Table 11 panel B
AR(1), rho=0.8 ... .3 ... iid data ... .84
- p. 22 Section 4.6
was 78% in manufactured data with no auto-correlation and 32% in AR(1) data with ρ = .8. The performance of randomization inference, 30% and 84 % is comparable
- p. 38 Table 11 panel B
AR(1), rho=0.8 Randomization Inference 50 .05 .3
Judge's note: Pages 22 and 38 images checked. The values match Table 11 Panel B (AR(1) rho=0.8 gives .3; iid gives .84). Only the order is reversed relative to the preceding benchmarks. No 'respectively' is used, so this is a minor readability point.
Second check: none (judge only)
Report a problem with this finding
Typo Table 1 prints "Number with potention serial correlation problem" and "Include laggged dependent variable"; these should read "potential" and "lagged".
- p. 27 Table 1
Number with potention serial correlation problem
- p. 27 Table 1
Include laggged dependent variable
Judge's note: Both misspellings are visible in the page 27 image.
Second check: none (judge only)
Report a problem with this finding
Typo Table 4 prints the autocorrelations for 'CPS agg, Changes in log wage' as '-.046, .032, 002' (missing decimal point). The SE for AR(1) ρ=-.4, 2% effect, is printed '.026)' without its opening parenthesis.
- p. 30 Table 4, Changes in log wage
-.046, .032, 002
- p. 30 Table 4, AR(1) rho=-.4 SE
.026)
Judge's note: Page 30 image confirms both typographic slips: '002' should read .002, and '.026)' should read (.026).
Second check: none (judge only)
Report a problem with this finding
Typo Table 4, last row AR(1) -.4, 2% Effect SE is printed ".026)" without an opening parenthesis.
- p. 30 Table 4, last row (AR(1), -.4), 2% Effect standard error
.026)
Judge's note: Seen on the full page 30 image; every other SE is fully parenthesised.
Second check: none (judge only)