Nonexperimental versus Experimental Estimates of Earnings Impacts

• Published in: The Annals of the American Academy of Political and Social Science 2003-09, Vol.589 (1), p.63-93
• Main Authors: Glazerman, Steven, Levy, Dan M., Myers, David
• Format: Article
• Language: English
• Subjects: Analytical estimating; Control groups; Datasets; Earnings; Employment; Estimation; Estimation bias; Estimation methods; Estimators; Evaluation; Evaluation Research; Experimental methods; Experiments; Job Training; Methodology; Methods; Political science; Program Evaluation; Replicative studies; Research Design; Research methods; Sampling bias; Selection bias; Social Programs; Social science research; Statistical Bias; Studies; Training; Welfare; Welfare Services; Why Experiments Matter
• ISSN: 0002-7162; 1552-3349
• DOI: 10.1177/0002716203254879

To assess nonexperimental (NX) evaluation methods in to context of welfare, job training, and employment services programs, the authors reexamined the results of twelve case studies intended to replicate impact estimates from an experimental evaluation by using NX methods. They found that the NX methods sometimes came close to replicating experimentally derived results but often produced estimates that differed by policy-relevant margins, which the authors interpret as estimates of bias. Although the authors identified several study design factors associated with smaller discrepancies, no combination of factors would consistently eliminate discrepancies. Even with a large number of impact estimates, the positive and negative bias estimates did not always cancel each other out. Thus, it was difficult to identify an aggregation strategy that consistently removed bias while answering a focused question about earnings impacts of a program. They conclude that although the empirical evidence from this literature can be used in the context of training and welfare programs to improve NX research designs, it cannot on its own justify the use of such designs.