Assessing Correspondence Between Experimental and Nonexperimental Estimates in Within-Study Comparisons

• Published in: Evaluation review 2018-04, Vol.42 (2), p.214-247
• Main Authors: Steiner, Peter M., Wong, Vivian C.
• Format: Article
• Language: English
• Subjects: Clinical trials; Design; Discontinuity; Educational evaluation; Educational tests & measurements; Equivalency Tests; Health technology assessment; Quasi-experimental methods; Simulation
• ISSN: 0193-841X; 1552-3926
• DOI: 10.1177/0193841X18773807

In within-study comparison (WSC) designs, treatment effects from a nonexperimental design, such as an observational study or a regression-discontinuity design, are compared to results obtained from a well-designed randomized control trial with the same target population. The goal of the WSC is to assess whether nonexperimental and experimental designs yield the same results in field settings. A common analytic challenge with WSCs, however, is the choice of appropriate criteria for determining whether nonexperimental and experimental results replicate. This article examines different distance-based correspondence measures for assessing correspondence in experimental and nonexperimental estimates. Distance-based measures investigate whether the difference in estimates is small enough to claim equivalence of methods. We use a simulation study to examine the statistical properties of common correspondence measures and recommend a new and straightforward approach that combines traditional significance testing and equivalence testing in the same framework. The article concludes with practical advice on assessing and interpreting results in WSC contexts.