Variance Estimation in Evaluations With No-Shows: A Comparison of Methods

In experimental evaluations of policy interventions, the so-called Bloom adjustment is commonly used to estimate the impact of the treatment on the treated. It does so by rescaling the estimated impact of the intention to treat—that is, the overall treatment-control group difference in outcomes for...

Full description

Saved in:
Bibliographic Details
Published in:The American journal of evaluation 2019-03, Vol.40 (1), p.104-118
Main Authors: Litwok, Daniel, Peck, Laura R.
Format: Article
Language:English
Subjects:
Compliance (Psychology)
Computation
Control Groups
Error of Measurement
Experimental Groups
Intention
Outcomes of Treatment
Program Evaluation
Scaling
Statistical Bias
Citations: Items that this one cites
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:In experimental evaluations of policy interventions, the so-called Bloom adjustment is commonly used to estimate the impact of the treatment on the treated. It does so by rescaling the estimated impact of the intention to treat—that is, the overall treatment-control group difference in outcomes for the entire experimental sample—by the percentage of cases that took up the treatment offer. The practice of also rescaling the variance, as is common in the literature, imposes simplifying assumptions that may lead to biased variance estimates. We compare variances using the Bloom adjustment to variances that capture all the estimation error. While the difference between these variances is negligible in three experimental evaluations, we highlight three conditions that could result in larger relative bias: large impacts, large variability in compliance, and/or substantial endogeneity bias. The presence of these conditions could potentially result in a different conclusion for policy or practice.
ISSN:1098-2140
1557-0878
DOI:10.1177/1098214017749318