External Validity in Policy Evaluations That Choose Sites Purposively

Evaluations of the impact of social programs are often carried out in multiple sites, such as school districts, housing authorities, local TANF offices, or One-Stop Career Centers. Most evaluations select sites purposively following a process that is nonrandom. Unfortunately, purposive site selectio...

Full description

Saved in:
Bibliographic Details
Published in:Journal of policy analysis and management 2013, Vol.32 (1), p.107-121
Main Authors: Olsen, Robert B., Orr, Larry L., Bell, Stephen H., Stuart, Elizabeth A.
Format: Article
Language:English
Subjects:
Bias
Careers
Conceptualization
Decision making models
Educational evaluation
Estimation bias
Ethnic groups
Evaluation
Evidence
Family
Housing
Housing authorities
Mathematical Formulas
Methods for Policy Analysis
Personnel evaluation
Policy
Policy analysis
Population
Population estimates
Population policy
Poverty relief
Probability
Public policy
Research biases
Sampling
Sampling bias
School districts
Schools
Site Selection
Social policy
Social services
Statistical Bias
Studies
Validity
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Evaluations of the impact of social programs are often carried out in multiple sites, such as school districts, housing authorities, local TANF offices, or One-Stop Career Centers. Most evaluations select sites purposively following a process that is nonrandom. Unfortunately, purposive site selection can produce a sample of sites that is not representative of the population of interest for the program. In this paper, we propose a conceptual model of purposive site selection. We begin with the proposition that a purposive sample of sites can usefully be conceptualized as a random sample of sites from some well-defined population, for which the sampling probabilities are unknown and vary across sites. This proposition allows us to derive a formal, yet intuitive, mathematical expression for the bias in the pooled impact estimate when sites are selected purposively. This formula helps us to better understand the consequences of selecting sites purposively, and the factors that contribute to the bias. Additional research is needed to obtain evidence on how large the bias tends to be in actual studies that select sites purposively, and to develop methods to increase the external validity of these studies.
ISSN:0276-8739
1520-6688
DOI:10.1002/pam.21660