Improving Sampling Probability Definitions with Predictive Algorithms

• Published in: Field methods 2023-05, Vol.35 (2), p.137-152
• Main Authors: Jannetti, Matthew, Carroll-Scott, Amy, Gilliam, Erikka, Headen, Irene, Beverly, Maggie, Lê-Scherban, Félice
• Format: Article
• Language: English
• Subjects: Accuracy; Algorithms; Artificial Intelligence; Classification; Definitions; Family (Sociological Unit); Households; Neighborhoods; Place Based Education; Prediction; Predictions; Probability; Sampling; Surveys
• ISSN: 1525-822X; 1552-3969
• DOI: 10.1177/1525822X221113181

Place-based initiatives often use resident surveys to inform and evaluate interventions. Sampling based on well-defined sampling frames is important but challenging for initiatives that target subpopulations. Databases that enumerate total population counts can produce overinclusive sampling frames, resulting in costly outreach to ineligible participants. Quantifying eligibility before sampling using machine learning algorithms can improve efficiency and reduce costs. We developed a model to improve sampling for the West Philly Promise Neighborhood’s biennial population-representative survey of households with children within a geographic footprint. This study proposes a method to estimate probability of study eligibility by building a well-calibrated predictive model using existing administrative data sources. Six machine-learning models were evaluated; logistic regression provided the best balance of accuracy and understandable probabilities. This approach can be a blueprint for other population-based studies whose sampling frames cannot be well defined using traditional sources.