Advantages of Matched Over Unmatched Opt-in Samples for Studying Criminal Justice Attitudes: A Research Note

• Published in: Crime and delinquency 2021-11, Vol.67 (12), p.1962-1981
• Main Authors: Graham, Amanda, Pickett, Justin T., Cullen, Francis T.
• Format: Article
• Language: English
• Subjects: Criminal justice; Criminology; Data quality; Generalizability; Popularity; Probability; Public opinion; Research methodology
• ISSN: 0011-1287; 1552-387X
• DOI: 10.1177/0011128720977439

Despite the growing popularity of online opt-in samples in criminology, recent work shows that resultant findings often do not generalize. Not all opt-in samples are alike, however, and matching may improve data quality. Replicating and extending prior work, we compare the generalizability of relational inferences from unmatched and matched opt-in samples. Estimating identical models for four criminal justice outcomes, we compare multivariate regression results from national matched (YouGov) and unmatched (MTurk) opt-in samples to those from the General Social Survey (GSS). YouGov coefficients are almost always in the same direction as GSS coefficients, especially when statistically significant, and are mostly of a similar magnitude; less than 10% of the YouGov and GSS coefficients differ significantly. By contrast, MTurk coefficients are more likely to be in the wrong direction, more likely to be much larger or smaller, and are about three times as likely to differ significantly from GSS coefficients. Matched opt-in samples provide a relatively inexpensive data source for criminal justice researchers, compared to probability samples, and also appear to carry a smaller generalizability penalty than unmatched samples. Our study suggests relational inferences from matched opt-in samples are more likely to generalize than those from unmatched samples.