Loading…

Are Poor Quality Data Just Random Responses?: A Crowdsourced Study of Delay Discounting in Alcohol Use Disorder

Crowdsourced methods of data collection such as Amazon Mechanical Turk (MTurk) have been widely adopted in addiction science. Recent reports suggest an increase in poor quality data on MTurk, posing a challenge to the validity of findings. However, empirical investigations of data quality in addicti...

Full description

Saved in:
Bibliographic Details
Published in:Experimental and clinical psychopharmacology 2022-08, Vol.30 (4), p.409-414
Main Authors: Craft, William H., Tegge, Allison N., Freitas-Lemos, Roberta, Tomlinson, Devin C., Bickel, Warren K.
Format: Article
Language:English
Subjects:
Citations: Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Crowdsourced methods of data collection such as Amazon Mechanical Turk (MTurk) have been widely adopted in addiction science. Recent reports suggest an increase in poor quality data on MTurk, posing a challenge to the validity of findings. However, empirical investigations of data quality in addiction-related samples are lacking. In this study of individuals with alcohol use disorder (AUD), we compared poor quality delay discounting data to randomly generated data. A reanalysis of prior published delay discounting data was conducted comparing included, excluded, and randomly generated data samples. Nonsystematic criteria were implemented as a measure of data quality. The excluded data was statistically different from the included sample but did not differ from randomly generated data on multiple metrics. Moreover, a response bias was identified in the excluded data. This study provides empirical evidence that poor quality delay discounting data in an AUD sample is not statistically different from randomly generated data, suggesting data quality concerns on MTurk persist in addiction samples. These findings support the use of rigorous methods of a priori defined criteria to remove poor quality data post hoc. Additionally, it highlights that the use of nonsystematic delay discounting criteria to remove poor quality data is rigorous and not simply a way of removing data that does not conform to an expected theoretical model. Public Health Significance This study provides empirical evidence that poor quality delay discounting data does not differ from random responding in a sample of individuals with alcohol use disorder. This highlights that previous reports of poor quality data on Amazon Mechanical Turk extend to addiction-related samples. Thus, the use of rigorous data quality controls and exclusion of poor quality data are warranted to ensure high-quality scientific findings.
ISSN:1064-1297
1936-2293
DOI:10.1037/pha0000549