Understanding Data Quality: Instructional Comprehension as a Practical Metric in Crowdsourced Investigations of Behavioral Economic Cigarette Demand

• Published in: Experimental and clinical psychopharmacology 2022-08, Vol.30 (4), p.415-423
• Main Authors: Freitas-Lemos, Roberta, Tegge, Allison N., Craft, William H., Tomlinson, Devin C., Stein, Jeffrey S., Bickel, Warren K.
• Format: Article
• Language: English
• Subjects: Behavioral Economics; Comprehension; Crowdsourcing; Data Collection; Experimental Instructions; Experimental Recruitment; Female; Human; Male; Online Experiments; Tobacco Smoking
• ISSN: 1064-1297; 1936-2293
• DOI: 10.1037/pha0000579

Crowdsourcing platforms allow researchers to quickly recruit and collect behavioral economic measures in substance-using populations, such as cigarette smokers. Despite the broad utility and flexibility, data quality issues have been an object of concern. In two separate studies recruiting cigarette smokers, we sought to investigate the association between a practical quality control measure (accuracy on an instruction quiz), on internal consistency of number of cigarettes smoked per day and purchasing patterns of tobacco products in an experimental tobacco marketplace (ETM; Study 1), and in a cigarette purchase task (CPT; Study 2). Participants (N = 312 in Study 1; N = 119 in Study 2) were recruited from Amazon mechanical turk. Both studies included task instructions, a quiz, a purchase task, cigarette usage and dependence questions, and demographics. The results show that participants who answered all instruction items correctly: (a) reported the number of cigarettes per day more consistently (partial η2 = 0.11, p < .001, Study 1; partial η2 = 0.09, p = .016, Study 2), (b) demonstrated increased model fit among the cigarette demand curves (partial η2 = 0.23, p < .001, Study 1; partial η2 = 0.08, p = .002, Study 2), and purchased tobacco products in the ETM more consistently with their current usage. We conclude that instruction quizzes before purchase tasks may be useful for researchers evaluating demand data. Instruction quizzes with multiple items may allow researchers to choose the level of data quality appropriate for their studies. Public Health Significance Increasing use of crowdsourced data along with concerns regarding data quality represent a challenge to scientific validity. We employed a practical metric for understanding data quality of behavioral economic cigarette demand, instructional comprehension. These methods could be deployed to improve data quality and scientific validity of crowdsourced data.