How serious is the 'carelessness' problem on Mechanical Turk?

This study compared the quality of survey data collected from Mechanical Turk (MTurk) workers and college students. Three groups of participants completed the same survey. MTurk respondents completed the survey as paid workers using the Mechanical Turk crowdsourcing platform. Student Online responde...

Full description

Saved in:
Bibliographic Details
Published in:International journal of social research methodology 2019-09, Vol.22 (5), p.441-449
Main Authors: Aruguete, Mara S., Huynh, Ho, Browne, Blaine L., Jurs, Bethany, Flint, Emilia, McCutcheon, Lynn E.
Format: Article
Language:English
Subjects:
Classrooms
College Students
Crowdsourcing
Data Collection
Error Patterns
Errors
Internet
MTurk
Multiculturalism & pluralism
online sampling
Online Surveys
Polls & surveys
Recruitment
Research Methodology
Respondents
response set
Responses
Sampling
Students
survey research
Validity
Workers
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:This study compared the quality of survey data collected from Mechanical Turk (MTurk) workers and college students. Three groups of participants completed the same survey. MTurk respondents completed the survey as paid workers using the Mechanical Turk crowdsourcing platform. Student Online respondents also completed the survey online after having been recruited in class. Finally, Student Paper-and-Pencil respondents completed the survey on paper in a classroom setting. Validity checks embedded in the survey were designed to gauge participants' haste and carelessness in survey completion. MTurk respondents were significantly more likely to fail validity checks by contradicting their own answers or simply completing the survey too quickly. Student groups showed fewer careless mistakes and longer completion times. The MTurk sample tended to be older, more educated, and more ethnically diverse than student samples. Results suggest that researchers should pay special attention to the use of validity checks when recruiting MTurk samples.
ISSN:1364-5579
1464-5300
DOI:10.1080/13645579.2018.1563966