Evaluating Amazon's Mechanical Turk as a tool for experimental behavioral research

Amazon Mechanical Turk (AMT) is an online crowdsourcing service where anonymous online workers complete web-based tasks for small sums of money. The service has attracted attention from experimental psychologists interested in gathering human subject data more efficiently. However, relative to tradi...

Full description

Saved in:
Bibliographic Details
Published in:PloS one 2013-03, Vol.8 (3), p.e57410-e57410
Main Authors: Crump, Matthew J C, McDonnell, John V, Gureckis, Todd M
Format: Article
Language:English
Subjects:
Attention
Behavioral Research
Cognitive ability
Computer Science
Cues
Data collection
Electronic commerce
Experiments
Humans
Internet
Learning
Learning Curve
Mechanical properties
Medicine
Priming
Privacy
Psychology
Reaction Time
Researchers
Science Policy
Social and Behavioral Sciences
Studies
Trends
Web browsers
Web sites
Workers
Workers' compensation
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Amazon Mechanical Turk (AMT) is an online crowdsourcing service where anonymous online workers complete web-based tasks for small sums of money. The service has attracted attention from experimental psychologists interested in gathering human subject data more efficiently. However, relative to traditional laboratory studies, many aspects of the testing environment are not under the experimenter's control. In this paper, we attempt to empirically evaluate the fidelity of the AMT system for use in cognitive behavioral experiments. These types of experiment differ from simple surveys in that they require multiple trials, sustained attention from participants, comprehension of complex instructions, and millisecond accuracy for response recording and stimulus presentation. We replicate a diverse body of tasks from experimental psychology including the Stroop, Switching, Flanker, Simon, Posner Cuing, attentional blink, subliminal priming, and category learning tasks using participants recruited using AMT. While most of replications were qualitatively successful and validated the approach of collecting data anonymously online using a web-browser, others revealed disparity between laboratory results and online results. A number of important lessons were encountered in the process of conducting these replications that should be of value to other researchers.
ISSN:1932-6203
1932-6203
DOI:10.1371/journal.pone.0057410