Efficient Worker Assignment in Crowdsourced Data Labeling Using Graph Signal Processing

The first step in solving a classification problem is to collect and label a sufficient amount of training data. Given the time and cost associated to data labeling, crowdsourcing systems (e.g., Amazon Mechanical Turk) are often used. However, one of the key disadvantages of crowdsourcing systems is...

Full description

Saved in:
Bibliographic Details
Main Authors: Maroto, Javier, Ortega, Antonio
Format: Conference Proceeding
Language:English
Subjects:
Covariance matrices
Crowdsourcing
Eigenvalues and eigenfunctions
Graph Signal Processing
Labeling
Laplace equations
Reliability
resource allocation
Task analysis
Online Access:Request full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:The first step in solving a classification problem is to collect and label a sufficient amount of training data. Given the time and cost associated to data labeling, crowdsourcing systems (e.g., Amazon Mechanical Turk) are often used. However, one of the key disadvantages of crowdsourcing systems is the presence of spammers or workers who are not as skilled or careful, thus leading to many false labels being assigned. This paper addresses this problem by proposing a novel algorithm based on graph signal sampling theory, which optimally assigns data to different workers for labeling by taking into account the expected quality of labeling provided by each worker. Our simulation of the labeling process using these schemes shows that the classification error can be reduced significantly with respect to a random assignment of workers.
ISSN:2379-190X
DOI:10.1109/ICASSP.2018.8462493