Beyond confusion matrix: learning from multiple annotators with awareness of instance features

Learning from multiple annotators aims to induce a high-quality classifier from training instances, where each of them is associated with a set of observed labels provided by multiple annotators under the impact of their varying abilities and own biases. When modeling the probability transition proc...

Full description

Saved in:
Bibliographic Details
Published in:Machine learning 2023-03, Vol.112 (3), p.1053-1075
Main Authors: Li, Jingzheng, Sun, Hailong, Li, Jiyi
Format: Article
Language:English
Subjects:
Artificial Intelligence
Classifiers
Computer architecture
Computer Science
Confusion
Control
Crowdsourcing
Datasets
Labeling
Labels
Learning
Machine Learning
Mechatronics
Methods
Modelling
Modules
Natural Language Processing (NLP)
Neural networks
Roads & highways
Robotics
Simulation and Modeling
Weakly Supervised Representation Learning
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Learning from multiple annotators aims to induce a high-quality classifier from training instances, where each of them is associated with a set of observed labels provided by multiple annotators under the impact of their varying abilities and own biases. When modeling the probability transition process from latent true labels to observed labels, most existing methods adopt class-level confusion matrices of annotators which assume that observed labels do not depend on the instance features and are just determined by the true labels. However, in practice the labeling process of annotators is impacted not only by the correlation between classes but also by the content of instances. Thus using only class-level confusion matrices to characterize the probability transition process may limit the performance that the classifier can achieve. In this work, we propose the noise transition matrix, that incorporates the impact of instance features on annotators’ performance based on confusion matrices. Furthermore, we propose a simple and effective learning framework, which consists of a classifier module and a noise transition matrix module in a unified neural network architecture. Experimental results on synthetic and real datasets demonstrate the noise transition matrix is better than the confusion matrix for modeling multiple annotators and the superiority of our method in comparison with state-of-the-art methods.
ISSN:0885-6125
1573-0565
DOI:10.1007/s10994-022-06211-x