Towards an inter-observer analysis framework for multimedia quality assessment

In multimedia quality assessment, observer ratings are typically averaged into mean opinion scores (MOS) to obtain a subjective ground truth for a set of stimuli. Valuable information about individual observer rating behaviour and inter-observer differences is lost that can be useful to improve subj...

Full description

Saved in:
Bibliographic Details
Main Authors: Engelke, Ulrich, Pitrey, Y., Le Callet, P.
Format: Conference Proceeding
Language:English
Subjects:
Correlation
Data mining
Inter-observer differences
inter-rater coefficients
Multimedia communication
multimedia quality assessment
Observers
Predictive models
Quality assessment
Three dimensional displays
Online Access:Request full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:In multimedia quality assessment, observer ratings are typically averaged into mean opinion scores (MOS) to obtain a subjective ground truth for a set of stimuli. Valuable information about individual observer rating behaviour and inter-observer differences is lost that can be useful to improve subjective experiment procedures and quality of experience prediction models. In this paper, we present an inter-observer analysis framework that addresses quality assessment from a different angle, setting the focus on observer differences rather than stimuli differences. The framework consists of procedures for inter-observer analysis as well as the necessary considerations during preparation and post-processing. The aim of this paper is to raise awareness that consideration of MOS alone simplifies quality assessment too much in the context of multimedia quality assessment.
DOI:10.1109/QoMEX.2011.6065700