Minimal test collections for low-cost evaluation of Audio Music Similarity and Retrieval systems

Reliable evaluation of Information Retrieval systems requires large amounts of relevance judgments. Making these annotations is not only tedious but also complex for many Music Information Retrieval tasks. As a result, performing such evaluations usually requires too much effort. A low-cost alternat...

Full description

Saved in:
Bibliographic Details
Published in:International journal of multimedia information retrieval 2013-03, Vol.2 (1), p.59-70
Main Authors: Urbano, Julián, Schedl, Markus
Format: Article
Language:English
Subjects:
Algorithms
Annotations
Audio data
Computer Science
Data Mining and Knowledge Discovery
Database Management
Estimates
Experiments
Image Processing and Computer Vision
Information retrieval
Information Storage and Retrieval
Information Systems Applications (incl.Internet)
Low cost
Multimedia Information Systems
Music
Queries
Random variables
Regular Paper
Similarity
Task complexity
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Reliable evaluation of Information Retrieval systems requires large amounts of relevance judgments. Making these annotations is not only tedious but also complex for many Music Information Retrieval tasks. As a result, performing such evaluations usually requires too much effort. A low-cost alternative is the application of Minimal Test Collections algorithms, which offer very reliable results while significantly reducing the required annotation effort. The idea is to represent effectiveness scores as random variables that can be estimated, iteratively selecting which documents to judge so that we can compute accurate estimates with a certain degree of confidence and with the least effort. In this paper we show the application of Minimal Test Collections to the evaluation of the Audio Music Similarity and Retrieval task, run by the annual MIREX evaluation campaign. An analysis with the MIREX 2007, 2009, 2010 and 2011 data shows that with as little as 2 % of the total judgments we can obtain accurate estimates of the ranking of systems. We also present a method to rank systems without making any annotations, which can be successfully used when little or no resources are available.
ISSN:2192-6611
2192-662X
DOI:10.1007/s13735-012-0030-4