Using score distributions to compare statistical significance tests for information retrieval evaluation

Statistical significance tests can provide evidence that the observed difference in performance between 2 methods is not due to chance. In information retrieval (IR), some studies have examined the validity and suitability of such tests for comparing search systems. We argue here that current method...

Full description

Saved in:
Bibliographic Details
Published in:Journal of the American Society for Information Science and Technology 2020-01, Vol.71 (1), p.98-113
Main Authors: Parapar, Javier, Losada, David E., Presedo‐Quindimil, Manuel A., Barreiro, Alvaro
Format: Article
Language:English
Subjects:
Computer simulation
Information retrieval
Measurement Techniques
Null hypothesis
Permutations
Regression analysis
Reliability analysis
Searching
Semiotics
Statistical analysis
Statistical significance
Statistical tests
Test validity and reliability
Truth
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Statistical significance tests can provide evidence that the observed difference in performance between 2 methods is not due to chance. In information retrieval (IR), some studies have examined the validity and suitability of such tests for comparing search systems. We argue here that current methods for assessing the reliability of statistical tests suffer from some methodological weaknesses, and we propose a novel way to study significance tests for retrieval evaluation. Using Score Distributions, we model the output of multiple search systems, produce simulated search results from such models, and compare them using various significance tests. A key strength of this approach is that we assess statistical tests under perfect knowledge about the truth or falseness of the null hypothesis. This new method for studying the power of significance tests in IR evaluation is formal and innovative. Following this type of analysis, we found that both the sign test and Wilcoxon signed test have more power than the permutation test and the t‐test. The sign test and Wilcoxon signed test also have good behavior in terms of type I errors. The bootstrap test shows few type I errors, but it has less power than the other methods tested.
ISSN:2330-1635
2330-1643
DOI:10.1002/asi.24203