The impact of IR-based classifier configuration on the performance and the effort of method-level bug localization

• Published in: Information and software technology 2018-10, Vol.102, p.160-174
• Main Authors: Tantithamthavorn, Chakkrit, Lemma Abebe, Surafel, Hassan, Ahmed E., Ihara, Akinori, Matsumoto, Kenichi
• Format: Article
• Language: English
• Subjects: Bug localization; Classifier configuration; Effort; Evaluation metrics; Top-k performance
• ISSN: 0950-5849; 1873-6025
• DOI: 10.1016/j.infsof.2018.06.001

•Investigating the impact of the IR-based classifier configuration on the top-k performance and the required effort.•Investigating whether the most efficient classifier configuration for file-level bug localization is also the most efficient at the method-level.•We execute a large space of classifier configuration, 3,172 in total, on 5,266 bug reports of two software systems, i.e., Eclipse and Mozilla.•The configuration has a large impact on both the top-k performance and the required effort for method-level bug localization.•IR-based configuration settings should be carefully selected and the required effort metrics should be included in future bug localization studies. IR-based bug localization is a classifier that assists developers in locating buggy source code entities (e.g., files and methods) based on the content of a bug report. Such IR-based classifiers have various parameters that can be configured differently (e.g., the choice of entity representation). In this paper, we investigate the impact of the choice of the IR-based classifier configuration on the top-k performance and the required effort to examine source code entities before locating a bug at the method level. We execute a large space of classifier configuration, 3172 in total, on 5266 bug reports of two software systems, i.e., Eclipse and Mozilla. We find that (1) the choice of classifier configuration impacts the top-k performance from 0.44% to 36% and the required effort from 4395 to 50,000 LOC; (2) classifier configurations with similar top-k performance might require different efforts; (3) VSM achieves both the best top-k performance and the least required effort for method-level bug localization; (4) the likelihood of randomly picking a configuration that performs within 20% of the best top-k classifier configuration is on average 5.4% and that of the least effort is on average 1%; (5) configurations related to the entity representation of the analyzed data have the most impact on both the top-k performance and the required effort; and (6) the most efficient classifier configuration obtained at the method-level can also be used at the file-level (and vice versa). Our results lead us to conclude that configuration has a large impact on both the top-k performance and the required effort for method-level bug localization, suggesting that the IR-based configuration settings should be carefully selected and the required effort metric should be included in future bug localization studie