Lightweight Assessment of Test-Case Effectiveness Using Source-Code-Quality Indicators

• Published in: IEEE transactions on software engineering 2021-04, Vol.47 (4), p.758-774
• Main Authors: Grano, Giovanni, Palomba, Fabio, Gall, Harald C.
• Format: Article
• Language: English
• Subjects: Automated software testing; Computational modeling; Cost analysis; Estimation; Feasibility studies; Indicators; Lightweight; Machine learning; Measurement; Mutation; mutation testing; Predictive models; Production; Quality control; software quality; Testing
• ISSN: 0098-5589; 1939-3520
• DOI: 10.1109/TSE.2019.2903057

Test cases are crucial to help developers preventing the introduction of software faults. Unfortunately, not all the tests are properly designed or can effectively capture faults in production code. Some measures have been defined to assess test-case effectiveness: the most relevant one is the mutation score, which highlights the quality of a test by generating the so-called mutants , i.e., variations of the production code that make it faulty and that the test is supposed to identify. However, previous studies revealed that mutation analysis is extremely costly and hard to use in practice. The approaches proposed by researchers so far have not been able to provide practical gains in terms of mutation testing efficiency. This leaves the problem of efficiently assessing test-case effectiveness as still open. In this paper, we investigate a novel, orthogonal, and lightweight methodology to assess test-case effectiveness: in particular, we study the feasibility to exploit production and test-code-quality indicators to estimate the mutation score of a test case. We first select a set of 67 factors and study their relation with test-case effectiveness. Then, we devise a mutation score estimation model exploiting such factors and investigate its performance as well as its most relevant features. The key results of the study reveal that our estimation model only based on static features has 86 percent of both F-Measure and AUC-ROC. This means that we can estimate the test-case effectiveness, using source-code-quality indicators, with high accuracy and without executing the tests. As a consequence, we can provide a practical approach that is beyond the typical limitations of current mutation testing techniques.