Comparing mutation coverage against branch coverage in an industrial setting

The state-of-the-practice in software development is driven by constant change fueled by continues integration servers. Such constant change demands for frequent and fully automated tests capable to detect faults immediately upon project build. As the fault detection capability of the test suite bec...

Full description

Saved in:
Bibliographic Details
Published in:International journal on software tools for technology transfer 2020-08, Vol.22 (4), p.365-388
Main Authors: Parsai, Ali, Demeyer, Serge
Format: Article
Language:English
Subjects:
Computer Science
Construction
Fault detection
Foundation for Mastering Change
Mutation
Software development
Software Engineering
Software Engineering/Programming and Operating Systems
Theory of Computation
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:The state-of-the-practice in software development is driven by constant change fueled by continues integration servers. Such constant change demands for frequent and fully automated tests capable to detect faults immediately upon project build. As the fault detection capability of the test suite becomes so important, modern software development teams continuously monitor the quality of the test suite as well. However, it appears that the state-of-the-practice is reluctant to adopt strong coverage metrics (namely mutation coverage), instead relying on weaker kinds of coverage (namely branch coverage). In this paper, we investigate three reasons that prohibit the adoption of mutation coverage in a continuous integration setting: (1) the difficulty of its integration into the build system, (2) the perception that branch coverage is “good enough”, and (3) the performance overhead during the build. Our investigation is based on a case study involving four open source systems and one industrial system. We demonstrate that mutation coverage reveals additional weaknesses in the test suite compared to branch coverage and that it is able to do so with an acceptable performance overhead during project build.
ISSN:1433-2779
1433-2787
DOI:10.1007/s10009-020-00567-y