Optimizing Test Placement for Module-Level Regression Testing

• Main Authors: Shi, August, Thummalapenta, Suresh, Lahiri, Shuvendu K., Bjorner, Nikolaj, Czerwonka, Jacek
• Format: Conference Proceeding
• Language: English
• Subjects: build system; Buildings; Google; Greedy algorithms; Metadata; module-level regression testing; Productivity; regression test selection; Software; Testing
• ISSN: 1558-1225
• DOI: 10.1109/ICSE.2017.69

Modern build systems help increase developer productivity by performing incremental building and testing. These build systems view a software project as a group of interdependent modules and perform regression test selection at the module level. However, many large software projects have imprecise dependency graphs that lead to wasteful test executions. If a test belongs to a module that has more dependencies than the actual dependencies of the test, then it is executed unnecessarily whenever a code change impacts those additional dependencies. In this paper, we formulate the problem of wasteful test executions due to suboptimal placement of tests in modules. We propose a greedy algorithm to reduce the number of test executions by suggesting test movements while considering historical build information and actual dependencies of tests. We have implemented our technique, called TestOptimizer, on top of CloudBuild, the build system developed within Microsoft over the last few years. We have evaluated the technique on five large proprietary projects. Our results show that the suggested test movements can lead to a reduction of 21.66 million test executions (17.09%) across all our subject projects. We received encouraging feedback from the developers of these projects; they accepted and intend to implement ≈80% of our reported suggestions.