Estimating the potential of program repair search spaces with commit analysis

• Published in: The Journal of systems and software 2022-06, Vol.188, p.111263, Article 111263
• Main Authors: Etemadi, Khashayar, Tarighat, Niloofar, Yadav, Siddharth, Martinez, Matias, Monperrus, Martin
• Format: Article
• Language: English
• Subjects: Commit analysis; Program repair; Search-space; Static code analysis
• ISSN: 0164-1212; 1873-1228; 1873-1228
• DOI: 10.1016/j.jss.2022.111263

The most natural method for evaluating program repair systems is to run them on bug datasets, such as Defects4J. Yet, using this evaluation technique on arbitrary real-world programs requires heavy configuration. In this paper, we propose a purely static method to evaluate the potential of the search space of repair approaches. This new method enables researchers and practitioners to encode the search spaces of repair approaches and select potentially useful ones without struggling with tool configuration and execution. We encode the search spaces by specifying the repair strategies they employ. Next, we use the specifications to check whether past commits lie in repair search spaces. For a repair approach, including many human-written past commits in its search space indicates its potential to generate useful patches. We implement our evaluation method in LighteR. LighteR gets a Git repository and outputs a list of commits whose source code changes lie in repair search spaces. We run LighteR on 55,309 commits from the history of 72 Github repositories with and show that LighteR’s precision and recall are 77% and 92%, respectively. Overall, our experiments show that our novel method is both lightweight and effective to study the search space of program repair approaches. •LighteR, a repair space analysis tool, used to detect commits lying in repair spaces.•Significant parts of search spaces of repair tools are unique.•LighteR is an efficient tool for assessing the potential of repair tools.