MR-Adopt: Automatic Deduction of Input Transformation Function for Metamorphic Testing

• Main Authors: Xu, Congying, Chen, Songqiang, Wu, Jiarong, Cheung, Shing-Chi, Terragni, Valerio, Zhu, Hengcheng, Cao, Jialun
• Format: Conference Proceeding
• Language: English
• Subjects: Code Generation; Codes; Debugging; Input Transformation; Large Language Models; Metamorphic Relation; Metamorphic Testing; Software; Software and its engineering > Software creation and management > Software verification and validation > Software defect analysis > Software testing and debugging; Software engineering; Software testing; Transforms
• ISSN: 2643-1572
• DOI: 10.1145/3691620.3696020

While a recent study reveals that many developer-written test cases can encode a reusable Metamorphic Relation (MR), over 70% of them directly hard-code the source input and follow-up input in the encoded relation. Such encoded MRs, which do not contain an explicit input transformation to transform the source inputs to corresponding follow-up inputs, cannot be reused with new source inputs to enhance test adequacy. In this paper, we propose MR-Adopt (Automatic Deduction Of inPut Transformation) to automatically deduce the input transformation from the hard-coded source and follow-up inputs, aiming to enable the encoded MRs to be reused with new source inputs. With typically only one pair of source and follow-up inputs available in an MR-encoded test case as the example, we leveraged LLMs to understand the intention of the test case and generate additional examples of source-followup input pairs. This helps to guide the generation of input transformations generalizable to multiple source inputs. Besides, to mitigate the issue that LLMs generate erroneous code, we refine LLM-generated transformations by removing MR-irrelevant code elements with data-flow analysis. Finally, we assess candidate transformations based on encoded output relations and select the best transformation as the result. Evaluation results show that MR-Adopt can generate input transformations applicable to all experimental source inputs for 72.00% of encoded MRs, which is 33.33% more than using vanilla GPT-3.5. By incorporating MR-Adopt-generated input transformations, encoded MR-based test cases can effectively enhance the test adequacy, increasing the line coverage and mutation score by 10.62% and 18.91%, respectively.