Towards Robustness of Deep Program Processing Models—Detection, Estimation, and Enhancement

• Published in: ACM transactions on software engineering and methodology 2022-07, Vol.31 (3), p.1-40, Article 50
• Main Authors: Zhang, Huangzhao, Fu, Zhiyi, Li, Ge, Ma, Lei, Zhao, Zhehao, Yang, Hua’an, Sun, Yizhe, Liu, Yang, Jin, Zhi
• Format: Article
• Language: English
• Subjects: Artificial intelligence; Computing methodologies; Software and its engineering; Software testing and debugging
• ISSN: 1049-331X; 1557-7392
• DOI: 10.1145/3511887

Deep learning (DL) has recently been widely applied to diverse source code processing tasks in the software engineering (SE) community, which achieves competitive performance (e.g., accuracy). However, the robustness, which requires the model to produce consistent decisions given minorly perturbed code inputs, still lacks systematic investigation as an important quality indicator. This article initiates an early step and proposes a framework CARROT for robustness detection, measurement, and enhancement of DL models for source code processing. We first propose an optimization-based attack technique CARROTA to generate valid adversarial source code examples effectively and efficiently. Based on this, we define the robustness metrics and propose robustness measurement toolkit CARROTM, which employs the worst-case performance approximation under the allowable perturbations. We further propose to improve the robustness of the DL models by adversarial training (CARROTT) with our proposed attack techniques. Our in-depth evaluations on three source code processing tasks (i.e., functionality classification, code clone detection, defect prediction) containing more than 3 million lines of code and the classic or SOTA DL models, including GRU, LSTM, ASTNN, LSCNN, TBCNN, CodeBERT, and CDLH, demonstrate the usefulness of our techniques for ❶ effective and efficient adversarial example detection, ❷ tight robustness estimation, and ❸ effective robustness enhancement.