Robust Android Malware Detection System Against Adversarial Attacks Using Q-Learning

• Published in: Information systems frontiers 2021-08, Vol.23 (4), p.867-882
• Main Authors: Rathore, Hemant, Sahay, Sanjay K., Nikam, Piyush, Sewak, Mohit
• Format: Article
• Language: English
• Subjects: Artificial neural networks; Business and Management; Control; Decision trees; Deep learning; Information systems; IT in Business; Machine learning; Malware; Management of Computing and Information Systems; Operations Research/Decision Theory; Robustness; Smartphones; Systems Theory
• ISSN: 1387-3326; 1572-9419
• DOI: 10.1007/s10796-020-10083-8

Since the inception of Andoroid OS, smartphones sales have been growing exponentially, and today it enjoys the monopoly in the smartphone marketplace. The widespread adoption of Android smartphones has drawn the attention of malware designers, which threatens the Android ecosystem. The current state-of-the-art Android malware detection systems are based on machine learning and deep learning models. Despite having superior performance, these models are susceptible to adversarial attack. Therefore in this paper, we developed eight Android malware detection models based on machine learning and deep neural network and investigated their robustness against the adversarial attacks. For the purpose, we created new variants of malware using Reinforcement Learning, which will be misclassified as benign by the existing Android malware detection models. We propose two novel attack strategies, namely single policy attack and multiple policy attack using reinforcement learning for white-box and grey-box scenario respectively. Putting ourselves in adversary’ shoes, we designed adversarial attacks on the detection models with the goal of maximising fooling rate, while making minimum modifications to the Android application and ensuring that the app’s functionality and behaviour does not change. We achieved an average fooling rate of 44.21% and 53.20% across all the eight detection models with maximum five modifications using a single policy attack and multiple policy attack, respectively. The highest fooling rate of 86.09% with five changes was attained against the decision tree based model using the multiple policy approach. Finally, we propose an adversarial defence strategy which reduces the average fooling rate by threefold to 15.22% against a single policy attack, thereby increasing the robustness of the detection models i.e. the proposed model can effectively detect variants (metamorphic) of malware. The experimental analysis shows that our proposed Android malware detection system using reinforcement learning is more robust against adversarial attacks.