Adaptive attention fusion network for cross-device GUI element re-identification in crowdsourced testing

• Published in: Neurocomputing (Amsterdam) 2024-05, Vol.580, p.127502, Article 127502
• Main Authors: Zhang, Li, Tsai, Wei-Tek
• Format: Article
• Language: English
• Subjects: Adaptive attention mechanism; Computer vision; Convolutional neural network; Cross-device testing; Crowdtesting; GUI element re-identification
• ISSN: 0925-2312; 1872-8286
• DOI: 10.1016/j.neucom.2024.127502

The rapid growth of mobile devices has ushered in an era of different device platforms. Different devices require a consistent user experience, especially with similar graphical user interfaces (GUIs). However, the different code bases of the various operating systems as well as the different GUI layouts and resolutions of the various devices pose a challenge for automated software testing. Crowdsourced software testing (CST) has emerged as a viable solution where crowdsourced workers perform tests on their own devices and provide detailed bug reports. Although CST is cost-effective, it is not very efficient and requires a large number of workers for manual testing. The potential of optimizing CST reproduction testing through computer vision remains largely untapped, especially when considering the uniformity of GUI elements on different devices. In this study, we present a novel deep learning model specifically designed to re-identify GUI elements in CST reproduction test scenarios, regardless of the underlying code changes on different devices. The model features a robust backbone network for feature extraction, an innovative attention mechanism with learnable factors to enhance the features of GUI elements and minimize interference from their backgrounds, and a classifier to determine matching labels for these elements. Our approach was validated on a large GUI element dataset containing 31,098 element images for training, 115,704 element images from real apps for testing, and 67 different background images. The results of our experiments underline the excellent accuracy of the model and the importance of each component. This work is a major step forward in improving the efficiency of reproduction testing in CST. The innovative solutions we propose could further reduce labor costs for CST platforms. •ERINet: A novel CNN model for re-identifying GUI elements across devices.•Attention mechanism with Learnable Factor enhances GUI element re-identification.•Dataset: 31,098 training, 115,704 testing GUI images, and 67 backgrounds.•Potential to revolutionize cross-device testing in crowd-sourced environments.