A Novel Heterogeneous Computing Middleware for Mobile AI Services

• Main Authors: Shao, Zihao, Su, Tonghua, Xu, Manyang, Liu, Qinglin, Han, Ruipeng, Wang, Zhongjie
• Format: Conference Proceeding
• Language: English
• Subjects: Computational modeling; Deep learning; Edge devices; Heterogeneous networks; Heterogeneous scheduling; Middleware service; Mobile applications; Mobile handsets; Processor scheduling; Schedules; Scheduling; User interfaces
• ISSN: 2767-9918
• DOI: 10.1109/EDGE55608.2022.00034

Mobile applications play an important role in edge computing. Recently, with the advancement of more powerful mobile devices, deep models have been widely deployed in mobile applications to provide more intelligent and convenient services to users. However, mobile devices still face critical challenges, such as restricted power, sensitive latency, limited memory. As emerging AI chips are integrated into mobile devices, mobile AI solutions are proposed to accelerate model execution and improve mobile service experience. Unfortunately, they only focus on the acceleration of single chip and ignore the heterogeneity among different chips. This paper proposes a mobile heterogeneous computing AI middleware, which provides developers of AI applications with the ability to accelerate model execution. This proposed middleware utilizes three heterogeneous chips, CPU, GPU, DSP for model scheduling at a layer-wise granularity. It is designed in five layers from top to bottom: User Interface Layer, Heterogeneous Scheduling Layer, Performance Collection Layer, Hardware Information Collection Layer, AI Framework Layer. We verified it in the environment of typical mobile device through a series of experiments. The results prove the high accuracy of data collection algorithm and the effectiveness of heterogeneous scheduling services. For AlexNet and Inception V3, our schedule plan achieves at least 36.39% improvement in inference time and at least 24.4% reduction in energy consumption.