Distributed Assignment With Load Balancing for DNN Inference at the Edge

• Published in: IEEE internet of things journal 2023-01, Vol.10 (2), p.1053-1065
• Main Authors: Xu, Yuzhe, Mohammed, Thaha, Di Francesco, Mario, Fischione, Carlo
• Format: Article
• Language: English
• Subjects: Algorithms; Artificial neural networks; Assignment problems; Cloud computing; Combinatorial optimization; computation offloading; Computational modeling; Computational modelling; Computer architecture; deep neural network (DNN) offloading; Deep neural network offloading; Deep neural networks; distributed inference; Edge computing; Edge server; Inference; Inference engines; Internet of Things; Job analysis; Load balancing; Network architecture; Network inference; Operations research; Optimization; Polynomial approximation; Polynomials; Servers; Task analysis; Training
• ISSN: 2327-4662; 2327-4662
• DOI: 10.1109/JIOT.2022.3205410

Inference carried out on pretrained deep neural networks (DNNs) is particularly effective as it does not require retraining and entails no loss in accuracy. Unfortunately, resource-constrained devices such as those in the Internet of Things may need to offload the related computation to more powerful servers, particularly, at the network edge. However, edge servers have limited resources compared to those in the cloud; therefore, inference offloading generally requires dividing the original DNN into different pieces that are then assigned to multiple edge servers. Related approaches in the state-of-the-art either make strong assumptions on the system model or fail to provide strict performance guarantees. This article specifically addresses these limitations by applying distributed assignment to DNN inference at the edge. In particular, it devises a detailed model of DNN-based inference, suitable for realistic scenarios involving edge computing. Optimal inference offloading with load balancing is also defined as a multiple assignment problem that maximizes proportional fairness. Moreover, a distributed algorithm for DNN inference offloading is introduced to solve such a problem in polynomial time with strong optimality guarantees. Finally, extensive simulations employing different data sets and DNN architectures establish that the proposed solution significantly improves upon the state-of-the-art in terms of inference time (1.14 to 2.62 times faster), load balance (with Jain's fairness index of 0.9), and convergence (one order of magnitude less iterations).