An Energy-Constrained Optimization-Based Structured Pruning Method for Deep Neural Network Compression

• Main Authors: Liu, Siping, Li, Qiang, Peng, Bo, Qiao, Yanping, Lin, Maosong, Zeng, Qiang
• Format: Conference Proceeding
• Language: English
• Subjects: Convolution; Deep learning; deep neural network; Energy consumption; energy-constrained optimization; Memory management; model compression; Neural networks; Performance evaluation; Proposals; structured pruning
• ISSN: 2576-7828
• DOI: 10.1109/ICCT56141.2022.10073392

Deep neural networks are widely used in modern intelligent applications due to their superior ability to express reality, and these intelligent applications run on highly energy-constrained edge devices. Neural network structured pruning is an efficient method to reduce the energy consumption of neural networks. This paper proposes an energy-constrained optimization-based structured pruning method to compress deep neural network. This method uses the energy budget provided by edge devices as a constraint for neural network pruning. The key idea is to formulate deep neural network structured pruning as an optimization problem, in which the energy estimate of each layer is used as the Frobenius norm optimization constraint for the convolution kernel weights. Then the importance of all convolution kernels in the network can be evaluated, thereby the neural network structure can be pruned according to the importance to reduce the energy consumption and memory size of the neural network. Compared with traditional heuristic structured pruning methods, our proposal enables neural networks to achieve higher accuracy with the same or lower energy budget.