A Dynamic Pruning Method on Multiple Sparse Structures in Deep Neural Networks

It is well known that enormous computational power and a mass of memory are needed in deep neural networks. That makes them difficult to apply in resource-limited environments. Therefore, many network compression and acceleration technologies have come out, among which connection pruning is widely a...

Full description

Saved in:
Bibliographic Details
Published in:IEEE access 2023-01, Vol.11, p.1-1
Main Authors: Hu, Jie, Lin, Peng, Zhang, Huajun, Lan, Zining, Chen, Wenxin, Xie, Kailiang, Chen, Siyun, Wang, Hao, Chang, Sheng
Format: Article
Language:English
Subjects:
Artificial neural networks
Columns (structural)
Convolutional neural networks
Deep learning
dynamic channel pruning
Filtering theory
Information filters
Iterative methods
network compression and acceleration
Neural networks
Pruning
Sparse matrices
structured pruning
Tensors
Training
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:It is well known that enormous computational power and a mass of memory are needed in deep neural networks. That makes them difficult to apply in resource-limited environments. Therefore, many network compression and acceleration technologies have come out, among which connection pruning is widely applied due to its effectiveness and convenience. A novel connection pruning method with full model capacity on multiple sparse structures is proposed in this paper. We design a simple and efficient function called Dynamic Processing Unit (DPU) to deal with the evaluated weights. Our method has the following features: (1) Instead of being pruned directly or set to 0, the weights are controlled by the DPU to determine whether they will work for subsequent forward of the network during the iteration of pruning training. (2) It supports the traditional multi-steps prune method as well as the end-to-end training mode which can get a compressed network in only one stage where training and pruning are fused. (3) It can learn multiple useful sparse structures, including but not limited in depth-wise, filter-wise, channel-wise, 2D-filter-wise, row-wise, column-wise, connection-wise and mixed sparse structures. Our method is tested on various widely-used datasets and models, such as LeNet and ResNet on MNIST and CIFAR-10. Significantly, it achieves good performance in all these cases. Some details about our method can be found at this https://github.com/hujie369/DPU.
ISSN:2169-3536
2169-3536
DOI:10.1109/ACCESS.2023.3267469