Structured Pruning for Efficient Convolutional Neural Networks via Incremental Regularization

Modern Convolutional Neural Networks (CNNs) are usually restricted by their massive computation and high storage. Parameter pruning is a promising approach for CNN compression and acceleration by eliminating redundant model parameters with tolerable performance degradation. Despite its effectiveness...

Full description

Saved in:
Bibliographic Details
Published in:IEEE journal of selected topics in signal processing 2020-05, Vol.14 (4), p.775-788
Main Authors: Wang, Huan, Hu, Xinyi, Zhang, Qiming, Wang, Yuehai, Yu, Lu, Hu, Haoji
Format: Article
Language:English
Subjects:
Acceleration
Artificial neural networks
Biological neural networks
Computational modeling
Convolutional neural network
Datasets
Empirical analysis
Fragility
Incremental regularization
Kernel
Machine learning
Mathematical models
Matrix decomposition
Model compression
Neural networks
Parameters
Performance degradation
Pruning
Regularization
Structured pruning
Training
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Modern Convolutional Neural Networks (CNNs) are usually restricted by their massive computation and high storage. Parameter pruning is a promising approach for CNN compression and acceleration by eliminating redundant model parameters with tolerable performance degradation. Despite its effectiveness, existing regularization-based parameter pruning methods usually drive weights towards zero with large and constant regularization factors, which neglects the fragility of the expressiveness of CNNs, and thus calls for a more gentle regularization scheme so that the networks can adapt during pruning. To achieve this, we propose a novel regularization-based pruning method, named IncReg , to incrementally assign different regularization factors to different weights based on their relative importance. Empirical analysis on CIFAR-10 dataset verifies the merits of IncReg. Further extensive experiments with popular CNNs on CIFAR-10 and ImageNet datasets show that IncReg achieves comparable to even better results compared with state-of-the-arts. Moreover, to resolve the problem that column pruning cannot be directly applied to off-the-shelf deep learning libraries for acceleration, we generalize IncReg from column pruning to spatial pruning, which can equip existing structured pruning methods (such as channel pruning) for further acceleration with ignorable accuracy loss. Our source codes and trained models are available at: https://github.com/mingsun-tse/caffe_increghttps://github.com/mingsun-tse/caffe_increg .
ISSN:1932-4553
1941-0484
DOI:10.1109/JSTSP.2019.2961233