Pruning Deep Neural Networks with L0-Constrained Optimization

Deep neural networks (DNNs) give state-of-the-art accuracy in many tasks, but they can require large amounts of memory storage, energy consumption, and long inference times. Modern DNNs can have hundreds of million parameters, which make it difficult for DNNs to be deployed in some applications with...

Full description

Saved in:
Bibliographic Details
Main Authors: Phan, Dzung T., Nguyen, Lam M., Nguyen, Nam H., Kalagnanam, Jayant R.
Format: Conference Proceeding
Language:English
Subjects:
Convergence
deep learning
L_0 norm optimization
Memory management
Neural networks
Optimization
pruning neural network
Quadratic programming
Rectifiers
Task analysis
Online Access:Request full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Deep neural networks (DNNs) give state-of-the-art accuracy in many tasks, but they can require large amounts of memory storage, energy consumption, and long inference times. Modern DNNs can have hundreds of million parameters, which make it difficult for DNNs to be deployed in some applications with low-resource environments. Pruning redundant connections without sacrificing accuracy is one of popular approaches to overcome these limitations. We propose two ℓ_0-constrained optimization models for pruning deep neural networks layer-by-layer. The first model is devoted to a general activation function, while the second one is specifically for a ReLU. We introduce an efficient cutting plane algorithm to solve the latter to optimality. Our experiments show that the proposed approach achieves competitive compression rates over several state-of-the-art baseline methods.
ISSN:2374-8486
DOI:10.1109/ICDM50108.2020.00152