GRIM: A General, Real-Time Deep Learning Inference Framework for Mobile Devices Based on Fine-Grained Structured Weight Sparsity

It is appealing but challenging to achieve real-time deep neural network (DNN) inference on mobile devices, because even the powerful modern mobile devices are considered as "resource-constrained" when executing large-scale DNNs. It necessitates the sparse model inference via weight prunin...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on pattern analysis and machine intelligence 2022-10, Vol.44 (10), p.6224-6239
Main Authors: Niu, Wei, Li, Zhengang, Ma, Xiaolong, Dong, Peiyan, Zhou, Gang, Qian, Xuehai, Lin, Xue, Wang, Yanzhi, Ren, Bin
Format: Article
Language:English
Subjects:
Artificial neural networks
Compiler optimization
Compilers
Computational modeling
deep neural networks
Electronic devices
Inference
Kernel
Machine learning
mobile computing
Mobile handsets
Model accuracy
model compression
Neural networks
Optimization
Performance evaluation
Pruning
Real time
real-time inference
Real-time systems
Recurrent neural networks
Sparsity
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:It is appealing but challenging to achieve real-time deep neural network (DNN) inference on mobile devices, because even the powerful modern mobile devices are considered as "resource-constrained" when executing large-scale DNNs. It necessitates the sparse model inference via weight pruning, i.e., DNN weight sparsity, and it is desirable to design a new DNN weight sparsity scheme that can facilitate real-time inference on mobile devices while preserving a high sparse model accuracy. This paper designs a novel mobile inference acceleration framework GRIM that is General to both convolutional neural networks (CNNs) and recurrent neural networks (RNNs) and that achieves Real-time execution and high accuracy, leveraging fine-grained structured sparse model Inference and compiler optimizations for Mobiles. We start by proposing a new fine-grained structured sparsity scheme through the Block-based Column-Row (BCR) pruning. Based on this new fine-grained structured sparsity, our GRIM framework consists of two parts: (a) the compiler optimization and code generation for real-time mobile inference; and (b) the BCR pruning optimizations for determining pruning hyperparameters and performing weight pruning. We compare GRIM with Alibaba MNN, TVM, TensorFlow-Lite, a sparse implementation based on CSR, PatDNN, and ESE (a representative FPGA inference acceleration framework for RNNs), and achieve up to 14.08\times 14.08× speedup.
ISSN:0162-8828
2160-9292
1939-3539
DOI:10.1109/TPAMI.2021.3089687