Dual Dynamic Inference: Enabling More Efficient, Adaptive, and Controllable Deep Inference

State-of-the-art convolutional neural networks (CNNs) yield record-breaking predictive performance, yet at the cost of high-energy-consumption inference, that prohibits their widely deployments in resource-constrained Internet of Things (IoT) applications. We propose a dual dynamic inference (DDI) f...

Full description

Saved in:
Bibliographic Details
Published in:IEEE journal of selected topics in signal processing 2020-05, Vol.14 (4), p.623-633
Main Authors: Wang, Yue, Shen, Jianghao, Hu, Ting-Kuei, Xu, Pengfei, Nguyen, Tan, Baraniuk, Richard, Wang, Zhangyang, Lin, Yingyan
Format: Article
Language:English
Subjects:
Adaptation models
Artificial neural networks
Complexity theory
Computational modeling
Constraints
Cost control
Dynamic inference
Dynamic scheduling
Energy costs
Inference
input-dependent
Internet of Things
multi-grained
Network latency
Performance evaluation
Performance prediction
resource-dependent
Stability
Tradeoffs
Training
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:State-of-the-art convolutional neural networks (CNNs) yield record-breaking predictive performance, yet at the cost of high-energy-consumption inference, that prohibits their widely deployments in resource-constrained Internet of Things (IoT) applications. We propose a dual dynamic inference (DDI) framework that highlights the following aspects: 1) we integrate both input-dependent and resource-dependent dynamic inference mechanisms under a unified framework in order to fit the varying IoT resource requirements in practice. DDI is able to both constantly suppress unnecessary costs for easy samples, and to halt inference for all samples to meet hard resource constraints enforced; 2) we propose a flexible multi-grained learning to skip (MGL2S) approach for input-dependent inference which allows simultaneous layer-wise and channel-wise skipping; 3) we extend DDI to complex CNN backbones such as DenseNet and show that DDI can be applied towards optimizing any specific resource goals including inference latency and energy cost. Extensive experiments demonstrate the superior inference accuracy-resource trade-off achieved by DDI, as well as the flexibility to control such a trade-off as compared to existing peer methods. Specifically, DDI can achieve up to 4 times computational savings with the same or even higher accuracy as compared to existing competitive baselines.
ISSN:1932-4553
1941-0484
DOI:10.1109/JSTSP.2020.2979669