Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Better

Deep learning has revolutionized the fields of computer vision, natural language understanding, speech recognition, information retrieval, and more. However, with the progressive improvements in deep learning models, their number of parameters, latency, and resources required to train, among others,...

Full description

Saved in:
Bibliographic Details
Published in:ACM computing surveys 2023-12, Vol.55 (12), p.1-37, Article 259
Main Author: Menghani, Gaurav
Format: Article
Language:English
Subjects:
Artificial intelligence
Computer science
Computer vision
Computing methodologies
Deep learning
Efficiency
Hardware
Information retrieval
Machine learning
Modelling
Natural language (computers)
Natural language processing
Speech recognition
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Deep learning has revolutionized the fields of computer vision, natural language understanding, speech recognition, information retrieval, and more. However, with the progressive improvements in deep learning models, their number of parameters, latency, and resources required to train, among others, have all increased significantly. Consequently, it has become important to pay attention to these footprint metrics of a model as well, not just its quality. We present and motivate the problem of efficiency in deep learning, followed by a thorough survey of the five core areas of model efficiency (spanning modeling techniques, infrastructure, and hardware) and the seminal work there. We also present an experiment-based guide along with code for practitioners to optimize their model training and deployment. We believe this is the first comprehensive survey in the efficient deep learning space that covers the landscape of model efficiency from modeling techniques to hardware support. It is our hope that this survey would provide readers with the mental model and the necessary understanding of the field to apply generic efficiency techniques to immediately get significant improvements, and also equip them with ideas for further research and experimentation to achieve additional gains.
ISSN:0360-0300
1557-7341
DOI:10.1145/3578938