Entropic Score metric: Decoupling Topology and Size in Training-free NAS

Neural Networks design is a complex and often daunting task, particularly for resource-constrained scenarios typical of mobile-sized models. Neural Architecture Search is a promising approach to automate this process, but existing competitive methods require large training time and computational res...

Full description

Saved in:
Bibliographic Details
Main Authors: Cavagnero, Niccolo, Robbiano, Luca, Pistilli, Francesca, Caputo, Barbara, Averta, Giuseppe
Format: Conference Proceeding
Language:English
Subjects:
Computational modeling
Computer architecture
Computer Vision
Efficient Neural Networks
Graphics processing units
Image Classification
Measurement
NAS
Network topology
Neural Architecture Search
Neural networks
Training
Online Access:Request full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Neural Networks design is a complex and often daunting task, particularly for resource-constrained scenarios typical of mobile-sized models. Neural Architecture Search is a promising approach to automate this process, but existing competitive methods require large training time and computational resources to generate accurate models. To overcome these limits, this paper contributes with: i) a novel training-free metric, named Entropic Score, to estimate model expressivity through the aggregated element-wise entropy of its activations; ii) a cyclic search algorithm to separately yet synergistically search model size and topology. Entropic Score shows remarkable ability in searching for the topology of the network, and a proper combination with LogSynflow, to search for model size, yields superior capability to completely design high-performance Hybrid Transformers for edge applications in less than 1 GPU hour, resulting in the fastest and most accurate NAS method for ImageNet classification. Code available here 1 .
ISSN:2473-9944
DOI:10.1109/ICCVW60793.2023.00158