OFA3: Automatic Selection of the Best Non-dominated Sub-networks for Ensembles

• Main Authors: Ito, Rafael Claro, Da Silva, Emely Pujolli, Von Zuben, Fernando J.
• Format: Conference Proceeding
• Language: English
• Subjects: Accuracy; Adaptive systems; Buildings; Costs; Neural networks; Source coding; Training
• ISSN: 2161-4407
• DOI: 10.1109/IJCNN60899.2024.10650685

Advancement of Neural Architecture Search (NAS) has the potential to significantly improve the efficiency and performance of machine learning systems, as well as enable the exploration of new architectures and applications across a wide range of fields. A promising direction for developing more scalable and adaptive neural network architectures is the Once-for-All (OFA), a NAS framework that decouples the training and the search stages, meaning that one super-network is trained once, and then multiple searches can be performed according to different deployment scenarios. More recently, the OFA 2 strategy improved the search stage of the OFA framework by taking advantage of the very low cost of sampling already trained sub-networks and by exploring the multi-objective nature of the problem: a set of non-dominated sub-networks are all obtained at once, with distinct trade-offs involving hardware constraints and accuracy. In this work, we propose OFA 3 , building high-performance ensembles by solving the problem of how to automatically select the optimal subset of the already obtained non-dominated sub-networks. Particularly when components of the ensemble can run in parallel, our results dominate any other configuration of the available sub-networks, taking accuracy and latency as the conflicting objectives. The source code is available at https://github.com/ito-rafael/once-for-all-3.