CURIOUS: Efficient Neural Architecture Search Based on a Performance Predictor and Evolutionary Search

• Published in: IEEE transactions on computer-aided design of integrated circuits and systems 2022-11, Vol.41 (11), p.4975-4990
• Main Authors: Hassantabar, Shayan, Dai, Xiaoliang, Jha, Niraj K.
• Format: Article
• Language: English
• Subjects: Accuracy; Architecture; Artificial intelligence; Artificial neural networks; Compression; Computational modeling; Computer architecture; convolutional neural network (CNN); Data mining; Datasets; Decision trees; deep learning; dimensionality reduction; DNA; Errors; Floating point arithmetic; Iterative methods; long short-term memory (LSTM); Mathematical models; neural architecture search (NAS); Neural networks; Parameters; Performance evaluation; Performance prediction; Regularization; Sampling methods; Search process; Task analysis; Training; transformer; Transformers
• ISSN: 0278-0070; 1937-4151
• DOI: 10.1109/TCAD.2022.3148202

Neural networks (NNs) have been successfully deployed in various applications of artificial intelligence. However, architectural design of NNs is still a challenging problem. This is due to the need to navigate a search space based on a large number of hyperparameters. This forces the search space of possible architectures to grow exponentially. Using a trial-and-error design approach is very time consuming and leads to suboptimal architectures. In addition, approaches, such as neural architecture search based on reinforcement learning and differentiable gradient-based architecture search, often incur huge computational costs or significant memory requirements. To address these challenges, we propose the CURIOUS NN synthesis methodology. It uses a performance predictor to efficiently navigate the architectural search space with an evolutionary search process. The predictor is built using quasi Monte-Carlo sampling, boosted decision tree regression, and an intelligent iterative sampling method. It is designed to be sample efficient. CURIOUS starts from a base architecture and explores the architectural search space to obtain a variant of the base architecture with the highest performance. This search framework is general and covers all important NN architecture types, e.g., feedforward NNs (FFNNs), convolutional NNs (CNNs), recurrent NNs (RNNs), and transformers. We evaluate the performance of CURIOUS on various datasets and base architectures. Through these experiments, we demonstrate significant performance improvements over the baseline architectures. For the MNIST dataset, our CNN architecture achieves an error rate of 0.66%, with 8.6\times fewer parameters compared to the LeNet-5 baseline. For the CIFAR-10 dataset, we use the ResNet architectures and residual networks with Shake-Shake regularization as the baselines. Our synthesized ResNet-18 has a 2.52% accuracy improvement over the original ResNet-18, 1.74% over ResNet-101, and 0.16% over ResNet-1001, while requiring comparable number of parameters and floating-point operations to the original ResNet-18. This result shows that instead of just increasing the number of layers to increase accuracy, an alternative is to use a better NN architecture with a small number of layers. In addition, CURIOUS achieves an error rate of just 2.69% with a variant of the residual architecture with Shake-Shake regularization. We also use the set o