A systematic review of hyperparameter optimization techniques in Convolutional Neural Networks

• Published in: Decision analytics journal 2024-06, Vol.11, p.100470, Article 100470
• Main Authors: Raiaan, Mohaimenul Azam Khan, Sakib, Sadman, Fahad, Nur Mohammad, Mamun, Abdullah Al, Rahman, Md. Anisur, Shatabda, Swakkhar, Mukta, Md. Saddam Hossain
• Format: Article
• Language: English
• Subjects: Activation function; Convolutional Neural Network; Hyperparameter optimization Analysis; Optimizer
• ISSN: 2772-6622; 2772-6622
• DOI: 10.1016/j.dajour.2024.100470

Convolutional Neural Network (CNN) is a prevalent topic in deep learning (DL) research for their architectural advantages. CNN relies heavily on hyperparameter configurations, and manually tuning these hyperparameters can be time-consuming for researchers, therefore we need efficient optimization techniques. In this systematic review, we explore a range of well used algorithms, including metaheuristic, statistical, sequential, and numerical approaches, to fine-tune CNN hyperparameters. Our research offers an exhaustive categorization of these hyperparameter optimization (HPO) algorithms and investigates the fundamental concepts of CNN, explaining the role of hyperparameters and their variants. Furthermore, an exhaustive literature review of HPO algorithms in CNN employing the above mentioned algorithms is undertaken. A comparative analysis is conducted based on their HPO strategies, error evaluation approaches, and accuracy results across various datasets to assess the efficacy of these methods. In addition to addressing current challenges in HPO, our research illuminates unresolved issues in the field. By providing insightful evaluations of the merits and demerits of various HPO algorithms, our objective is to assist researchers in determining a suitable method for a particular problem and dataset. By highlighting future research directions and synthesizing diversified knowledge, our survey contributes significantly to the ongoing development of CNN hyperparameter optimization. •Present a comprehensive review of hyperparameters of Convolution Neural Networks.•Categorize ten hyperparameter optimization algorithms into four classes.•Examine the hyperparameter optimization algorithms by highlighting their strengths and weaknesses.•Assess the performance of hyperparameter optimization algorithms on benchmark datasets.•Identify four open issues and suggest future research paths for hyperparameter optimization techniques in convolutional neural networks.