Variational Reinforcement Learning for Hyper-Parameter Tuning of Adaptive Evolutionary Algorithm

• Published in: IEEE transactions on emerging topics in computational intelligence 2023-10, Vol.7 (5), p.1-16
• Main Authors: Zhang, Haotian, Sun, Jianyong, Wang, Yuhao, Shi, Jialong, Xu, Zongben
• Format: Article
• Language: English
• Subjects: Adaptive algorithms; evolutionary algorithm; Evolutionary algorithms; Evolutionary computation; expectation-maximization; Genetic algorithms; Machine learning; Machine learning algorithms; Optimization; parameter tuning; Parameters; Random variables; Reinforcement learning; Sociology; Statistics; Task analysis; Trajectory; Tuning; Variational inference
• ISSN: 2471-285X; 2471-285X
• DOI: 10.1109/TETCI.2022.3221483

The performance of an evolutionary algorithm (EA) is deeply affected by its control parameter's setting. It has become a trend in recent studies to treat the control parameter as a random variable. In these studies, the associated distribution of the control parameter is updated at each generation and new parameter setting is sampled from the distribution. The distribution's parameter (called hyper-parameter) is thus critical to the algorithmic performance. In this paper, we propose a variational learning framework to tune the hyper-parameters of EA, in which the expectation-maximization (EM) algorithm and a reinforcement learning algorithm are combined. To verify the effectiveness of the proposed method which is named Reinforcement EM (REM), we apply it to tune the hyper-parameters of the distributions of two important parameters, i.e. the scaling parameter (F) and crossover rate (CR), of differential evolution (DE) and an adaptive DE algorithm. In addition, we propose to use the meta-learning technique to learn good initial distributions for the hyper-parameters of F and CR so that the REM can effectively adapt to a new optimization problem. Experimental results obtained on the CEC 2018 test suite show that with the tuned hyper-parameters, DE and the adaptive DE can achieve significantly better performance than their counterparts with empirical parameter settings and with parameters tuned by some widely-used tuning methods, including ParamILS, F-Race and Bayesian optimization algorithm.