Dealing with categorical and integer-valued variables in Bayesian Optimization with Gaussian processes

• Published in: Neurocomputing (Amsterdam) 2020-03, Vol.380, p.20-35
• Main Authors: Garrido-Merchán, Eduardo C., Hernández-Lobato, Daniel
• Format: Article
• Language: English
• Subjects: Bayesian optimization; Categorical variables; Gaussian processes; Integer-valued variables; Parameter tuning
• ISSN: 0925-2312; 1872-8286
• DOI: 10.1016/j.neucom.2019.11.004

•Bayesian Optimization can be applied to optimization problems with categorical and integer-valued variables.•Existing methods relying on Gaussian processes may get stuck in such a setting.•A transformation of the covariance function is proposed to deal with categorical and integer-valued variables.•Empirical results show that the proposed method outperforms other approaches for Bayesian optimization. Some optimization problems are characterized by an objective that is very expensive, that lacks an analytical expression, and whose evaluations can be contaminated by noise. Bayesian Optimization (BO) methods can be used to solve these problems efficiently. BO relies on a probabilistic model of the objective, which is typically a Gaussian process (GP). This model is used to compute an acquisition function that estimates the expected utility (for solving the optimization problem) of evaluating the objective at each potential new point. A problem with GPs is, however, that they assume real-valued input variables and cannot easily deal with categorical or integer-valued values. Common methods to account for these variables, before evaluating the objective, include assuming they are real and then using a one-hot encoding, for categorical variables, or rounding to the closest integer, for integer-valued variables. We show that this leads to suboptimal results and introduce a novel approach to tackle categorical or integer-valued input variables within the context of BO with GPs. Several synthetic and real-world experiments support our hypotheses and show that our approach outperforms the results of standard BO using GPs on problems with categorical or integer-valued input variables.