Resource Efficient Bayesian Optimization

• Main Authors: Juneja, Namit, Chandola, Varun, Zola, Jaroslaw, Wodo, Olga, Desai, Parth
• Format: Conference Proceeding
• Language: English
• Subjects: active learning; Bayes methods; Bayesian optimization; Cloud computing; Computational modeling; Costs; Expected Improvement; Gaussian processes; Machine learning; Optimization methods; Resource-efficient op-timization; Training
• ISSN: 2159-6190
• DOI: 10.1109/CLOUD62652.2024.00012

We propose a resource-efficient Bayesian Optimization (BO) formulation that can provide the same convergence guarantees as traditional BO, while ensuring that the opti-mization makes efficient use of the available cloud or high-performance computing (HPC) resources. The paper is motivated by the fact that for many optimization problems that lend themselves well to BO, like hyper-parameter optimization for training large machine learning models, the single function evaluation cost depends on the model parameters as well as system parameters. The proposed Resource Efficient Bayesian Optimization (REBO) algorithm is a novel formulation that exploits this dependence and provides significant cost benefits for users who want to deploy BO on cloud and HPC resources that are characterized by availability of compute resources with varying costs and expected performance benefits. We demonstrate the effectiveness of REBO, in terms of convergence and resource-efficiency, on a variety of machine learning hyper-parameter optimization applications.