Learning Hierarchical Acquisition Functions for Bayesian Optimization

A common challenge in engineering design applications of Bayesian optimization is the presence of non-computable domains, which are often called hidden or unknown constraints. In these cases, the required function values cannot be computed in certain regions of the design space. In this work, we propose a novel method to handle hidden constraints by modifying a portion of the acquisition function of a typical Bayesian optimization algorithm using classifiers. Specifically, the proposed approach considers the exploitation part of the acquisition function, allowing the optimization algorithm to search unexplored regions of the design space. Convergence properties are maintained regardless of the type of classifier used. Results obtained for numerical experiments on analytical test problems confirm the usefulness of the proposed method of handling hidden constraints in the context of Bayesian optimization.

This paper incorporates Bayesian estimation and optimization into a portfolio selection framework, particularly for high-dimensional portfolios in which the number of assets is larger than the number of observations. We leverage a constrained minimization approach, called the linear programming optimal (LPO) portfolio, to directly estimate effective parameters appearing in the optimal portfolio. We propose two refinements for the LPO strategy. First, we explore improved Bayesian estimates, instead of sample estimates, of the covariance matrix of asset returns. Second, we introduce Bayesian optimization (BO) to replace traditional grid-search cross-validation (CV) in tuning hyperparameters of the LPO strategy. We further propose modifications in the BO algorithm by (1) taking into account the time-dependent nature of financial problems and (2) extending the commonly used expected improvement acquisition function to include a tunable trade-off with the improvement’s variance. Allowing a general case of noisy observations, we theoretically derive the sublinear convergence rate of BO under the newly proposed EIVar and thus our algorithm has no regret. Our empirical studies confirm that the adjusted BO results in portfolios with higher out-of-sample Sharpe ratio, certainty equivalent, and lower turnover compared to those tuned with CV. This superior performance is achieved with a significant reduction in time elapsed, thus also addressing time-consuming issues of CV. Furthermore, LPO with Bayesian estimates outperforms the original proposal of LPO, as well as the benchmark equally weighted and plugin strategies.

pBO-2GP-3B: A batch parallel known/unknown constrained Bayesian optimization with feasibility classification and its applications in computational fluid dynamics

ABSTRACTBayesian optimization is a popular and versatile approach that is well suited to solve challenging optimization problems. Their popularity comes from their effective minimization of expensive function evaluations, their capability to leverage gradients, and their efficient use of noisy data. Bayesian optimizers have commonly been applied to global unconstrained problems, with limited development for many other classes of problems. In this article, two alternative methods are developed that enable rapid and deep convergence of nonlinearly‐constrained local optimization problems using a Bayesian optimizer. The first method uses an exact augmented Lagrangian and the second augments the minimization of the acquisition function to contain additional constraints. Both of these methods can be applied to nonlinear equality constraints, unlike most previous methods developed for constrained Bayesian optimizers. The new methods are applied with a gradient‐enhanced Bayesian optimizer and enable deeper convergence for three nonlinearly‐constrained unimodal optimization problems than previously developed methods for constrained Bayesian optimization. In addition, both new methods enable the Bayesian optimizer to reach a desired tolerance with fewer function evaluations than popular quasi‐Newton optimizers from SciPy and MATLAB for unimodal problems with 2 to 30 variables. The Bayesian optimizer had similar results using both methods. It is recommended that users first try using the second method, which adds constraints to the acquisition function minimization, since its parameters are more intuitive to tune for new problems.

This paper proposes a Sequential Model Based Optimization framework for solving optimization problems characterized by a black-box, multi-extremal, expensive and partially defined objective function, under unknown constraints. This is a typical setting for simulation-optimization problems, where the objective function cannot be computed for some configurations of the decision/control variables due to the violation of some (unknown) constraint. The framework is organized in two consecutive phases, the first uses a Support Vector Machine classifier to approximate the boundary of the unknown feasible region within the search space, the second uses Bayesian Optimization to find a globally optimal (feasible) solution. A relevant difference with traditional Bayesian Optimization is that the optimization process is performed on the estimated feasibility region, only, instead of the entire search space. Some results on three 2D test functions and a real case study for the Pump Scheduling Optimization in Water Distribution Networks are reported. The proposed framework proved to be more effective and efficient than Bayesian Optimization approaches using a penalty for function evaluations outside the feasible region.

Rather than optimizing expensive objective functions such as complex engineering simulations directly, Bayesian optimization methodologies fit a surrogate model (typically Kriging or a Gaussian Process) on evaluations of the objective function(s). To determine the next evaluation, an acquisition function is optimized (also referred to as infill criterion or sampling policy) which incorporates the model prediction and uncertainty and balances exploration and exploitation. Therefore, Bayesian optimization methodologies replace a single optimization of the objective function by a sequence of optimization problems: this makes sense as the acquisition function is cheap-to-evaluate whereas the objective is not. Depending on the goal different acquisition functions are available: multi-objective acquisition functions are relatively new and this chapter gives a state-of-the-art overview and illustrates some approaches based on hypervolume improvement. It is shown that the quality of the model is crucial for the performance of Bayesian optimization and illustrate this by using the more flexible Student-t processes as surrogate models.

Materials exploration requires the optimization of a multidimensional space including the chemical composition and synthesis parameters such as temperature and pressure. Bayesian optimization has attracted attention as a method for efficient multidimensional optimization. Appropriate choices of the acquisition function and initial values of the hyperparameters of the kernel functions are essential for the Bayesian optimization of synthesis conditions in a small number of experiments. However, to date, there has been little discussion on how to tune Bayesian optimization for materials exploration, and no guidelines have been provided for materials scientists. In this study, we investigated the optimum initial values of the hyperparameters in Bayesian optimization using one-dimensional model functions that mimic actual materials syntheses. The optimal lengthscale and variance for different process windows of materials synthesis were investigated. It was shown that the use of an appropriate acquisition function and suitable initial values of the hyperparameters of the kernel functions enable the optimization of synthesis conditions in a small number of trials. These results provide insight for enabling fully automated and autonomous materials synthesis using Bayesian optimization and robotics for materials exploration.

Bayesian Optimization (BO) is a framework for black-box optimization that is especially suitable for expensive cost functions. Among the main parts of a BO algorithm, the acquisition function is of fundamental importance, since it guides the optimization algorithm by translating the uncertainty of the regression model in a utility measure for each point to be evaluated. Considering such aspect, selection and design of acquisition functions are one of the most popular research topics in BO. Since no single acquisition function was proved to have better performance in all tasks, a well-established approach consists of selecting different acquisition functions along the iterations of a BO execution. In such an approach, the GP-Hedge algorithm is a widely used option given its simplicity and good performance. Despite its success in various applications, GP-Hedge shows an undesirable characteristic of accounting on all past performance measures of each acquisition function to select the next function to be used. In this case, good or bad values obtained in an initial iteration may impact the choice of the acquisition function for the rest of the algorithm. This fact may induce a dominant behavior of an acquisition function and impact the final performance of the method. Aiming to overcome such limitation, in this work we propose a variant of GP-Hedge, named No-PASt-BO, that reduce the influence of far past evaluations. Moreover, our method presents a built-in normalization that avoids the functions in the portfolio to have similar probabilities, thus improving the exploration. The obtained results on both synthetic and real-world optimization tasks indicate that No-PASt-BO presents competitive performance and always outperforms GP-Hedge.

Bayesian Optimization for Field-Scale Geological Carbon Storage

Optimizing multiple, non-preferential objectives for mixed-variable, expensive black-box problems is important in many areas of engineering and science. The expensive, noisy, black-box nature of these problems makes them ideal candidates for Bayesian optimization (BO). Mixed-variable and multi-objective problems, however, are a challenge due to BO's underlying smooth Gaussian process surrogate model. Current multi-objective BO algorithms cannot deal with mixed-variable problems. We present MixMOBO, the first mixed-variable, multi-objective Bayesian optimization framework for such problems. Using MixMOBO, optimal Pareto-fronts for multi-objective, mixed-variable design spaces can be found efficiently while ensuring diverse solutions. The method is sufficiently flexible to incorporate different kernels and acquisition functions, including those that were developed for mixed-variable or multi-objective problems by other authors. We also present HedgeMO, a modified Hedge strategy that uses a portfolio of acquisition functions for multi-objective problems. We present a new acquisition function, SMC. Our results show that MixMOBO performs well against other mixed-variable algorithms on synthetic problems. We apply MixMOBO to the real-world design of an architected material and show that our optimal design, which was experimentally fabricated and validated, has a normalized strain energy density $10^4$ times greater than existing structures.

Driven by increased integration of renewable energy sources, the widespread decarbonization of power systems has led to energy price fluctuations that require greater adaptability and flexibility from grid users in order to maximize profits. Industrial loads equipped with flexible resources can optimize energy consumption rather than merely reacting to immediate events, thereby capitalizing on volatile energy prices. However, the absence of sufficient measured data in industrial processes limits the ability to fully harness this flexibility. To address this challenge, we presents a black-box optimization model for optimizing the energy consumption of cooling systems in the aquaculture industry using Extreme Gradient Boosting (XGBoost) and Bayesian Optimization (BO). XGBoost is employed to establish a nonlinear relationship between cooling system power consumption and available measured data. Based on this model, Bayesian Optimization with the Lower Confidence Bound (LCB) acquisition function is used to determine the optimal discharge temperature of water into breeding pools, minimizing day-ahead electricity costs. The proposed approach is validated using real-world data from a case study at the Port of Hirtshals, Denmark based on measurements from 2023. Our findings illustrate that leveraging the inherent flexibility of industrial processes can yield financial benefits while providing valuable signals for grid operators to adjust consumption behaviors through appropriate price mechanisms. Furthermore, machine learning techniques prove effective in optimizing energy consumption for industries with limited measured data, delivering accurate and practical estimations.

Bayesian optimization seeks the global optimum of a black-box, objective function f (x), in the fewest possible iterations. Recent work applied knowledge of the true value of the optimum to the Gaussian Process probabilistic model typically used in Bayesian optimization. This, together with a new acquisition function called Confidence Bound Minimization, resulted in a Gaussian probabilistic posterior in which the predictions were no greater than the known maximum (and no less than for minimum). Our novel work applies Confidence Bound Minimization to Bayesian optimization with Student's-t Processes, a probabilistic alternative which addresses known weaknesses in Gaussian Processes - outliers' probability and the calculation of posterior covariance. The new model is applied to the problem of hyperparameter tuning for an XGBoost classifier. Experiments show superior regret minimization and predictive accuracy, versus the popular Expected Improvement acquisition function. Combining Confidence Bound Minimization with a transformed Student's-t Process probabilistic model and known optima produces superior training regret minimization and posterior predictions for the Six-Hump Camel(2D) and Levy(4D) benchmark problems, which do not fall below true minima.

Gaussian Process based Bayesian Optimization is largely adopted for solving problems where the inputs are in Euclidean spaces. In this paper we associate the inputs to discrete probability distributions which are elements of the probability simplex. To search in the new design space, we need a distance between distributions. The optimal transport distance (aka Wasserstein distance) is chosen due to its mathematical structure and the computational strategies enabled by it. Both the GP and the acquisition function is generalized to an acquisition functional over the probability simplex. To optimize this functional two methods are proposed, one based on auto differentiation and the other based on proximal-point algorithm and the gradient flow. Finally, we report a preliminary set of computational results on a class of problems whose dimension ranges from 5 to 100. These results show that embedding the Bayesian optimization process in the probability simplex enables an effective algorithm whose performance over standard Bayesian optimization improves with the increase of problem dimensionality.

Conventional molecular geometry searches on a potential energy surface (PES) utilize energy gradients from quantum chemical calculations. However, replacing energy calculations with noisy quantum computer measurements generates errors in the energies, which makes geometry optimization using the energy gradient difficult. One gradient-free optimization method that can potentially solve this problem is Bayesian optimization (BO). To use BO in geometry search, an acquisition function (AF), which involves an objective variable, must be defined suitably. In this study, we propose a strategy for geometry searches using BO and examine the appropriate AFs to explore two critical structures: the global minimum (GM) on the singlet ground state (S0) and the most stable conical intersection (CI) point between S0 and the singlet excited state. We applied our strategy to two molecules and located the GM and the most stable CI geometries with high accuracy for both molecules. We also succeeded in the geometry searches even when artificial random noises were added to the energies to simulate geometry optimization using noisy quantum computer measurements.

Bayesian optimization (BO) based on the Gaussian process (GP) surrogate model has attracted extensive attention in the field of optimization and design of experiments (DoE). It usually faces two problems: the unstable GP prediction due to the ill-conditioned Gram matrix of the kernel and the difficulty of determining the trade-off parameter between exploitation and exploration. To solve these problems, we investigate the K-optimality, aiming at minimizing the condition number. Firstly, the Sequentially Bayesian K-optimal design (SBKO) is proposed to ensure the stability of the GP prediction, where the K-optimality is given as the acquisition function. We show that the SBKO reduces the integrated posterior variance and maximizes the hyper-parameters’ information gain simultaneously. Secondly, a K-optimal enhanced Bayesian Optimization (KO-BO) approach is given for the optimization problems, where the K-optimality is used to define the trade-off balance parameters which can be output automatically. Specifically, we focus our study on the K-optimal enhanced Expected Improvement algorithm (KO-EI). Numerical examples show that the SBKO generally outperforms the Monte Carlo, Latin hypercube sampling, and sequential DoE approaches by maximizing the posterior variance with the highest precision of prediction. Furthermore, the study of the optimization problem shows that the KO-EI method beats the classical EI method due to its higher convergence rate and smaller variance.