No-PASt-BO: Normalized Portfolio Allocation Strategy for Bayesian Optimization

Rather than optimizing expensive objective functions such as complex engineering simulations directly, Bayesian optimization methodologies fit a surrogate model (typically Kriging or a Gaussian Process) on evaluations of the objective function(s). To determine the next evaluation, an acquisition function is optimized (also referred to as infill criterion or sampling policy) which incorporates the model prediction and uncertainty and balances exploration and exploitation. Therefore, Bayesian optimization methodologies replace a single optimization of the objective function by a sequence of optimization problems: this makes sense as the acquisition function is cheap-to-evaluate whereas the objective is not. Depending on the goal different acquisition functions are available: multi-objective acquisition functions are relatively new and this chapter gives a state-of-the-art overview and illustrates some approaches based on hypervolume improvement. It is shown that the quality of the model is crucial for the performance of Bayesian optimization and illustrate this by using the more flexible Student-t processes as surrogate models.

ABSTRACTBayesian optimization is a popular and versatile approach that is well suited to solve challenging optimization problems. Their popularity comes from their effective minimization of expensive function evaluations, their capability to leverage gradients, and their efficient use of noisy data. Bayesian optimizers have commonly been applied to global unconstrained problems, with limited development for many other classes of problems. In this article, two alternative methods are developed that enable rapid and deep convergence of nonlinearly‐constrained local optimization problems using a Bayesian optimizer. The first method uses an exact augmented Lagrangian and the second augments the minimization of the acquisition function to contain additional constraints. Both of these methods can be applied to nonlinear equality constraints, unlike most previous methods developed for constrained Bayesian optimizers. The new methods are applied with a gradient‐enhanced Bayesian optimizer and enable deeper convergence for three nonlinearly‐constrained unimodal optimization problems than previously developed methods for constrained Bayesian optimization. In addition, both new methods enable the Bayesian optimizer to reach a desired tolerance with fewer function evaluations than popular quasi‐Newton optimizers from SciPy and MATLAB for unimodal problems with 2 to 30 variables. The Bayesian optimizer had similar results using both methods. It is recommended that users first try using the second method, which adds constraints to the acquisition function minimization, since its parameters are more intuitive to tune for new problems.

Both evolutionary Bayesian optimization (EBO) and Bayesian evolutionary optimization (BEO) are used to solve time-consuming optimization problems, where in the EBO method, the evolutionary algorithm is adopted to optimize the acquisition function for finding its optimal solution to be evaluated using the time consuming objective functions, while in the BEO approach, surrogates are utilized to replace the expensive objective functions and the acquisition function is used in the model management to choose individuals for evaluations using the expensive objective functions. In this paper, we perform comparisons on these two approaches to see which method is better for solving computationally expensive multi-objective problems. The expected improvement (EI) is used as the acquisition function in both EBO and BEO approaches. Two strategies are proposed to select individuals to be evaluated using the real computationally expensive objectives. The experimental results conducted on ZDT and DTLZ test problems showed that the BEO approach is more efficient than the EBO approach for solving multi-objective problems.

pBO-2GP-3B: A batch parallel known/unknown constrained Bayesian optimization with feasibility classification and its applications in computational fluid dynamics

Optimizing multiple, non-preferential objectives for mixed-variable, expensive black-box problems is important in many areas of engineering and science. The expensive, noisy, black-box nature of these problems makes them ideal candidates for Bayesian optimization (BO). Mixed-variable and multi-objective problems, however, are a challenge due to BO's underlying smooth Gaussian process surrogate model. Current multi-objective BO algorithms cannot deal with mixed-variable problems. We present MixMOBO, the first mixed-variable, multi-objective Bayesian optimization framework for such problems. Using MixMOBO, optimal Pareto-fronts for multi-objective, mixed-variable design spaces can be found efficiently while ensuring diverse solutions. The method is sufficiently flexible to incorporate different kernels and acquisition functions, including those that were developed for mixed-variable or multi-objective problems by other authors. We also present HedgeMO, a modified Hedge strategy that uses a portfolio of acquisition functions for multi-objective problems. We present a new acquisition function, SMC. Our results show that MixMOBO performs well against other mixed-variable algorithms on synthetic problems. We apply MixMOBO to the real-world design of an architected material and show that our optimal design, which was experimentally fabricated and validated, has a normalized strain energy density $10^4$ times greater than existing structures.

This paper incorporates Bayesian estimation and optimization into a portfolio selection framework, particularly for high-dimensional portfolios in which the number of assets is larger than the number of observations. We leverage a constrained minimization approach, called the linear programming optimal (LPO) portfolio, to directly estimate effective parameters appearing in the optimal portfolio. We propose two refinements for the LPO strategy. First, we explore improved Bayesian estimates, instead of sample estimates, of the covariance matrix of asset returns. Second, we introduce Bayesian optimization (BO) to replace traditional grid-search cross-validation (CV) in tuning hyperparameters of the LPO strategy. We further propose modifications in the BO algorithm by (1) taking into account the time-dependent nature of financial problems and (2) extending the commonly used expected improvement acquisition function to include a tunable trade-off with the improvement’s variance. Allowing a general case of noisy observations, we theoretically derive the sublinear convergence rate of BO under the newly proposed EIVar and thus our algorithm has no regret. Our empirical studies confirm that the adjusted BO results in portfolios with higher out-of-sample Sharpe ratio, certainty equivalent, and lower turnover compared to those tuned with CV. This superior performance is achieved with a significant reduction in time elapsed, thus also addressing time-consuming issues of CV. Furthermore, LPO with Bayesian estimates outperforms the original proposal of LPO, as well as the benchmark equally weighted and plugin strategies.

We propose an extrinsic Bayesian optimization (eBO) framework for general optimization problems on manifolds. Bayesian optimization algorithms build a surrogate of the objective function by employing Gaussian processes and utilizing the uncertainty in that surrogate by deriving an acquisition function. This acquisition function represents the probability of improvement based on the kernel of the Gaussian process, which guides the search in the optimization process. The critical challenge for designing Bayesian optimization algorithms on manifolds lies in the difficulty of constructing valid covariance kernels for Gaussian processes on general manifolds. Our approach is to employ extrinsic Gaussian processes by first embedding the manifold onto some higher dimensional Euclidean space via equivariant embeddings and then constructing a valid covariance kernel on the image manifold after the embedding. This leads to efficient and scalable algorithms for optimization over complex manifolds. Simulation study and real data analyses are carried out to demonstrate the utilities of our eBO framework by applying the eBO to various optimization problems over manifolds such as the sphere, the Grassmannian, and the manifold of positive definite matrices.

Materials exploration requires the optimization of a multidimensional space including the chemical composition and synthesis parameters such as temperature and pressure. Bayesian optimization has attracted attention as a method for efficient multidimensional optimization. Appropriate choices of the acquisition function and initial values of the hyperparameters of the kernel functions are essential for the Bayesian optimization of synthesis conditions in a small number of experiments. However, to date, there has been little discussion on how to tune Bayesian optimization for materials exploration, and no guidelines have been provided for materials scientists. In this study, we investigated the optimum initial values of the hyperparameters in Bayesian optimization using one-dimensional model functions that mimic actual materials syntheses. The optimal lengthscale and variance for different process windows of materials synthesis were investigated. It was shown that the use of an appropriate acquisition function and suitable initial values of the hyperparameters of the kernel functions enable the optimization of synthesis conditions in a small number of trials. These results provide insight for enabling fully automated and autonomous materials synthesis using Bayesian optimization and robotics for materials exploration.

Bayesian Optimization for Field-Scale Geological Carbon Storage

Engineering design optimization problems are commonly multi-objective, involve computationally expensive simulations, and have uncertainty. In these problems, consideration of uncertainty can produce feasibly robust optimized design solutions, i.e., optimized and feasible under uncertainty. In general, methods for solving multi-objective robust optimization problems can be computationally expensive. To ease the computational burden and make these methods more accessible, a surrogate-based bi-objective Bayesian feasibility robust optimization method from the literature has been extended in this paper to higher numbers of objectives. A key component of the proposed method is a new acquisition function. The acquisition function proposed in this paper uses a Monte Carlo approach to approximate the expected centroidal improvement of the solution set. The proposed acquisition function is compared against exact (closed form) acquisition function-based approaches and other methods from the literature. The experimental results from this study suggest the multi-objective robust optimization framework is insensitive to the selection of acquisition function.

Driven by increased integration of renewable energy sources, the widespread decarbonization of power systems has led to energy price fluctuations that require greater adaptability and flexibility from grid users in order to maximize profits. Industrial loads equipped with flexible resources can optimize energy consumption rather than merely reacting to immediate events, thereby capitalizing on volatile energy prices. However, the absence of sufficient measured data in industrial processes limits the ability to fully harness this flexibility. To address this challenge, we presents a black-box optimization model for optimizing the energy consumption of cooling systems in the aquaculture industry using Extreme Gradient Boosting (XGBoost) and Bayesian Optimization (BO). XGBoost is employed to establish a nonlinear relationship between cooling system power consumption and available measured data. Based on this model, Bayesian Optimization with the Lower Confidence Bound (LCB) acquisition function is used to determine the optimal discharge temperature of water into breeding pools, minimizing day-ahead electricity costs. The proposed approach is validated using real-world data from a case study at the Port of Hirtshals, Denmark based on measurements from 2023. Our findings illustrate that leveraging the inherent flexibility of industrial processes can yield financial benefits while providing valuable signals for grid operators to adjust consumption behaviors through appropriate price mechanisms. Furthermore, machine learning techniques prove effective in optimizing energy consumption for industries with limited measured data, delivering accurate and practical estimations.

Bayesian optimization seeks the global optimum of a black-box, objective function f (x), in the fewest possible iterations. Recent work applied knowledge of the true value of the optimum to the Gaussian Process probabilistic model typically used in Bayesian optimization. This, together with a new acquisition function called Confidence Bound Minimization, resulted in a Gaussian probabilistic posterior in which the predictions were no greater than the known maximum (and no less than for minimum). Our novel work applies Confidence Bound Minimization to Bayesian optimization with Student's-t Processes, a probabilistic alternative which addresses known weaknesses in Gaussian Processes - outliers' probability and the calculation of posterior covariance. The new model is applied to the problem of hyperparameter tuning for an XGBoost classifier. Experiments show superior regret minimization and predictive accuracy, versus the popular Expected Improvement acquisition function. Combining Confidence Bound Minimization with a transformed Student's-t Process probabilistic model and known optima produces superior training regret minimization and posterior predictions for the Six-Hump Camel(2D) and Levy(4D) benchmark problems, which do not fall below true minima.

Gaussian Process based Bayesian Optimization is largely adopted for solving problems where the inputs are in Euclidean spaces. In this paper we associate the inputs to discrete probability distributions which are elements of the probability simplex. To search in the new design space, we need a distance between distributions. The optimal transport distance (aka Wasserstein distance) is chosen due to its mathematical structure and the computational strategies enabled by it. Both the GP and the acquisition function is generalized to an acquisition functional over the probability simplex. To optimize this functional two methods are proposed, one based on auto differentiation and the other based on proximal-point algorithm and the gradient flow. Finally, we report a preliminary set of computational results on a class of problems whose dimension ranges from 5 to 100. These results show that embedding the Bayesian optimization process in the probability simplex enables an effective algorithm whose performance over standard Bayesian optimization improves with the increase of problem dimensionality.

Conventional molecular geometry searches on a potential energy surface (PES) utilize energy gradients from quantum chemical calculations. However, replacing energy calculations with noisy quantum computer measurements generates errors in the energies, which makes geometry optimization using the energy gradient difficult. One gradient-free optimization method that can potentially solve this problem is Bayesian optimization (BO). To use BO in geometry search, an acquisition function (AF), which involves an objective variable, must be defined suitably. In this study, we propose a strategy for geometry searches using BO and examine the appropriate AFs to explore two critical structures: the global minimum (GM) on the singlet ground state (S0) and the most stable conical intersection (CI) point between S0 and the singlet excited state. We applied our strategy to two molecules and located the GM and the most stable CI geometries with high accuracy for both molecules. We also succeeded in the geometry searches even when artificial random noises were added to the energies to simulate geometry optimization using noisy quantum computer measurements.

First Order Bayesian Optimization (FOBO) is a sample efficient sequential approach to find the global maxima of an expensive-to-evaluate black-box objective function by suitably querying for the function and its gradient evaluations. Such methods assume Gaussian process (GP) models for both, the function and its gradient, and use them to construct an acquisition function that identifies the next query point. In this paper, we propose a class of practical FOBO algorithms that efficiently utilizes the information from the gradient GP to identify potential query points with zero gradients. We construct a multi-level acquisition function where in the first step, we optimize a lower level acquisition function with multiple restarts to identify potential query points with zero gradient value. We then use the upper level acquisition function to rank these query points based on their function values to potentially identify the global maximum. As a final step, the potential point of maximum is chosen as the actual query point. We validate the performance of our proposed algorithms on several test functions and show that our algorithms outperform state-of-the-art FOBO algorithms. We also illustrate the application of our algorithms in finding optimal set of hyper-parameters in machine learning and in learning the optimal policy in reinforcement learning tasks.