Bayesian optimization for mixed-variable, multi-objective problems

Expensive black-box problems are usually optimized by Bayesian Optimization (BO) since it can reduce evaluation costs via cheaper surrogates. The most popular model used in Bayesian Optimization is the Gaussian process (GP) whose posterior is based on a joint GP prior built by initial observations, so the posterior is also a Gaussian process. Observations are often not noise-free, so in most of these cases, a noisy transformation of the objective space is observed. Many single objective optimization algorithms have succeeded in extending efficient global optimization (EGO) to noisy circumstances, while ParEGO fails to consider noise. In order to deal with noisy expensive black-box problems, we extending ParEGO to noisy optimization according to adding a Gaussian noisy error while approximating the surrogate. We call it noisy-ParEGO and results of S-metric indicate that the algorithm works well on optimizing noisy expensive multiobjective black-box problems.

Both evolutionary Bayesian optimization (EBO) and Bayesian evolutionary optimization (BEO) are used to solve time-consuming optimization problems, where in the EBO method, the evolutionary algorithm is adopted to optimize the acquisition function for finding its optimal solution to be evaluated using the time consuming objective functions, while in the BEO approach, surrogates are utilized to replace the expensive objective functions and the acquisition function is used in the model management to choose individuals for evaluations using the expensive objective functions. In this paper, we perform comparisons on these two approaches to see which method is better for solving computationally expensive multi-objective problems. The expected improvement (EI) is used as the acquisition function in both EBO and BEO approaches. Two strategies are proposed to select individuals to be evaluated using the real computationally expensive objectives. The experimental results conducted on ZDT and DTLZ test problems showed that the BEO approach is more efficient than the EBO approach for solving multi-objective problems.

This study addresses a shuffled frog leaping algorithm for solving the multi-objective reactive power dispatch problem in a power system. Optimal Reactive Power Dispatch (ORPD) is formulated as a nonlinear, multi-modal and mixed-variable problem. The intended technique is based on the minimization of the real power loss, minimization of voltage deviation and maximization of the voltage stability margin. Generator voltages, capacitor banks and tap positions of tap changing transformers are used as optimization variables of this problem. A memetic meta-heuristic named as shuffled frog-leaping algorithm is intended to solve multi-objective optimal reactive power dispatch problems considering voltage stability margin and voltage deviation. The Shuffled Frog-Leaping Algorithm (SFLA) is a population-based cooperative search metaphor inspired by natural memetics. The algorithm contains elements of local search and global information exchange. The most important benefit of this algorithm is higher speed of convergence to a better solution. The intended method is applied to ORPD problem on IEEE 57 bus power systems and compared with two versions of differential evolutionary algorithm. The simulation results show the effectiveness of the intended method.

Engineering design optimization problems are commonly multi-objective, involve computationally expensive simulations, and have uncertainty. In these problems, consideration of uncertainty can produce feasibly robust optimized design solutions, i.e., optimized and feasible under uncertainty. In general, methods for solving multi-objective robust optimization problems can be computationally expensive. To ease the computational burden and make these methods more accessible, a surrogate-based bi-objective Bayesian feasibility robust optimization method from the literature has been extended in this paper to higher numbers of objectives. A key component of the proposed method is a new acquisition function. The acquisition function proposed in this paper uses a Monte Carlo approach to approximate the expected centroidal improvement of the solution set. The proposed acquisition function is compared against exact (closed form) acquisition function-based approaches and other methods from the literature. The experimental results from this study suggest the multi-objective robust optimization framework is insensitive to the selection of acquisition function.

ABSTRACTBayesian optimization is a popular and versatile approach that is well suited to solve challenging optimization problems. Their popularity comes from their effective minimization of expensive function evaluations, their capability to leverage gradients, and their efficient use of noisy data. Bayesian optimizers have commonly been applied to global unconstrained problems, with limited development for many other classes of problems. In this article, two alternative methods are developed that enable rapid and deep convergence of nonlinearly‐constrained local optimization problems using a Bayesian optimizer. The first method uses an exact augmented Lagrangian and the second augments the minimization of the acquisition function to contain additional constraints. Both of these methods can be applied to nonlinear equality constraints, unlike most previous methods developed for constrained Bayesian optimizers. The new methods are applied with a gradient‐enhanced Bayesian optimizer and enable deeper convergence for three nonlinearly‐constrained unimodal optimization problems than previously developed methods for constrained Bayesian optimization. In addition, both new methods enable the Bayesian optimizer to reach a desired tolerance with fewer function evaluations than popular quasi‐Newton optimizers from SciPy and MATLAB for unimodal problems with 2 to 30 variables. The Bayesian optimizer had similar results using both methods. It is recommended that users first try using the second method, which adds constraints to the acquisition function minimization, since its parameters are more intuitive to tune for new problems.

This paper incorporates Bayesian estimation and optimization into a portfolio selection framework, particularly for high-dimensional portfolios in which the number of assets is larger than the number of observations. We leverage a constrained minimization approach, called the linear programming optimal (LPO) portfolio, to directly estimate effective parameters appearing in the optimal portfolio. We propose two refinements for the LPO strategy. First, we explore improved Bayesian estimates, instead of sample estimates, of the covariance matrix of asset returns. Second, we introduce Bayesian optimization (BO) to replace traditional grid-search cross-validation (CV) in tuning hyperparameters of the LPO strategy. We further propose modifications in the BO algorithm by (1) taking into account the time-dependent nature of financial problems and (2) extending the commonly used expected improvement acquisition function to include a tunable trade-off with the improvement’s variance. Allowing a general case of noisy observations, we theoretically derive the sublinear convergence rate of BO under the newly proposed EIVar and thus our algorithm has no regret. Our empirical studies confirm that the adjusted BO results in portfolios with higher out-of-sample Sharpe ratio, certainty equivalent, and lower turnover compared to those tuned with CV. This superior performance is achieved with a significant reduction in time elapsed, thus also addressing time-consuming issues of CV. Furthermore, LPO with Bayesian estimates outperforms the original proposal of LPO, as well as the benchmark equally weighted and plugin strategies.

Bayesian Co-evolutionary Optimization based entropy search for high-dimensional many-objective optimization

In this paper we consider a particular class of nonlinear optimization problems involving both continuous and discrete variables. The distinguishing feature of this class of nonlinear mixed variable optimization problems is that the structure and the number of variables of the problem depend on the values of some discrete variables. In particular, we define a general algorithm model for the solution of this class of problems, that draws inspiration from the approach recently proposed by Audet and Dennis [SIAM J. Optim., 11 (2001), pp. 573--594], and is based on the strategy of combining in a suitable way a local search with respect to the continuous variables and a local search with respect to the discrete variables. We prove global convergence of the algorithm model without specifying the local continuous search, but only identifying some reasonable requirements. Moreover, we define a particular derivative-free algorithm for solving mixed variable programming problems where the continuous variables are linearly constrained and derivative information is not available. Finally, we report numerical results obtained by the proposed algorithm in solving a real optimal design problem. These results show the effectiveness of the approach.

Rather than optimizing expensive objective functions such as complex engineering simulations directly, Bayesian optimization methodologies fit a surrogate model (typically Kriging or a Gaussian Process) on evaluations of the objective function(s). To determine the next evaluation, an acquisition function is optimized (also referred to as infill criterion or sampling policy) which incorporates the model prediction and uncertainty and balances exploration and exploitation. Therefore, Bayesian optimization methodologies replace a single optimization of the objective function by a sequence of optimization problems: this makes sense as the acquisition function is cheap-to-evaluate whereas the objective is not. Depending on the goal different acquisition functions are available: multi-objective acquisition functions are relatively new and this chapter gives a state-of-the-art overview and illustrates some approaches based on hypervolume improvement. It is shown that the quality of the model is crucial for the performance of Bayesian optimization and illustrate this by using the more flexible Student-t processes as surrogate models.

Bayesian Optimization (BO) is a framework for black-box optimization that is especially suitable for expensive cost functions. Among the main parts of a BO algorithm, the acquisition function is of fundamental importance, since it guides the optimization algorithm by translating the uncertainty of the regression model in a utility measure for each point to be evaluated. Considering such aspect, selection and design of acquisition functions are one of the most popular research topics in BO. Since no single acquisition function was proved to have better performance in all tasks, a well-established approach consists of selecting different acquisition functions along the iterations of a BO execution. In such an approach, the GP-Hedge algorithm is a widely used option given its simplicity and good performance. Despite its success in various applications, GP-Hedge shows an undesirable characteristic of accounting on all past performance measures of each acquisition function to select the next function to be used. In this case, good or bad values obtained in an initial iteration may impact the choice of the acquisition function for the rest of the algorithm. This fact may induce a dominant behavior of an acquisition function and impact the final performance of the method. Aiming to overcome such limitation, in this work we propose a variant of GP-Hedge, named No-PASt-BO, that reduce the influence of far past evaluations. Moreover, our method presents a built-in normalization that avoids the functions in the portfolio to have similar probabilities, thus improving the exploration. The obtained results on both synthetic and real-world optimization tasks indicate that No-PASt-BO presents competitive performance and always outperforms GP-Hedge.

Materials exploration requires the optimization of a multidimensional space including the chemical composition and synthesis parameters such as temperature and pressure. Bayesian optimization has attracted attention as a method for efficient multidimensional optimization. Appropriate choices of the acquisition function and initial values of the hyperparameters of the kernel functions are essential for the Bayesian optimization of synthesis conditions in a small number of experiments. However, to date, there has been little discussion on how to tune Bayesian optimization for materials exploration, and no guidelines have been provided for materials scientists. In this study, we investigated the optimum initial values of the hyperparameters in Bayesian optimization using one-dimensional model functions that mimic actual materials syntheses. The optimal lengthscale and variance for different process windows of materials synthesis were investigated. It was shown that the use of an appropriate acquisition function and suitable initial values of the hyperparameters of the kernel functions enable the optimization of synthesis conditions in a small number of trials. These results provide insight for enabling fully automated and autonomous materials synthesis using Bayesian optimization and robotics for materials exploration.

Bayesian Optimization for Field-Scale Geological Carbon Storage

Game theory finds nowadays a broad range of applications in engineering and machine learning. However, in a derivative-free, expensive black-box context, very few algorithmic solutions are available to find game equilibria. Here, we propose a novel Gaussian-process based approach for solving games in this context. We follow a classical Bayesian optimization framework, with sequential sampling decisions based on acquisition functions. Two strategies are proposed, based either on the probability of achieving equilibrium or on the stepwise uncertainty reduction paradigm. Practical and numerical aspects are discussed in order to enhance the scalability and reduce computation time. Our approach is evaluated on several synthetic game problems with varying number of players and decision space dimensions. We show that equilibria can be found reliably for a fraction of the cost (in terms of black-box evaluations) compared to classical, derivative-based algorithms. The method is available in the R package GPGame available on CRAN at https://cran.r-project.org/package=GPGame .

Driven by increased integration of renewable energy sources, the widespread decarbonization of power systems has led to energy price fluctuations that require greater adaptability and flexibility from grid users in order to maximize profits. Industrial loads equipped with flexible resources can optimize energy consumption rather than merely reacting to immediate events, thereby capitalizing on volatile energy prices. However, the absence of sufficient measured data in industrial processes limits the ability to fully harness this flexibility. To address this challenge, we presents a black-box optimization model for optimizing the energy consumption of cooling systems in the aquaculture industry using Extreme Gradient Boosting (XGBoost) and Bayesian Optimization (BO). XGBoost is employed to establish a nonlinear relationship between cooling system power consumption and available measured data. Based on this model, Bayesian Optimization with the Lower Confidence Bound (LCB) acquisition function is used to determine the optimal discharge temperature of water into breeding pools, minimizing day-ahead electricity costs. The proposed approach is validated using real-world data from a case study at the Port of Hirtshals, Denmark based on measurements from 2023. Our findings illustrate that leveraging the inherent flexibility of industrial processes can yield financial benefits while providing valuable signals for grid operators to adjust consumption behaviors through appropriate price mechanisms. Furthermore, machine learning techniques prove effective in optimizing energy consumption for industries with limited measured data, delivering accurate and practical estimations.

Bayesian optimization seeks the global optimum of a black-box, objective function f (x), in the fewest possible iterations. Recent work applied knowledge of the true value of the optimum to the Gaussian Process probabilistic model typically used in Bayesian optimization. This, together with a new acquisition function called Confidence Bound Minimization, resulted in a Gaussian probabilistic posterior in which the predictions were no greater than the known maximum (and no less than for minimum). Our novel work applies Confidence Bound Minimization to Bayesian optimization with Student's-t Processes, a probabilistic alternative which addresses known weaknesses in Gaussian Processes - outliers' probability and the calculation of posterior covariance. The new model is applied to the problem of hyperparameter tuning for an XGBoost classifier. Experiments show superior regret minimization and predictive accuracy, versus the popular Expected Improvement acquisition function. Combining Confidence Bound Minimization with a transformed Student's-t Process probabilistic model and known optima produces superior training regret minimization and posterior predictions for the Six-Hump Camel(2D) and Levy(4D) benchmark problems, which do not fall below true minima.