Multi-objective Bayesian Optimization for Engineering Simulation

Bayesian optimization seeks the global optimum of a black-box, objective function f (x), in the fewest possible iterations. Recent work applied knowledge of the true value of the optimum to the Gaussian Process probabilistic model typically used in Bayesian optimization. This, together with a new acquisition function called Confidence Bound Minimization, resulted in a Gaussian probabilistic posterior in which the predictions were no greater than the known maximum (and no less than for minimum). Our novel work applies Confidence Bound Minimization to Bayesian optimization with Student's-t Processes, a probabilistic alternative which addresses known weaknesses in Gaussian Processes - outliers' probability and the calculation of posterior covariance. The new model is applied to the problem of hyperparameter tuning for an XGBoost classifier. Experiments show superior regret minimization and predictive accuracy, versus the popular Expected Improvement acquisition function. Combining Confidence Bound Minimization with a transformed Student's-t Process probabilistic model and known optima produces superior training regret minimization and posterior predictions for the Six-Hump Camel(2D) and Levy(4D) benchmark problems, which do not fall below true minima.

Gaussian processes (Kriging) are interpolating data-driven models that are frequently applied in various disciplines. Often, Gaussian processes are trained on datasets and are subsequently embedded as surrogate models in optimization problems. These optimization problems are nonconvex and global optimization is desired. However, previous literature observed computational burdens limiting deterministic global optimization to Gaussian processes trained on few data points. We propose a reduced-space formulation for deterministic global optimization with trained Gaussian processes embedded. For optimization, the branch-and-bound solver branches only on the free variables and McCormick relaxations are propagated through explicit Gaussian process models. The approach also leads to significantly smaller and computationally cheaper subproblems for lower and upper bounding. To further accelerate convergence, we derive envelopes of common covariance functions for GPs and tight relaxations of acquisition functions used in Bayesian optimization including expected improvement, probability of improvement, and lower confidence bound. In total, we reduce computational time by orders of magnitude compared to state-of-the-art methods, thus overcoming previous computational burdens. We demonstrate the performance and scaling of the proposed method and apply it to Bayesian optimization with global optimization of the acquisition function and chance-constrained programming. The Gaussian process models, acquisition functions, and training scripts are available open-source within the “MeLOn—MachineLearning Models for Optimization” toolbox (https://git.rwth-aachen.de/avt.svt/public/MeLOn).

Bayesian optimization (BO) based on the Gaussian process (GP) surrogate model has attracted extensive attention in the field of optimization and design of experiments (DoE). It usually faces two problems: the unstable GP prediction due to the ill-conditioned Gram matrix of the kernel and the difficulty of determining the trade-off parameter between exploitation and exploration. To solve these problems, we investigate the K-optimality, aiming at minimizing the condition number. Firstly, the Sequentially Bayesian K-optimal design (SBKO) is proposed to ensure the stability of the GP prediction, where the K-optimality is given as the acquisition function. We show that the SBKO reduces the integrated posterior variance and maximizes the hyper-parameters’ information gain simultaneously. Secondly, a K-optimal enhanced Bayesian Optimization (KO-BO) approach is given for the optimization problems, where the K-optimality is used to define the trade-off balance parameters which can be output automatically. Specifically, we focus our study on the K-optimal enhanced Expected Improvement algorithm (KO-EI). Numerical examples show that the SBKO generally outperforms the Monte Carlo, Latin hypercube sampling, and sequential DoE approaches by maximizing the posterior variance with the highest precision of prediction. Furthermore, the study of the optimization problem shows that the KO-EI method beats the classical EI method due to its higher convergence rate and smaller variance.

ABSTRACTBayesian optimization is a popular and versatile approach that is well suited to solve challenging optimization problems. Their popularity comes from their effective minimization of expensive function evaluations, their capability to leverage gradients, and their efficient use of noisy data. Bayesian optimizers have commonly been applied to global unconstrained problems, with limited development for many other classes of problems. In this article, two alternative methods are developed that enable rapid and deep convergence of nonlinearly‐constrained local optimization problems using a Bayesian optimizer. The first method uses an exact augmented Lagrangian and the second augments the minimization of the acquisition function to contain additional constraints. Both of these methods can be applied to nonlinear equality constraints, unlike most previous methods developed for constrained Bayesian optimizers. The new methods are applied with a gradient‐enhanced Bayesian optimizer and enable deeper convergence for three nonlinearly‐constrained unimodal optimization problems than previously developed methods for constrained Bayesian optimization. In addition, both new methods enable the Bayesian optimizer to reach a desired tolerance with fewer function evaluations than popular quasi‐Newton optimizers from SciPy and MATLAB for unimodal problems with 2 to 30 variables. The Bayesian optimizer had similar results using both methods. It is recommended that users first try using the second method, which adds constraints to the acquisition function minimization, since its parameters are more intuitive to tune for new problems.

In multi-objective Bayesian optimization, an infill criterion is an important part, as it is the indicator to evaluate how much good a new set of solutions is, compared to a Pareto-front approximation set. This paper presents a deterministic algorithm for computing the Expected R2 Indicator for bi-objective problems and studies its use as an infill criterion in Bayesian Global Optimization. The R2-Indicator was introduced in 1998 by M. Hansen and A. Jaszkiewicz for performance assessment in multi-objective optimization and is more recently also used in indicator-based multi-criterion evolutionary algorithms (IBEAs). In Bayesian Global Optimization, we propose the Expected R2-indicator Improvement (ER2I) as an infill criterion. It is defined as the expected decrease of the R2 indicator by a point that is sampled from a predictive Gaussian distribution. The ER2I can also be used as a pre-selection criterion in surrogate-assisted IBEAs. It provides an alternative to the Expected Hypervolume-Indicator Improvement (EHVI) that requires a reference point, bounding the Pareto front from above. In contrast, the ER2I works with a utopian reference point that bounds the Pareto front from below. In addition, the ER2I supports preference modelling with utility functions and its computation time grows only linearly with the number of considered weight combinations. It is straightforward to approximate the ER2I by Monte Carlo Integration, but so far a deterministic algorithm to solve the non-linear integral remained unknown. We outline a deterministic algorithm for the computation of the bi-objective ER2I with Chebychev utility functions. Moreover, we study monotonicity properties of the ER2I w.r.t. parameters of the predictive distribution and numerical simulations demonstrate fast convergence to Pareto fronts of different shapes and the ability of the ER2I Bayesian optimization to fill gaps in the Pareto front approximation.

Abstarct Engineering design problems typically require optimizing a quality measure by finding the right combination of controllable input parameters. In Additive Manufacturing (AM), the output characteristics of the process can often be non-stationary functions of the process parameters. Bayesian Optimization (BO) is a methodology to optimize such “black-box” functions, i.e., the input–output relationship is unknown and expensive to compute. Optimization tasks involving “black-box” functions widely use BO with Gaussian Process (GP) regression surrogate model. Using GPs with standard kernels is insufficient for modeling non-stationary functions, while GPs with non-stationary kernels are typically over-parameterized. On the other hand, a Deep Gaussian Process (DGP) can overcome GPs’ shortcomings by considering a composition of multiple GPs. Inference in a DGP is challenging due to its structure resulting in a non-Gaussian posterior, and using DGP as a surrogate model for BO is not straightforward. Stochastic Imputation (SI)-based inference is promising in speed and accuracy for BO. This work proposes a bootstrap aggregation-based procedure to effectively utilize the SI-based inference for BO with a DGP surrogate model. The proposed BO algorithm DGP-SI-BO is faster and empirically better than the state-of-the-art BO method in optimizing non-stationary functions. Several analytical test functions and a case study in metal AM simulation demonstrate the applicability of the proposed method.

Both evolutionary Bayesian optimization (EBO) and Bayesian evolutionary optimization (BEO) are used to solve time-consuming optimization problems, where in the EBO method, the evolutionary algorithm is adopted to optimize the acquisition function for finding its optimal solution to be evaluated using the time consuming objective functions, while in the BEO approach, surrogates are utilized to replace the expensive objective functions and the acquisition function is used in the model management to choose individuals for evaluations using the expensive objective functions. In this paper, we perform comparisons on these two approaches to see which method is better for solving computationally expensive multi-objective problems. The expected improvement (EI) is used as the acquisition function in both EBO and BEO approaches. Two strategies are proposed to select individuals to be evaluated using the real computationally expensive objectives. The experimental results conducted on ZDT and DTLZ test problems showed that the BEO approach is more efficient than the EBO approach for solving multi-objective problems.

We propose an extrinsic Bayesian optimization (eBO) framework for general optimization problems on manifolds. Bayesian optimization algorithms build a surrogate of the objective function by employing Gaussian processes and utilizing the uncertainty in that surrogate by deriving an acquisition function. This acquisition function represents the probability of improvement based on the kernel of the Gaussian process, which guides the search in the optimization process. The critical challenge for designing Bayesian optimization algorithms on manifolds lies in the difficulty of constructing valid covariance kernels for Gaussian processes on general manifolds. Our approach is to employ extrinsic Gaussian processes by first embedding the manifold onto some higher dimensional Euclidean space via equivariant embeddings and then constructing a valid covariance kernel on the image manifold after the embedding. This leads to efficient and scalable algorithms for optimization over complex manifolds. Simulation study and real data analyses are carried out to demonstrate the utilities of our eBO framework by applying the eBO to various optimization problems over manifolds such as the sphere, the Grassmannian, and the manifold of positive definite matrices.

Source localization with a geoacoustic model requires optimizing the model over a parameter space of range and depth with the objective of matching a predicted sound field to a field measured on an array. We propose a sample-efficient sequential Bayesian optimization strategy that models the objective function as a Gaussian process (GP) surrogate model conditioned on observed data. Using the mean and covariance functions of the GP, a heuristic acquisition function proposes a candidate in parameter space to sample, balancing exploitation (sampling around the best observed objective function value) and exploration (sampling in regions of high variance in the GP). The candidate sample is evaluated, and the GP conditioned on the updated data. Optimization proceeds sequentially until a fixed budget of evaluations is expended. We demonstrate source localization for a shallow-water waveguide using Monte Carlo simulations and experimental data from an acoustic source tow. Compared to grid search and quasi-random sampling strategies, simulations and experimental results indicate the Bayesian optimization strategy converges on optimal solutions rapidly.

Bayesian Optimization for Field-Scale Geological Carbon Storage

Bayesian Optimization (BO) is a framework for black-box optimization that is especially suitable for expensive cost functions. Among the main parts of a BO algorithm, the acquisition function is of fundamental importance, since it guides the optimization algorithm by translating the uncertainty of the regression model in a utility measure for each point to be evaluated. Considering such aspect, selection and design of acquisition functions are one of the most popular research topics in BO. Since no single acquisition function was proved to have better performance in all tasks, a well-established approach consists of selecting different acquisition functions along the iterations of a BO execution. In such an approach, the GP-Hedge algorithm is a widely used option given its simplicity and good performance. Despite its success in various applications, GP-Hedge shows an undesirable characteristic of accounting on all past performance measures of each acquisition function to select the next function to be used. In this case, good or bad values obtained in an initial iteration may impact the choice of the acquisition function for the rest of the algorithm. This fact may induce a dominant behavior of an acquisition function and impact the final performance of the method. Aiming to overcome such limitation, in this work we propose a variant of GP-Hedge, named No-PASt-BO, that reduce the influence of far past evaluations. Moreover, our method presents a built-in normalization that avoids the functions in the portfolio to have similar probabilities, thus improving the exploration. The obtained results on both synthetic and real-world optimization tasks indicate that No-PASt-BO presents competitive performance and always outperforms GP-Hedge.

Information-based Bayesian optimization (BO) algorithms have achieved state-of-the-art performance in optimizing a black-box objective function. However, they usually require several approximations or simplifying assumptions (without clearly understanding their effects on the BO performance) and/or their generalization to batch BO is computationally unwieldy, especially with an increasing batch size. To alleviate these issues, this paper presents a novel trusted-maximizers entropy search (TES) acquisition function: It measures how much an input query contributes to the information gain on the maximizer over a finite set of trusted maximizers, i.e., inputs optimizing functions that are sampled from the Gaussian process posterior belief of the objective function. Evaluating TES requires either only a stochastic approximation with sampling or a deterministic approximation with expectation propagation, both of which are investigated and empirically evaluated using synthetic benchmark objective functions and real-world optimization problems, e.g., hyperparameter tuning of a convolutional neural network and synthesizing ‘physically realizable’ faces to fool a black-box face recognition system. Though TES can naturally be generalized to a batch variant with either approximation, the latter is amenable to be scaled to a much larger batch size in our experiments.

First Order Bayesian Optimization (FOBO) is a sample efficient sequential approach to find the global maxima of an expensive-to-evaluate black-box objective function by suitably querying for the function and its gradient evaluations. Such methods assume Gaussian process (GP) models for both, the function and its gradient, and use them to construct an acquisition function that identifies the next query point. In this paper, we propose a class of practical FOBO algorithms that efficiently utilizes the information from the gradient GP to identify potential query points with zero gradients. We construct a multi-level acquisition function where in the first step, we optimize a lower level acquisition function with multiple restarts to identify potential query points with zero gradient value. We then use the upper level acquisition function to rank these query points based on their function values to potentially identify the global maximum. As a final step, the potential point of maximum is chosen as the actual query point. We validate the performance of our proposed algorithms on several test functions and show that our algorithms outperform state-of-the-art FOBO algorithms. We also illustrate the application of our algorithms in finding optimal set of hyper-parameters in machine learning and in learning the optimal policy in reinforcement learning tasks.

Optimizing multiple, non-preferential objectives for mixed-variable, expensive black-box problems is important in many areas of engineering and science. The expensive, noisy, black-box nature of these problems makes them ideal candidates for Bayesian optimization (BO). Mixed-variable and multi-objective problems, however, are a challenge due to BO's underlying smooth Gaussian process surrogate model. Current multi-objective BO algorithms cannot deal with mixed-variable problems. We present MixMOBO, the first mixed-variable, multi-objective Bayesian optimization framework for such problems. Using MixMOBO, optimal Pareto-fronts for multi-objective, mixed-variable design spaces can be found efficiently while ensuring diverse solutions. The method is sufficiently flexible to incorporate different kernels and acquisition functions, including those that were developed for mixed-variable or multi-objective problems by other authors. We also present HedgeMO, a modified Hedge strategy that uses a portfolio of acquisition functions for multi-objective problems. We present a new acquisition function, SMC. Our results show that MixMOBO performs well against other mixed-variable algorithms on synthetic problems. We apply MixMOBO to the real-world design of an architected material and show that our optimal design, which was experimentally fabricated and validated, has a normalized strain energy density $10^4$ times greater than existing structures.

Optimisation of linear structures is a well-developed and highly relevant research field. In particular, for linear problems with differentiable objective functions, gradient-based methods are well suited. In engineering practice, structural nonlinear behaviour and imperfections different from ideal design often need to be considered. Optimised lightweight structures, such as long-span bridges and stadium roofs, are more sensitive to sudden failure through buckling. Imperfections or perturbations in geometry and loading can drastically reduce the buckling load of a structure compared to that of a perfect structure. It is, however, computationally expensive to analyse structural problems with nonlinearities and imperfections and to compute the derivatives of the respective objective functions, rendering gradient-based methods unsuitable. In the present work, we use a Bayesian optimisation technique with a Gaussian process as the surrogate model for the described structural problems with random imperfections. Bayesian optimisation is exceptionally well suited for expensive black-box objective functions with no analytical expressions. To rapidly determine the random buckling loads, we consider an extended system of equilibrium equations for directly computing the buckling loads, i.e. bifurcation and snap-through points on the load-displacement path. This approach is exceedingly efficient for repeatedly computing buckling loads for different imperfections. We use the complex-step derivative approximation to determine the required directional derivatives of matrices. A low discrepancy sequence, Sobol sequence, is adopted for sampling from imperfections