Bayesian Approaches to Surrogate-Assisted Evolutionary Multi-objective Optimization: A Comparative Study

An adaptive Bayesian approach to surrogate-assisted evolutionary multi-objective optimization

Surrogate algorithms such as Bayesian optimisation are especially designed for black-box optimisation problems with expensive objectives, such as hyperparameter tuning or simulation-based optimisation. In the literature, these algorithms are usually evaluated with synthetic benchmarks which are well established but have no expensive objective, and only on one or two real-life applications which vary wildly between papers. There is a clear lack of standardisation when it comes to benchmarking surrogate algorithms on real-life, expensive, black-box objective functions. This makes it very difficult to draw conclusions on the effect of algorithmic contributions and to give substantial advice on which method to use when. A new benchmark library, EXPObench, provides first steps towards such a standardisation. The library is used to provide an extensive comparison of six different surrogate algorithms on four expensive optimisation problems from different real-life applications. This has led to new insights regarding the relative importance of exploration, the evaluation time of the objective, and the used model. We also provide rules of thumb for which surrogate algorithm to use in which situation. A further contribution is that we make the algorithms and benchmark problem instances publicly available, contributing to more uniform analysis of surrogate algorithms. Most importantly, we include the results of the six algorithms on all evaluated problem instances. This unique new dataset lowers the bar for researching new methods as the number of expensive evaluations required for comparison and for the creation of new surrogate models is significantly reduced.

Rather than optimizing expensive objective functions such as complex engineering simulations directly, Bayesian optimization methodologies fit a surrogate model (typically Kriging or a Gaussian Process) on evaluations of the objective function(s). To determine the next evaluation, an acquisition function is optimized (also referred to as infill criterion or sampling policy) which incorporates the model prediction and uncertainty and balances exploration and exploitation. Therefore, Bayesian optimization methodologies replace a single optimization of the objective function by a sequence of optimization problems: this makes sense as the acquisition function is cheap-to-evaluate whereas the objective is not. Depending on the goal different acquisition functions are available: multi-objective acquisition functions are relatively new and this chapter gives a state-of-the-art overview and illustrates some approaches based on hypervolume improvement. It is shown that the quality of the model is crucial for the performance of Bayesian optimization and illustrate this by using the more flexible Student-t processes as surrogate models.

Optimizing multiple, non-preferential objectives for mixed-variable, expensive black-box problems is important in many areas of engineering and science. The expensive, noisy, black-box nature of these problems makes them ideal candidates for Bayesian optimization (BO). Mixed-variable and multi-objective problems, however, are a challenge due to BO's underlying smooth Gaussian process surrogate model. Current multi-objective BO algorithms cannot deal with mixed-variable problems. We present MixMOBO, the first mixed-variable, multi-objective Bayesian optimization framework for such problems. Using MixMOBO, optimal Pareto-fronts for multi-objective, mixed-variable design spaces can be found efficiently while ensuring diverse solutions. The method is sufficiently flexible to incorporate different kernels and acquisition functions, including those that were developed for mixed-variable or multi-objective problems by other authors. We also present HedgeMO, a modified Hedge strategy that uses a portfolio of acquisition functions for multi-objective problems. We present a new acquisition function, SMC. Our results show that MixMOBO performs well against other mixed-variable algorithms on synthetic problems. We apply MixMOBO to the real-world design of an architected material and show that our optimal design, which was experimentally fabricated and validated, has a normalized strain energy density $10^4$ times greater than existing structures.

Bayesian Co-evolutionary Optimization based entropy search for high-dimensional many-objective optimization

Bayesian Optimization (BO) is a framework for black-box optimization that is especially suitable for expensive cost functions. Among the main parts of a BO algorithm, the acquisition function is of fundamental importance, since it guides the optimization algorithm by translating the uncertainty of the regression model in a utility measure for each point to be evaluated. Considering such aspect, selection and design of acquisition functions are one of the most popular research topics in BO. Since no single acquisition function was proved to have better performance in all tasks, a well-established approach consists of selecting different acquisition functions along the iterations of a BO execution. In such an approach, the GP-Hedge algorithm is a widely used option given its simplicity and good performance. Despite its success in various applications, GP-Hedge shows an undesirable characteristic of accounting on all past performance measures of each acquisition function to select the next function to be used. In this case, good or bad values obtained in an initial iteration may impact the choice of the acquisition function for the rest of the algorithm. This fact may induce a dominant behavior of an acquisition function and impact the final performance of the method. Aiming to overcome such limitation, in this work we propose a variant of GP-Hedge, named No-PASt-BO, that reduce the influence of far past evaluations. Moreover, our method presents a built-in normalization that avoids the functions in the portfolio to have similar probabilities, thus improving the exploration. The obtained results on both synthetic and real-world optimization tasks indicate that No-PASt-BO presents competitive performance and always outperforms GP-Hedge.

In this paper, we present a surrogate-based multi-objective evolutionary optimization approach to optimize airfoil aerodynamic designs. Our approach makes use of multiple surrogate models which operate in parallel with the aim of combining their features when solving a costly multi-objective optimization problem. The proposed approach is used to solve five multiobjective airfoil aerodynamic optimization problems. We compare the performance of a multi-objective evolutionary algorithm with surrogates with respect to the same approach without using surrogates. Our preliminary results indicate that our proposal can achieve a substantial reduction in the number of objective function evaluations, which has obvious advantages for dealing with expensive objective functions such as those involved in aeronautical optimization problems.

Optimizing hyperparameters is crucial for enhancing the performance of deep learning (DL) models. The process of configuring optimal hyperparameters, known as hyperparameter tuning, can be performed using various methods. Traditional approaches like grid search and random search have significant limitations. In contrast, Bayesian optimization (BO) utilizes a surrogate model and an acquisition function to intelligently navigate the hyperparameter space, aiming to provide deeper insights into performance disparities between naïve and advanced methods. This study evaluates BO's efficacy compared to baseline methods such as random search, manual search, and grid search across multiple DL architectures, including multi-layer perceptron (MLP), convolutional neural network (CNN), and LeNet, applied to the Modified National Institute of Standards and Technology (MNIST) and CIFAR-10 datasets. The findings indicate that BO, employing the tree-structured parzen estimator (TPE) search method and expected improvement (EI) acquisition function, surpasses alternative methods in intricate DL architectures such as LeNet and CNN. However, grid search shows superior performance in smaller DL architectures like MLP. This study also adopts a multi-objective (MO) perspective, balancing conflicting performance objectives such as accuracy, F1 score, and model size (parameter count). This MO assessment offers a comprehensive understanding of how these performance metrics interact and influence each other, leading to more informed hyperparameter tuning decisions.

PID Controllers are used in almost 95% of control loops in the most process industry. Since the design of a controller for an industrial system is complex due to the presence of model uncertainties, nonlinearities, and disturbances, these controllers are usually not tuned optimally and performance is compromised. Robust fixed structure control is a remedy to the problem, and such a controller design is an optimization problem, characterized by nonlinear multimodal large pursuit space, tight constraints, and expensive objective function. The classic optimization method is not appropriate here as a solution set is to be computed. Evolutionary algorithm, a robust search, and optimization method can handle this problem. Covariance Matrix Adapted Evolutionary strategy is successfully utilised to design robust PID controller for Sodium chlorate process. The performance in terms Integral Square Error (ISE) for set point tracking, robust stability and disturbance attenuation was found satisfactory.

Bayesian Optimization for Field-Scale Geological Carbon Storage

This work proposed an evolutionary building facility control optimization method that re-utilized a promising solution set archived during the optimization process. Evolutionary optimization was suited to solve multi-objective optimization problems. Evolutionary algorithms especially for addressing real-world optimization problems often employed an archive solution set to increase the approximation quality of the Pareto front, the optimal trade-off among objective functions. The proposed method not only employed the archive solution set but also reutilized it for the search. The proposed method updated the searching solution set by the archive solution set at regular intervals. Experimental results showed the re-use of the archived solutions contributed to obtaining solutions that could not be obtained by the conventional method without re-utilization of the archive solution set.

The growing complexity of engineering modeling and design problems demands effective strategies for optimization of computationally expensive objective functions. To this end, we focus on knowledge-based, variable-fidelity optimization of expensive functions through a tried and tested, yet still rapidly evolving art called space mapping optimization. Fitting into the arena of surrogatebased optimization, space-mapping optimization is a model-driven optimization process where the model is an iteratively updated surrogate derived from a valid, low-fidelity or physics-based coarse model. Space mapping takes several forms. Here, we present and formulate the original input space mapping concept, as well as the more recent implicit and output space mapping concepts. Corresponding surrogate models are presented, classified, and discussed. A proposed optimization flow is explained. Then we illustrate both input space mapping and implicit space mapping through the space mapping optimization of a simple, technology-free wedgecutting problem. We also present tuning space mapping, a powerful methodology, but one that requires extra engineering knowledge of the problem under investigation. To confirm our work, we select representative examples from the fields of microwave and antenna engineering, including filter and antenna designs.

ABSTRACTBayesian optimization is a popular and versatile approach that is well suited to solve challenging optimization problems. Their popularity comes from their effective minimization of expensive function evaluations, their capability to leverage gradients, and their efficient use of noisy data. Bayesian optimizers have commonly been applied to global unconstrained problems, with limited development for many other classes of problems. In this article, two alternative methods are developed that enable rapid and deep convergence of nonlinearly‐constrained local optimization problems using a Bayesian optimizer. The first method uses an exact augmented Lagrangian and the second augments the minimization of the acquisition function to contain additional constraints. Both of these methods can be applied to nonlinear equality constraints, unlike most previous methods developed for constrained Bayesian optimizers. The new methods are applied with a gradient‐enhanced Bayesian optimizer and enable deeper convergence for three nonlinearly‐constrained unimodal optimization problems than previously developed methods for constrained Bayesian optimization. In addition, both new methods enable the Bayesian optimizer to reach a desired tolerance with fewer function evaluations than popular quasi‐Newton optimizers from SciPy and MATLAB for unimodal problems with 2 to 30 variables. The Bayesian optimizer had similar results using both methods. It is recommended that users first try using the second method, which adds constraints to the acquisition function minimization, since its parameters are more intuitive to tune for new problems.

Bayesian optimization seeks the global optimum of a black-box, objective function f (x), in the fewest possible iterations. Recent work applied knowledge of the true value of the optimum to the Gaussian Process probabilistic model typically used in Bayesian optimization. This, together with a new acquisition function called Confidence Bound Minimization, resulted in a Gaussian probabilistic posterior in which the predictions were no greater than the known maximum (and no less than for minimum). Our novel work applies Confidence Bound Minimization to Bayesian optimization with Student's-t Processes, a probabilistic alternative which addresses known weaknesses in Gaussian Processes - outliers' probability and the calculation of posterior covariance. The new model is applied to the problem of hyperparameter tuning for an XGBoost classifier. Experiments show superior regret minimization and predictive accuracy, versus the popular Expected Improvement acquisition function. Combining Confidence Bound Minimization with a transformed Student's-t Process probabilistic model and known optima produces superior training regret minimization and posterior predictions for the Six-Hump Camel(2D) and Levy(4D) benchmark problems, which do not fall below true minima.

Most evolutionary multiobjective optimization algorithms are designed to approximate the entire Pareto front. During the last decade, a series of preference-based evolutionary algorithms have been developed, where a part of the Pareto front is approximated by incorporating the preferences of a Decision Maker. However, only a few such algorithms are able to obtain well-distributed solutions covering the complete “region of interest” that is determined by a reference point. In this paper, a preference-based evolutionary algorithm for approximating the region of interest is proposed. It is based on the state-of-the-art genetic algorithm NSGA-II and the CHIM approach introduced in the NBI method which is used to obtain uniformly distributed solutions in the region of interest. The efficiency of the proposed algorithm has been experimentally evaluated and compared to other state-of-the-art multiobjective preference-based evolutionary algorithms by solving a set of multiobjective optimization benchmark problems. It has been shown that the incorporation of the Decision Maker’s preferences and the CHIM approach into the NSGA-II algorithm allows approximating the whole region of interest accurately while maintaining a good distribution of the obtained solutions.