Benchmarking surrogate-based optimisation algorithms on expensive black-box functions

Bayesian optimization (BO) is a low-cost global optimization tool for expensive black-box objective functions, where we learn from prior evaluated designs, update a posterior surrogate Gaussian process model, and select new designs for future evaluation using an acquisition function. This research focuses upon developing a BO model with multiple black-box objective functions. In the standard multi-objective (MO) optimization problem, the weighted Tchebycheff method is efficiently used to find both convex and non-convex Pareto frontiers. This approach requires knowledge of utopia values before we start optimization. However, in the BO framework, since the functions are expensive to evaluate, it is very expensive to obtain the utopia values as a prior knowledge. Therefore, in this paper, we develop a MO-BO framework where we calibrate with multiple linear regression (MLR) models to estimate the utopia value for each objective as a function of design input variables; the models are updated iteratively with sampled training data from the proposed MO-BO. These iteratively estimated mean utopia values are used to formulate the weighted Tchebycheff MO acquisition function. The proposed approach is implemented in two numerical test examples and one engineering design problem of optimizing thin tube geometries under constant loading of temperature and pressure, with minimizing the risk of creep-fatigue failure and design cost, along with risk-based and manufacturing constraints. Finally, the model accuracy with frequentist, Bayesian and without MLR-based calibration are compared to true Pareto solutions.

Both evolutionary Bayesian optimization (EBO) and Bayesian evolutionary optimization (BEO) are used to solve time-consuming optimization problems, where in the EBO method, the evolutionary algorithm is adopted to optimize the acquisition function for finding its optimal solution to be evaluated using the time consuming objective functions, while in the BEO approach, surrogates are utilized to replace the expensive objective functions and the acquisition function is used in the model management to choose individuals for evaluations using the expensive objective functions. In this paper, we perform comparisons on these two approaches to see which method is better for solving computationally expensive multi-objective problems. The expected improvement (EI) is used as the acquisition function in both EBO and BEO approaches. Two strategies are proposed to select individuals to be evaluated using the real computationally expensive objectives. The experimental results conducted on ZDT and DTLZ test problems showed that the BEO approach is more efficient than the EBO approach for solving multi-objective problems.

Accelerating surrogate assisted evolutionary algorithms for expensive multi-objective optimization via explainable machine learning

Constrained Bayesian optimization optimizes a black-box objective function subject to black-box constraints. For simplicity, most existing works assume that multiple constraints are independent. To ask, when and how does dependence between constraints help?, we remove this assumption and implement probability of feasibility with dependence (Dep-PoF) by applying multiple output Gaussian processes (MOGPs) as surrogate models and using expectation propagation to approximate the probabilities. We compare Dep-PoF and the independent version PoF. We propose two new acquisition functions incorporating Dep-PoF and test them on synthetic and practical benchmarks. Our results are largely negative: incorporating dependence between the constraints does not help much. Empirically, incorporating dependence between constraints may be useful if: (i) the solution is on the boundary of the feasible region(s) or (ii) the feasible set is very small. When these conditions are satisfied, the predictive covariance matrix from the MOGP may be poorly approximated by a diagonal matrix and the off-diagonal matrix elements may become important. Dep-PoF may apply to settings where (i) the constraints and their dependence are totally unknown and (ii) experiments are so expensive that any slightly better Bayesian optimization procedure is preferred. But, in most cases, Dep-PoF is indistinguishable from PoF.

Fuel cell trucks offer significant environmental and performance advantages such as zero emissions, fast hydrogen refueling, and low noise, making them suitable for long-distance heavy-load transportation. Reducing hydrogen consumption to save operating costs has consistently been a critical focus in fuel cell truck research. To reduce hydrogen consumption and improve driving range, this paper proposes a fuzzy logic control method with uncertain rules leveraging digital twin technology. First, a digital twin model of the fuel cell truck is established, incorporating uncertain rule-based fuzzy logic control. Next, we set the parameters of the fuzzy logic controller as decision variables and define the hydrogen consumption of the fuel cell truck during the controlled period as the objective function, which is a black-box objective function. Furthermore, we employ an algorithm combining the surrogate model-based algorithm and the adaptive particle swarm optimization algorithm to solve the constructed optimization problem and determine the optimal fuzzy logic control parameters. Finally, the effectiveness and superiority of the proposed digital twin-based fuzzy logic control method with uncertain rules are verified through simulations and hardware-in-the-loop experiments.

Optimisation of linear structures is a well-developed and highly relevant research field. In particular, for linear problems with differentiable objective functions, gradient-based methods are well suited. In engineering practice, structural nonlinear behaviour and imperfections different from ideal design often need to be considered. Optimised lightweight structures, such as long-span bridges and stadium roofs, are more sensitive to sudden failure through buckling. Imperfections or perturbations in geometry and loading can drastically reduce the buckling load of a structure compared to that of a perfect structure. It is, however, computationally expensive to analyse structural problems with nonlinearities and imperfections and to compute the derivatives of the respective objective functions, rendering gradient-based methods unsuitable. In the present work, we use a Bayesian optimisation technique with a Gaussian process as the surrogate model for the described structural problems with random imperfections. Bayesian optimisation is exceptionally well suited for expensive black-box objective functions with no analytical expressions. To rapidly determine the random buckling loads, we consider an extended system of equilibrium equations for directly computing the buckling loads, i.e. bifurcation and snap-through points on the load-displacement path. This approach is exceedingly efficient for repeatedly computing buckling loads for different imperfections. We use the complex-step derivative approximation to determine the required directional derivatives of matrices. A low discrepancy sequence, Sobol sequence, is adopted for sampling from imperfections

In this paper, we propose the StepDIRECT algorithm for derivative-free optimization (DFO), in which the black-box objective function has a stepwise landscape. Our framework is based on the well-known DIRECT algorithm. By incorporating the local variability to explore the flatness, we provide a new criterion to select the potentially optimal hyper-rectangles. In addition, we introduce a stochastic local search algorithm performing on potentially optimal hyper-rectangles to improve the solution quality and convergence speed. Global convergence of the StepDIRECT algorithm is provided. Numerical experiments on optimization for random forest models and hyper-parameter tuning are presented to support the efficacy of our algorithm. The proposed StepDIRECT algorithm shows competitive performance results compared with other state-of-the-art baseline DFO methods including the original DIRECT algorithm.

First Order Bayesian Optimization (FOBO) is a sample efficient sequential approach to find the global maxima of an expensive-to-evaluate black-box objective function by suitably querying for the function and its gradient evaluations. Such methods assume Gaussian process (GP) models for both, the function and its gradient, and use them to construct an acquisition function that identifies the next query point. In this paper, we propose a class of practical FOBO algorithms that efficiently utilizes the information from the gradient GP to identify potential query points with zero gradients. We construct a multi-level acquisition function where in the first step, we optimize a lower level acquisition function with multiple restarts to identify potential query points with zero gradient value. We then use the upper level acquisition function to rank these query points based on their function values to potentially identify the global maximum. As a final step, the potential point of maximum is chosen as the actual query point. We validate the performance of our proposed algorithms on several test functions and show that our algorithms outperform state-of-the-art FOBO algorithms. We also illustrate the application of our algorithms in finding optimal set of hyper-parameters in machine learning and in learning the optimal policy in reinforcement learning tasks.

Direct zigzag search for discrete multi-objective optimization

The CompModels package for R provides a suite of computer model test functions that can be used for computer model prediction/emulation, uncertainty quantification, and calibration.Moreover, the CompModels package is especially well suited for the sequential optimization of computer models.The package is a mix of real-world physics problems, known mathematical functions, and black-box functions that have been converted into computer models with the goal of Bayesian (i.e., sequential) optimization in mind.Likewise, the package contains computer models that represent either the constrained or unconstrained optimization case, each with varying levels of difficulty.In this paper, we illustrate the use of the package with both real-world examples and black-box functions by solving constrained optimization problems via Bayesian optimization.Ultimately, the package is shown to provide users with a source of computer model test functions that are reproducible, shareable, and that can be used for benchmarking of novel optimization methods.

A principled approach to design using high fidelity fluid-structure interaction simulations

Information-based Bayesian optimization (BO) algorithms have achieved state-of-the-art performance in optimizing a black-box objective function. However, they usually require several approximations or simplifying assumptions (without clearly understanding their effects on the BO performance) and/or their generalization to batch BO is computationally unwieldy, especially with an increasing batch size. To alleviate these issues, this paper presents a novel trusted-maximizers entropy search (TES) acquisition function: It measures how much an input query contributes to the information gain on the maximizer over a finite set of trusted maximizers, i.e., inputs optimizing functions that are sampled from the Gaussian process posterior belief of the objective function. Evaluating TES requires either only a stochastic approximation with sampling or a deterministic approximation with expectation propagation, both of which are investigated and empirically evaluated using synthetic benchmark objective functions and real-world optimization problems, e.g., hyperparameter tuning of a convolutional neural network and synthesizing ‘physically realizable’ faces to fool a black-box face recognition system. Though TES can naturally be generalized to a batch variant with either approximation, the latter is amenable to be scaled to a much larger batch size in our experiments.

Process optimization of a Non-circular drawing (NCD) sequence of a pearlitic steel wire was performed to improve the mechanical properties of a drawn wire based on surrogate assisted meta-heuristic algorithms. The objective function was introduced to minimize inhomogeneity of effective strain distribution at the cross-section of the drawn wire, which could deteriorate delamination characteristics of the drawn wires. The design variables introduced were die geometry and reduction of area of the NCD sequence. Several surrogate models and their combinations with the weighted sum technique were utilized. In the process optimization of the NCD sequence, the surrogate models were used to predict effective strain distributions at the cross-section of the drawn wire. Optimization using Differential evolution (DE) algorithm was performed, while the objective function was calculated from the predicted effective strains. The accuracy of all surrogate models was investigated, while optimum results were compared with the previous study available in the literature. It was found that hybrid surrogate models can improve prediction accuracy compared to a single surrogate model. The best result was obtained from the combination of Kriging (KG) and Support vector regression (SVR) models, while the second best was obtained from the combination of four surrogate models: Polynomial response surface (PRS), Radial basic function (RBF), KG, and SVR. The optimum results found in this study showed better effective strain homogeneity at the cross-section of the drawn wire with the same total reduction of area of the previous work available in the literature for fewer number of passes. The multi-surrogate models with the weighted sum technique were found to be powerful in improving the delamination characteristics of the drawn wire and reducing the production cost.

The presence of black-box functions in engineering design, which are usually computation-intensive, demands efficient global optimization methods. This work proposes a new global optimization method for black-box functions. The global optimization method is based on a novel mode-pursuing sampling (MPS) method which systematically generates more sample points in the neighborhood of the function mode while statistically covers the entire search space. Quadratic regression is performed to detect the region containing the global optimum. The sampling and detection process iterates until the global optimum is obtained. Through intensive testing, this method is found to be effective, efficient, robust, and applicable to both continuous and discontinuous functions. It supports simultaneous computation and applies to both unconstrained and constrained optimization problems. Because it does not call any existing global optimization tool, it can be used as a standalone global optimization method for inexpensive problems as well. Limitation of the method is also identified and discussed.

A Surrogate-Based Optimization Method with Dynamic Adaptation for High-Dimensional Mixed-Integer Problems