Adaptive cost-aware Bayesian optimization

Information-based Bayesian optimization (BO) algorithms have achieved state-of-the-art performance in optimizing a black-box objective function. However, they usually require several approximations or simplifying assumptions (without clearly understanding their effects on the BO performance) and/or their generalization to batch BO is computationally unwieldy, especially with an increasing batch size. To alleviate these issues, this paper presents a novel trusted-maximizers entropy search (TES) acquisition function: It measures how much an input query contributes to the information gain on the maximizer over a finite set of trusted maximizers, i.e., inputs optimizing functions that are sampled from the Gaussian process posterior belief of the objective function. Evaluating TES requires either only a stochastic approximation with sampling or a deterministic approximation with expectation propagation, both of which are investigated and empirically evaluated using synthetic benchmark objective functions and real-world optimization problems, e.g., hyperparameter tuning of a convolutional neural network and synthesizing ‘physically realizable’ faces to fool a black-box face recognition system. Though TES can naturally be generalized to a batch variant with either approximation, the latter is amenable to be scaled to a much larger batch size in our experiments.

The CompModels package for R provides a suite of computer model test functions that can be used for computer model prediction/emulation, uncertainty quantification, and calibration.Moreover, the CompModels package is especially well suited for the sequential optimization of computer models.The package is a mix of real-world physics problems, known mathematical functions, and black-box functions that have been converted into computer models with the goal of Bayesian (i.e., sequential) optimization in mind.Likewise, the package contains computer models that represent either the constrained or unconstrained optimization case, each with varying levels of difficulty.In this paper, we illustrate the use of the package with both real-world examples and black-box functions by solving constrained optimization problems via Bayesian optimization.Ultimately, the package is shown to provide users with a source of computer model test functions that are reproducible, shareable, and that can be used for benchmarking of novel optimization methods.

Combinatorial optimization problems form a class of appealing theoretical and practical problems attractive for their complexity and known hardness. They are often NP-hard and as such not solvable by exact methods. Combinatorial optimization problems are subject to numerous heuristic and metaheuristic algorithms, including genetic algorithms. In this paper, we present two new permutation encodings for genetic algorithms and experimentally evaluate the influence of the encodings on the performance and result of genetic algorithm on two synthetic and real-world optimization problems.

Surrogate algorithms such as Bayesian optimisation are especially designed for black-box optimisation problems with expensive objectives, such as hyperparameter tuning or simulation-based optimisation. In the literature, these algorithms are usually evaluated with synthetic benchmarks which are well established but have no expensive objective, and only on one or two real-life applications which vary wildly between papers. There is a clear lack of standardisation when it comes to benchmarking surrogate algorithms on real-life, expensive, black-box objective functions. This makes it very difficult to draw conclusions on the effect of algorithmic contributions and to give substantial advice on which method to use when. A new benchmark library, EXPObench, provides first steps towards such a standardisation. The library is used to provide an extensive comparison of six different surrogate algorithms on four expensive optimisation problems from different real-life applications. This has led to new insights regarding the relative importance of exploration, the evaluation time of the objective, and the used model. We also provide rules of thumb for which surrogate algorithm to use in which situation. A further contribution is that we make the algorithms and benchmark problem instances publicly available, contributing to more uniform analysis of surrogate algorithms. Most importantly, we include the results of the six algorithms on all evaluated problem instances. This unique new dataset lowers the bar for researching new methods as the number of expensive evaluations required for comparison and for the creation of new surrogate models is significantly reduced.

Metaheuristic algorithms are constructed to solve optimization problems, but they cannot solve all the problems with best solutions. This work proposes a novel self-adaptive metaheuristic optimization algorithm, named Optimal Stochastic Process Optimizer (OSPO), which can solve different kinds of optimization problems with promising performance. Specifically, OSPO regards the procedure of optimization as a realization of stochastic process, and with the help of Subjective Probability Distribution Function (SPDF) and Receding Sampling Strategy proposed in this paper, OSPO can control the exploration-exploitation property online by the adaptive modification of the parameters in SPDF. This adaptive exploration-exploitation property of OSPO contributes to dealing with different kinds of problems; thus, it makes OSPO have the potential to solve at least a vast majority of optimization problems. The proposed algorithm is first benchmarked on uni-modal, multi-modal and composite test functions both in low and high dimensions. The results are verified by comparative studies with seven well-performed metaheuristic algorithms. Then, 21 real-world optimization problems are used to further investigate the effectiveness of OSPO. The winners of CEC2020 Competition on Real-World Single Objective Constrained Optimization, SASS algorithm, sCMAgES algorithm, EnMODE algorithm and COLSHADE algorithm are used as four comparative algorithms in real-world optimization problems. The analysis of simulations demonstrates that OSPO is able to provide very competitive performance compared to the comparative meta-heuristics both in benchmark functions and in real-world optimization problems; thus, the potential of OSPO to solve at least a vast majority of optimization problems is verified. A corresponding MATLAB APP demo is available on https://github.com/JiahongXu123/OSPO-algorithm.git.

Teaching learning-based optimization is one of the widely accepted metaheuristic algorithms inspired by teaching and learning within classrooms. It has successfully addressed several real-world optimization problems, but it may still be trapped in local optima and may suffer from the problem of premature convergence in the case of solving some challenging optimization problems. To overcome these drawbacks and to achieve an appropriate percentage of exploitation and exploration, this study presents a new modified teaching learning-based optimization algorithm called the fuzzy adaptive teaching learning-based optimization algorithm. The proposed fuzzy adaptive teaching learning-based optimization algorithm uses three measures from the search space, namely, quality measure, diversification measure, and intensification measure. As the 50-50 probabilities for exploitation and exploration in the basic teaching learning-based optimization algorithm may be counterproductive, the Mamdani-type fuzzy inference system of the new algorithm takes these measures as a crisp inputs and generates selection as crisp output to choose either exploitation or exploration based on the current search requirement. This fuzzy-based adaptive selection helps to adequately balance global search or exploration and local search or exploitation operations during the search process as these operations are intrinsically dynamic. The performance of the fuzzy adaptive teaching learning-based optimization is evaluated against other metaheuristic algorithms including basic teaching learning-based optimization on 23 unconstrained global test functions. Moreover, adaptive teaching learning-based optimization is used to search for near-optimal values for the four parameters of the COCOMO II model, which are then tested for validity on a software project of NASA. Analysis and comparison of the obtained results indicate the efficiency and competitiveness of the proposed algorithm in addressing unconstrained continuous optimization tasks.

ABSTRACTBayesian optimization is a popular and versatile approach that is well suited to solve challenging optimization problems. Their popularity comes from their effective minimization of expensive function evaluations, their capability to leverage gradients, and their efficient use of noisy data. Bayesian optimizers have commonly been applied to global unconstrained problems, with limited development for many other classes of problems. In this article, two alternative methods are developed that enable rapid and deep convergence of nonlinearly‐constrained local optimization problems using a Bayesian optimizer. The first method uses an exact augmented Lagrangian and the second augments the minimization of the acquisition function to contain additional constraints. Both of these methods can be applied to nonlinear equality constraints, unlike most previous methods developed for constrained Bayesian optimizers. The new methods are applied with a gradient‐enhanced Bayesian optimizer and enable deeper convergence for three nonlinearly‐constrained unimodal optimization problems than previously developed methods for constrained Bayesian optimization. In addition, both new methods enable the Bayesian optimizer to reach a desired tolerance with fewer function evaluations than popular quasi‐Newton optimizers from SciPy and MATLAB for unimodal problems with 2 to 30 variables. The Bayesian optimizer had similar results using both methods. It is recommended that users first try using the second method, which adds constraints to the acquisition function minimization, since its parameters are more intuitive to tune for new problems.

In recent years, there has been a growing interest in optimization problems with uncertainty. Intuitionistic fuzzy optimization is one of the approaches for investigating such real-world optimization problems that contain uncertain data and, therefore, that are not well defined. Nonetheless, there is not enough discussion on the optimality conditions for optimal solutions in intuitionistic fuzzy optimization problems (IFOPs) and methods for solving such uncertain extremum problems. Although the derivative notion is a very important property of optimization models, there are many real-world problems and processes with uncertainty that cannot be modeled as differentiable intuitionistic fuzzy functions (IFFs). In this work, therefore, a new notion of generalized derivative is defined. Namely, the definition of symmetric gH-derivative is introduced for nonsmooth IFOPs and some properties of this concept are analyzed. Further, the Karush–Kuhn–Tucker optimality conditions are established for an optimal solution in an intuitionistic fuzzy extremum problem with symmetric gH-differentiable IFFs and inequality constraints. Meanwhile, several examples are given to illuminate the obtained results.

Optimisation of linear structures is a well-developed and highly relevant research field. In particular, for linear problems with differentiable objective functions, gradient-based methods are well suited. In engineering practice, structural nonlinear behaviour and imperfections different from ideal design often need to be considered. Optimised lightweight structures, such as long-span bridges and stadium roofs, are more sensitive to sudden failure through buckling. Imperfections or perturbations in geometry and loading can drastically reduce the buckling load of a structure compared to that of a perfect structure. It is, however, computationally expensive to analyse structural problems with nonlinearities and imperfections and to compute the derivatives of the respective objective functions, rendering gradient-based methods unsuitable. In the present work, we use a Bayesian optimisation technique with a Gaussian process as the surrogate model for the described structural problems with random imperfections. Bayesian optimisation is exceptionally well suited for expensive black-box objective functions with no analytical expressions. To rapidly determine the random buckling loads, we consider an extended system of equilibrium equations for directly computing the buckling loads, i.e. bifurcation and snap-through points on the load-displacement path. This approach is exceedingly efficient for repeatedly computing buckling loads for different imperfections. We use the complex-step derivative approximation to determine the required directional derivatives of matrices. A low discrepancy sequence, Sobol sequence, is adopted for sampling from imperfections

Nowadays, real-world optimization problems are becoming increasingly complex tasks, prompting the development of nature-inspired algorithms that mimic biological phenomena to improve search performance and solution quality. The Mud Ring Algorithm (MRA), inspired by the cooperative hunting behavior of bottlenose dolphins, has shown promise but remains sensitive to parameter settings, especially when balancing exploration and exploitation. To address these limitations, this paper proposes the Enhanced Mud Ring Algorithm (EMRA), which introduces a novel mechanism to more effectively manage the exploration-exploitation tradeoff. This modification allows the algorithm to escape local minima and explore the solution space more effectively. Numerical experiments on some standard benchmark functions as well as the difficult Congress on Evolutionary Computation 2019 benchmark suite show that EMRA outperforms the original MRA in terms of accuracy performance and computational cost, especially when dealing with high-dimension and multi-peak functions. In addition, EMRA was used in three complex engineering optimization problem designs (welded beam, pressure vessel, and tension spring), and the results were found to be more accurate and reliable than MRA. These results validate the strength and applicability of EMRA as a general optimization tool to tackle complex problems from many different disciplines, including real-world problems requiring the exhaustive exploration of multiple options. In summary, this study shows that EMRA is an effective extension of metaheuristic optimization that is applicable in real-world problems.

Many real-world optimisation problems are defined over both categorical and continuous variables, yet efficient optimisation methods such as Bayesian Optimisation (BO) are ill-equipped to handle such mixed-variable search spaces. The optimisation breadth introduced by categorical variables in the mixed-input setting has seen recent approaches operating on local trust regions, but these methods can be greedy in suboptimal regions of the search space. In this paper, we adopt a holistic view and aim to consolidate optimisation of the categorical and continuous sub-spaces under a single acquisition metric. We develop a tree-based method which retains a global view of the optimisation spaces by identifying regions in the search space with high potential candidates which we call value proposals. Our method uses these proposals to make selections on both the categorical and continuous components of the input. We show that this approach significantly outperforms existing mixed-variable optimisation approaches across several mixed-variable black-box optimisation tasks.

In the last years, there has been a large growth in the research of computational techniques inspired in nature. This area, named Bioinspired Computing, has provided biologically motivated solutions for several real world problems. Among Bioinspired Computing techniques, one can mention Artificial Neural Networks (ANN), Evolutionary Algorithms (EA), Artificial Immune Systems (AIS) and Ant Colony Optimization (ACO). ACO is a meta-heuristic based on the structure and behavior of ant colonies. It has been successfully applied to several optimization problems. Several real world optimization problems may change the configuration of its search space with time. These problems are known as dynamic optimization problems. This chapter presents the main concepts of ACO and show how it can be applied to solve real optimization problems on dynamic environments. As a case study, it will be illustrated how ACO can be applied to process scheduling problems. The chapter starts describing how real ants forage for food, environmental modifications related to this activity (for example, the blockage of a path) and the effects of these modifications in the colony. After, a brief review of previous works on Ant Colony Optimization is presented. For computer simulations, the activities of an ant colony may be modeled as a graph. After modeling the problem of foraging for food (as a graph), the goal (finding the shortest path at a given moment) is defined and the mechanism to solve the problem is presented. The need to model an on-line problem as a graph, the goal of finding the shortest path and the mechanisms adopted for solving the problem (an adapted version of the AntSystem) are detailed. Before ACO is applied to the process scheduling problem, the problem is analyzed and modeled using a graph representation. Next, simulation results obtained in a set of experiments are presented, which are validated by results obtained in a real implementation. Important issues related with the use of ACO for process scheduling, like parameter adjustments, are discussed. In the conclusion of this chapter, we point out a few future directions for ACO researches. The use of computational techniques inspired in nature has become very frequent in the last years. This area, named Bioinspired Computing, provides efficient biologically motivated

Bayesian optimization (BO) is a versatile and robust global optimization method under uncertainty. However, most of the BO algorithms were developed for problems with only continuous variables. For practical engineering optimization, discrete variables are also prevalent. BO methods based on Gaussian process (GP) surrogates also suffers from the curse-of-dimensionality problem. To address these challenges, in this paper, a Bayesian discrete optimization algorithm is introduced to solve permutation-based combinatorial problems. A new kernel function is developed based on position distances for permutation. To improve the efficiency and scalability of the algorithm, a sparse GP model based on inducing points is further developed, where the simulated annealing algorithm is applied to select inducing points. The new algorithm is demonstrated and tested with the production scheduling problem for additive manufacturing. Experimental results show that the proposed algorithm can find a better solution with limited evaluations than state-of-the-art algorithms.

Bayesian Optimization (BO) has been widely applied to optimize expensive black-box functions while retaining sample efficiency. However, scaling BO to high-dimensional spaces remains challenging. Existing literature proposes performing standard BO in multiple local trust regions (TuRBO) for heterogeneous modeling of the objective function and avoiding over-exploration. Despite its advantages, using local Gaussian Processes (GPs) reduces sampling efficiency compared to a global GP. To enhance sampling efficiency while preserving heterogeneous modeling, we propose to construct multiple local quadratic models using gradients and Hessians from a global GP, and select new sample points by solving the bound-constrained quadratic program. Additionally, we address the issue of vanishing gradients of GPs in high-dimensional spaces. We provide a convergence analysis and demonstrate through experimental results that our method enhances the efficacy of TuRBO and outperforms a wide range of high-dimensional BO techniques on synthetic functions and real-world applications.

While Bayesian Optimization (BO) is a very popular method for optimizing expensive black-box functions, it fails to leverage the experience of domain experts. This causes BO to waste function evaluations on bad design choices (e.g., machine learning hyperparameters) that the expert already knows to work poorly. To address this issue, we introduce Bayesian Optimization with a Prior for the Optimum (BOPrO). BOPrO allows users to inject their knowledge into the optimization process in the form of priors about which parts of the input space will yield the best performance, rather than BO’s standard priors over functions, which are much less intuitive for users. BOPrO then combines these priors with BO’s standard probabilistic model to form a pseudo-posterior used to select which points to evaluate next. We show that BOPrO is around \(6.67\times \) faster than state-of-the-art methods on a common suite of benchmarks, and achieves a new state-of-the-art performance on a real-world hardware design application. We also show that BOPrO converges faster even if the priors for the optimum are not entirely accurate and that it robustly recovers from misleading priors.