Bayesian Optimization with a Prior for the Optimum

Bayesian Optimization (BO) has been widely applied to optimize expensive black-box functions while retaining sample efficiency. However, scaling BO to high-dimensional spaces remains challenging. Existing literature proposes performing standard BO in multiple local trust regions (TuRBO) for heterogeneous modeling of the objective function and avoiding over-exploration. Despite its advantages, using local Gaussian Processes (GPs) reduces sampling efficiency compared to a global GP. To enhance sampling efficiency while preserving heterogeneous modeling, we propose to construct multiple local quadratic models using gradients and Hessians from a global GP, and select new sample points by solving the bound-constrained quadratic program. Additionally, we address the issue of vanishing gradients of GPs in high-dimensional spaces. We provide a convergence analysis and demonstrate through experimental results that our method enhances the efficacy of TuRBO and outperforms a wide range of high-dimensional BO techniques on synthetic functions and real-world applications.

Aim/Purpose: Traditional hyperparameter tuning methods’ inefficiency and low accuracy in diabetes prediction. Application and evaluation of Bayesian optimization aim to increase the accuracy of diabetes prediction models. Background: Bayesian optimization (BO) has emerged as a powerful technique for hyperparameter tuning in machine learning models with its methodical approach for optimizing complex functions with expensive evaluations. Selecting the optimal hyperparameters for models can significantly increase prediction accuracy in the healthcare sector, particularly for diabetes prediction, hence improving patient outcomes and resource management. Methodology: Among the well-known diabetes databases used in the study is the Pima Indian Diabetes Database. Among the machine learning models developed and whose hyperparameters are modified via Bayesian optimization are Support Vector Machines (SVM), Random Forest, and Gradient Boosting Machines (GBM). The optimization process is compared with traditional tuning methods to assess improvements in model performance. Contribution: Problems include the increasing prevalence of diabetes worldwide and the role vital function prediction models play in early diagnosis and therapy. Especially when dealing with enormous datasets common in healthcare, grid search, and random search are sometimes computationally taxing and inefficient. However, Bayesian optimization promises a more practical and economical approach by selecting hyperparameters iteratively based on earlier evaluations. Findings: Bayesian optimization not only yields faster calculation times but also outperforms traditional methods. More precisely, models adapted by Bayesian optimization show greater sensitivity and specificity, which are crucial for accurate and prompt diabetes diagnosis. Recommendation for Researchers: By highlighting the specific elements and network characteristics that are behind the anomalies it discovers, on a real-time or large scale dataset. Future Research: This work can be improvised using several recent artificial intelligence algorithms with the integration of IoT on real-time datasets.

Adaptive cost-aware Bayesian optimization

The CompModels package for R provides a suite of computer model test functions that can be used for computer model prediction/emulation, uncertainty quantification, and calibration.Moreover, the CompModels package is especially well suited for the sequential optimization of computer models.The package is a mix of real-world physics problems, known mathematical functions, and black-box functions that have been converted into computer models with the goal of Bayesian (i.e., sequential) optimization in mind.Likewise, the package contains computer models that represent either the constrained or unconstrained optimization case, each with varying levels of difficulty.In this paper, we illustrate the use of the package with both real-world examples and black-box functions by solving constrained optimization problems via Bayesian optimization.Ultimately, the package is shown to provide users with a source of computer model test functions that are reproducible, shareable, and that can be used for benchmarking of novel optimization methods.

ABSTRACTBayesian optimization is a popular and versatile approach that is well suited to solve challenging optimization problems. Their popularity comes from their effective minimization of expensive function evaluations, their capability to leverage gradients, and their efficient use of noisy data. Bayesian optimizers have commonly been applied to global unconstrained problems, with limited development for many other classes of problems. In this article, two alternative methods are developed that enable rapid and deep convergence of nonlinearly‐constrained local optimization problems using a Bayesian optimizer. The first method uses an exact augmented Lagrangian and the second augments the minimization of the acquisition function to contain additional constraints. Both of these methods can be applied to nonlinear equality constraints, unlike most previous methods developed for constrained Bayesian optimizers. The new methods are applied with a gradient‐enhanced Bayesian optimizer and enable deeper convergence for three nonlinearly‐constrained unimodal optimization problems than previously developed methods for constrained Bayesian optimization. In addition, both new methods enable the Bayesian optimizer to reach a desired tolerance with fewer function evaluations than popular quasi‐Newton optimizers from SciPy and MATLAB for unimodal problems with 2 to 30 variables. The Bayesian optimizer had similar results using both methods. It is recommended that users first try using the second method, which adds constraints to the acquisition function minimization, since its parameters are more intuitive to tune for new problems.

Bayesian optimization has proven effective for optimizing expensive-to-evaluate functions in Chemical Engineering. However, valuable physical insights from domain experts are often overlooked. This article introduces a collaborative Bayesian optimization approach that re-integrates human input into the data-driven decision-making process. By combining high-throughput Bayesian optimization with discrete decision theory, experts can influence the selection of experiments via a discrete choice. We propose a multi-objective approach togenerate a set of high-utility and distinct solutions, from which the expert selects the desired solution for evaluation at each iteration. Our methodology maintains the advantages of Bayesian optimization while incorporating expert knowledge and improving accountability. The approach is demonstrated across various case studies, including bioprocess optimization and reactor geometry design, demonstrating that even with an uninformed practitioner, the algorithm recovers the regret of standard Bayesian optimization. By including continuous expert opinion, the proposed method enables faster convergence and improved accountability for Bayesian optimization in engineering systems.

Surrogate algorithms such as Bayesian optimisation are especially designed for black-box optimisation problems with expensive objectives, such as hyperparameter tuning or simulation-based optimisation. In the literature, these algorithms are usually evaluated with synthetic benchmarks which are well established but have no expensive objective, and only on one or two real-life applications which vary wildly between papers. There is a clear lack of standardisation when it comes to benchmarking surrogate algorithms on real-life, expensive, black-box objective functions. This makes it very difficult to draw conclusions on the effect of algorithmic contributions and to give substantial advice on which method to use when. A new benchmark library, EXPObench, provides first steps towards such a standardisation. The library is used to provide an extensive comparison of six different surrogate algorithms on four expensive optimisation problems from different real-life applications. This has led to new insights regarding the relative importance of exploration, the evaluation time of the objective, and the used model. We also provide rules of thumb for which surrogate algorithm to use in which situation. A further contribution is that we make the algorithms and benchmark problem instances publicly available, contributing to more uniform analysis of surrogate algorithms. Most importantly, we include the results of the six algorithms on all evaluated problem instances. This unique new dataset lowers the bar for researching new methods as the number of expensive evaluations required for comparison and for the creation of new surrogate models is significantly reduced.

Bayesian optimization (BO) is concerned with efficient optimization using probabilistic methods. Predictive entropy search (PES) is a popular and successful BO strategy to find a point that maximizes the information gained about the optima location of an unknown function. Since the PES analytical form is intractable, it requires approximations and is computationally expensive. These approximations may degrade PES performance in terms of accuracy and efficiency. In this paper, we propose an alternative scheme - predictive variance reduction search (PVRS) - to find a point that maximally reduces the uncertainty at the perceived optima locations. The optimization converges to the true optimum when the uncertainty at all perceived optima locations is vanished. Our novel modification is beneficial in two ways. First, PVRS can be computed in closed-form, unlike the approximations made in PES. Second, PVRS is simple and easy to implement. As a result, the proposed PVRS gains huge speed up for scalable BO whilst showing favorable optimization efficiency. Furthermore, we extend our PVRS framework for batch setting where we select multiple experiments for parallel evaluations at each iteration. Empirically, we demonstrate the effectiveness of the PVRS on both benchmark functions and real-world applications in standard and batch BO settings.

This paper proposes a Sequential Model Based Optimization framework for solving optimization problems characterized by a black-box, multi-extremal, expensive and partially defined objective function, under unknown constraints. This is a typical setting for simulation-optimization problems, where the objective function cannot be computed for some configurations of the decision/control variables due to the violation of some (unknown) constraint. The framework is organized in two consecutive phases, the first uses a Support Vector Machine classifier to approximate the boundary of the unknown feasible region within the search space, the second uses Bayesian Optimization to find a globally optimal (feasible) solution. A relevant difference with traditional Bayesian Optimization is that the optimization process is performed on the estimated feasibility region, only, instead of the entire search space. Some results on three 2D test functions and a real case study for the Pump Scheduling Optimization in Water Distribution Networks are reported. The proposed framework proved to be more effective and efficient than Bayesian Optimization approaches using a penalty for function evaluations outside the feasible region.

We propose an extrinsic Bayesian optimization (eBO) framework for general optimization problems on manifolds. Bayesian optimization algorithms build a surrogate of the objective function by employing Gaussian processes and utilizing the uncertainty in that surrogate by deriving an acquisition function. This acquisition function represents the probability of improvement based on the kernel of the Gaussian process, which guides the search in the optimization process. The critical challenge for designing Bayesian optimization algorithms on manifolds lies in the difficulty of constructing valid covariance kernels for Gaussian processes on general manifolds. Our approach is to employ extrinsic Gaussian processes by first embedding the manifold onto some higher dimensional Euclidean space via equivariant embeddings and then constructing a valid covariance kernel on the image manifold after the embedding. This leads to efficient and scalable algorithms for optimization over complex manifolds. Simulation study and real data analyses are carried out to demonstrate the utilities of our eBO framework by applying the eBO to various optimization problems over manifolds such as the sphere, the Grassmannian, and the manifold of positive definite matrices.

Black-box zero-th order optimizationis a central primitive for applications in fields as diverse as finance, physics, and engineering. In a common formulation of this problem, a designer sequentially attempts candidate solutions, receiving noisy feedback on the value of each attempt from the system. In this paper, we study scenarios in which feedback is also provided on the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">safety</i> of the attempted solution, and the optimizer is constrained to limit the number of unsafe solutions that are tried throughout the optimization process. Focusing on methods based on Bayesian optimization (BO), prior art has introduced an optimization scheme – referred to as <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">SafeOpt</small> – that is guaranteed not to select <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">any</i> unsafe solution with a controllable probability over feedback noise as long as strict assumptions on the safety constraint function are met. In this paper, a novel BO-based approach is introduced that satisfies safety requirements irrespective of properties of the constraint function. This strong theoretical guarantee is obtained at the cost of allowing for an arbitrary, controllable but non-zero, rate of violation of the safety constraint. The proposed method, referred to as <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Safe-Bocp</small>, builds on online conformal prediction (CP) and is specialized to the cases in which feedback on the safety constraint is either noiseless or noisy. Experimental results on synthetic and real-world data validate the advantages and flexibility of the proposed <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Safe-Bocp</small>.

In Numerical Weather Prediction (NWP) models, such as the Weather Research and Forecasting (WRF) model, parameter uncertainty in physics parameterization schemes significantly impacts model output. Our study adopts a Bayesian probabilistic approach, building on prior research that identified temperature (T) and relative humidity (Rh) as sensitive to three key WRF parameters during southeast Australia's extreme heat events. Using Gaussian process regression‐based Bayesian Optimisation (G‐BO), we accurately estimated the optimal distributions for these parameters. Results show that the default values are outside their corresponding optimal distribution bounds for two of the three parameters, suggesting the need to reconsider these default values. Additionally, the robustness of the optimal parameter distributions is validated by their application to an independent extreme heat event, not included in the optimisation process. In this test, the optimized parameters substantially improved the simulation of T and Rh, highlighting their effectiveness in enhancing simulation accuracy during extreme heat conditions.

The accelerated discovery and optimization of materials relies on the integration of advanced experimental techniques with data-driven methodologies. In this work, Bayesian optimization (BO) is applied to optimize the ultrasonic spray pyrolysis (USP) process for the deposition of copper oxides, targeting high-quality Ga2O3-Cu2O heterojunctions for optoelectronic applications. By employing BO with an initial data set of 12 samples and conducting 4 USP parameter optimization cycles, significant improvements in device performance are achieved, with the open-circuit voltage increasing from 288 to 804 mV. During the optimization process, the performance of the model declines, necessitating the identification of a reliable subset of samples from the full data set. Through the application of BO, the cross-validation error of the model is minimized based on the sample selection, whereby accuracy is restored and generalizability is achieved. The subsequent model evaluation reveals two distinct deposition regimes, each characterized by unique process conditions, leading to specific material properties and device performances. These findings not only demonstrate the application of a data-driven experimental workflow in the context of thin film deposition but also highlight the importance of robust data validation and model evaluation.

Optimisation of linear structures is a well-developed and highly relevant research field. In particular, for linear problems with differentiable objective functions, gradient-based methods are well suited. In engineering practice, structural nonlinear behaviour and imperfections different from ideal design often need to be considered. Optimised lightweight structures, such as long-span bridges and stadium roofs, are more sensitive to sudden failure through buckling. Imperfections or perturbations in geometry and loading can drastically reduce the buckling load of a structure compared to that of a perfect structure. It is, however, computationally expensive to analyse structural problems with nonlinearities and imperfections and to compute the derivatives of the respective objective functions, rendering gradient-based methods unsuitable. In the present work, we use a Bayesian optimisation technique with a Gaussian process as the surrogate model for the described structural problems with random imperfections. Bayesian optimisation is exceptionally well suited for expensive black-box objective functions with no analytical expressions. To rapidly determine the random buckling loads, we consider an extended system of equilibrium equations for directly computing the buckling loads, i.e. bifurcation and snap-through points on the load-displacement path. This approach is exceedingly efficient for repeatedly computing buckling loads for different imperfections. We use the complex-step derivative approximation to determine the required directional derivatives of matrices. A low discrepancy sequence, Sobol sequence, is adopted for sampling from imperfections

Science and Engineering applications are typically associated with expensive optimization problem to identify optimal design solutions and states of the system of interest. Bayesian optimization and active learning compute surrogate models through efficient adaptive sampling schemes to assist and accelerate this search task toward a given optimization goal. Both those methodologies are driven by specific infill/learning criteria which quantify the utility with respect to the set goal of evaluating the objective function for unknown combinations of optimization variables. While the two fields have seen an exponential growth in popularity in the past decades, their dualism and synergy have received relatively little attention to date. This paper discusses and formalizes the synergy between Bayesian optimization and active learning as symbiotic adaptive sampling methodologies driven by common principles. In particular, we demonstrate this unified perspective through the formalization of the analogy between the Bayesian infill criteria and active learning criteria as driving principles of both the goal-driven procedures. To support our original perspective, we propose a general classification of adaptive sampling techniques to highlight similarities and differences between the vast families of adaptive sampling, active learning, and Bayesian optimization. Accordingly, the synergy is demonstrated mapping the Bayesian infill criteria with the active learning criteria, and is formalized for searches informed by both a single information source and multiple levels of fidelity. In addition, we provide guidelines to apply those learning criteria investigating the performance of different Bayesian schemes for a variety of benchmark problems to highlight benefits and limitations over mathematical properties that characterize real-world applications.