Bayesian optimization for congestion pricing problems: A general framework and its instability

We propose an extrinsic Bayesian optimization (eBO) framework for general optimization problems on manifolds. Bayesian optimization algorithms build a surrogate of the objective function by employing Gaussian processes and utilizing the uncertainty in that surrogate by deriving an acquisition function. This acquisition function represents the probability of improvement based on the kernel of the Gaussian process, which guides the search in the optimization process. The critical challenge for designing Bayesian optimization algorithms on manifolds lies in the difficulty of constructing valid covariance kernels for Gaussian processes on general manifolds. Our approach is to employ extrinsic Gaussian processes by first embedding the manifold onto some higher dimensional Euclidean space via equivariant embeddings and then constructing a valid covariance kernel on the image manifold after the embedding. This leads to efficient and scalable algorithms for optimization over complex manifolds. Simulation study and real data analyses are carried out to demonstrate the utilities of our eBO framework by applying the eBO to various optimization problems over manifolds such as the sphere, the Grassmannian, and the manifold of positive definite matrices.

Device sizing is a challenging problem for analog circuit design. Traditional methods depend on domain knowledge and intensive simulations to search for feasible parameters. Recent studies apply the Bayesian optimization (BO) and a Gaussian process (GP) model in analog circuit synthesis to improve efficiency. The BO framework automatically selects the parameter candidates by inferring the surrogate GP model. However, naive BO employs a sequential updating strategy which is inefficient in a multicore environment. Besides, the widely used GP model requires costly high fidelity data, which are obtained from fine simulations. In this article, we propose a constrained batch BO approach with a multifidelity (MF) model to solve the above difficulties. The batch BO exploits parallel computing and selects promising parameters by multiple acquisition function ensemble. In addition, the MF GP model adapts the low fidelity data obtained from coarse simulations. Specifically, the proposed method incorporates information gain in a weighted clustering algorithm to refine the parameter candidates. As a result, the proposed method maintains the candidates’ quality and diversity, which speeds up the optimization convergence. In the experiments, we demonstrate the efficiency of the proposed approach on three real-world circuits. The results show that our approach reduces the simulation costs by at least 54.6% compared to the state-of-the-art baselines.

pBO-2GP-3B: A batch parallel known/unknown constrained Bayesian optimization with feasibility classification and its applications in computational fluid dynamics

The simultaneous design of processes and enabling materials such as solvents, catalysts, and adsorbents is challenging because molecular‐ and process‐level decisions are strongly interdependent. Sequential approaches often yield suboptimal results since improvements in material properties may not translate into superior process performance. We propose a multiscale Bayesian Optimization (BO) framework that explicitly couples molecular‐ and process‐level design in a bi‐level structure. The inner loop solves deterministic process models, while the outer BO loop explores molecular descriptors, reducing dimensionality, improving sample efficiency, and handling process constraints. A case study on multiscale reactor and catalyst codesign demonstrates that this hybrid strategy achieves optimal performance with substantially lower computational effort than direct BO over the full decision space. BO, when aligned with the natural process—material hierarchy—thus provides an efficient methodology for multiscale design.

This paper reports a Drug Formulation System to Generics with Optimization-Based Dosage Calibration to hasten the creation of bioequivalent generic drugs. The methodology uses constrained multi-objective Bayesian Optimization (BO) to maximize probability of bioequivalence (P(BE)) and at the same time meet the critical quality attributes e.g., dissolution, friability and hardness of the tablet. The system uses BoTorch (including Ax), a PyTorch-based optimization library, to combine mechanistic modeling with surrogate learning to minimize the need to rely on trial-and-error experimentation. It has been demonstrated that the BO framework can reach P(BE) ≥ 0.90 in much fewer optimization cycles than both the conventional response surface methodology and genetic algorithms and also delivers better constraint satisfaction and computational efficiency. The proposed system can provide an uncertainty-sensitive formulation design solution that offers a regulatory-conformant avenue to decrease development time, decrease the cost of experiments, and improve reliability in generic drug approvals.

Given the increasing importance of machine learning (ML) in our lives, several algorithmic fairness techniques have been proposed to mitigate biases in the outcomes of the ML models. However, most of these techniques are specialized to cater to a single family of ML models and a specific definition of fairness, limiting their adaptibility in practice. We introduce a general constrained Bayesian optimization (BO) framework to optimize the performance of any ML model while enforcing one or multiple fairness constraints. BO is a model-agnostic optimization method that has been successfully applied to automatically tune the hyperparameters of ML models. We apply BO with fairness constraints to a range of popular models, including random forests, gradient boosting, and neural networks, showing that we can obtain accurate and fair solutions by acting solely on the hyperparameters. We also show empirically that our approach is competitive with specialized techniques that enforce model-specific fairness constraints, and outperforms preprocessing methods that learn fair representations of the input data. Moreover, our method can be used in synergy with such specialized fairness techniques to tune their hyperparameters. Finally, we study the relationship between fairness and the hyperparameters selected by BO. We observe a correlation between regularization and unbiased models, explaining why acting on the hyperparameters leads to ML models that generalize well and are fair.

Estimation of internal states in a Li-ion battery using BiLSTM with Bayesian hyperparameter optimization

Liquid organic hydrogen carriers (LOHCs) are a promising option for hydrogen storage, but a high efficiency of the LOHC cycle is essential in order to serve as an attractive technology in the context of decarbonization. This paper presents different methods to optimize steady‐state operating points of a LOHC dehydrogenation (DH) reactor. For this purpose, an analytical model of a DH reactor is described, which is extended to a hybrid model (HM) to achieve sufficient model accuracy. The model quality is subsequently validated by measurements. For the optimization of the steady‐state operation point, a Bayesian optimization framework is presented and compared to classical model‐based optimization. The methods are evaluated with the HM as well as with experimental results.

This contribution presents a new development in the design of control system based on evolving Gaussian process (GP) models . GP models provide a probabilistic, nonparametric modelling approach for black-box identification of nonlinear dynamic systems. They can highlight areas of the input space where prediction quality is poor, due to the lack of data or its complexity, by indicating the higher variance around the predicted mean. GP models contain noticeably less coefficients to be optimised than commonly used parametric models. While GP models are Bayesian models, their output is normal distribution, expressed in terms of mean and variance. Latter can be interpreted as a confidence in prediction and used in many fields, especially in control system. Evolving GP model is the concept approach within which various ways of model adaptations can be used. Successful control system needs as much as possible data about process to be controlled . If the prior knowledge about the system to be controlled is scarce or the system varies with time or operating region, this control problem can be solved with an iterative method which adapts model with information obtained with streaming data and concurrently optimises hyperparameter values. This contribution provides: a survey of adaptive control algorithms for dynamic systems described in publications where GP models have been used for control design, a novel and improved closed-loop controller with evolving GP models and an example for the illustration of proposed control algorithm.

Bayesian optimization (BO) is a versatile and robust global optimization method under uncertainty. However, most of the BO algorithms were developed for problems with only continuous variables. For practical engineering optimization, discrete variables are also prevalent. BO methods based on Gaussian process (GP) surrogates also suffers from the curse-of-dimensionality problem. To address these challenges, in this paper, a Bayesian discrete optimization algorithm is introduced to solve permutation-based combinatorial problems. A new kernel function is developed based on position distances for permutation. To improve the efficiency and scalability of the algorithm, a sparse GP model based on inducing points is further developed, where the simulated annealing algorithm is applied to select inducing points. The new algorithm is demonstrated and tested with the production scheduling problem for additive manufacturing. Experimental results show that the proposed algorithm can find a better solution with limited evaluations than state-of-the-art algorithms.

Gaussian processes (Kriging) are interpolating data-driven models that are frequently applied in various disciplines. Often, Gaussian processes are trained on datasets and are subsequently embedded as surrogate models in optimization problems. These optimization problems are nonconvex and global optimization is desired. However, previous literature observed computational burdens limiting deterministic global optimization to Gaussian processes trained on few data points. We propose a reduced-space formulation for deterministic global optimization with trained Gaussian processes embedded. For optimization, the branch-and-bound solver branches only on the free variables and McCormick relaxations are propagated through explicit Gaussian process models. The approach also leads to significantly smaller and computationally cheaper subproblems for lower and upper bounding. To further accelerate convergence, we derive envelopes of common covariance functions for GPs and tight relaxations of acquisition functions used in Bayesian optimization including expected improvement, probability of improvement, and lower confidence bound. In total, we reduce computational time by orders of magnitude compared to state-of-the-art methods, thus overcoming previous computational burdens. We demonstrate the performance and scaling of the proposed method and apply it to Bayesian optimization with global optimization of the acquisition function and chance-constrained programming. The Gaussian process models, acquisition functions, and training scripts are available open-source within the “MeLOn—MachineLearning Models for Optimization” toolbox (https://git.rwth-aachen.de/avt.svt/public/MeLOn).

Designing preference elicitation (PE) methodologies that can quickly ascertain a user’s top item preferences in a cold-start setting is a key challenge for building effective and personalized conversational recommendation (ConvRec) systems. While large language models (LLMs) enable fully natural language (NL) PE dialogues, we hypothesize that monolithic LLM NL-PE approaches lack the multi-turn, decision-theoretic reasoning required to effectively balance the exploration and exploitation of user preferences towards an arbitrary item set. In contrast, traditional Bayesian optimization PE methods define theoretically optimal PE strategies, but cannot generate arbitrary NL queries or reason over content in NL item descriptions – requiring users to express preferences via ratings or comparisons of unfamiliar items. To overcome the limitations of both approaches, we formulate NL-PE in a Bayesian Optimization (BO) framework that seeks to actively elicit NL feedback to identify the best recommendation. Key challenges in generalizing BO to deal with natural language feedback include determining: (a) how to leverage LLMs to model the likelihood of NL preference feedback as a function of item utilities, and (b) how to design an acquisition function for NL BO that can elicit preferences in the infinite space of language. We demonstrate our framework in a novel NL-PE algorithm, PEBOL, which uses: 1) Natural Language Inference (NLI) between user preference utterances and NL item descriptions to maintain Bayesian preference beliefs, and 2) BO strategies such as Thompson Sampling (TS) and Upper Confidence Bound (UCB) to guide LLM query generation. We numerically evaluate our methods in controlled simulations, finding that after 10 turns of dialogue, PEBOL can achieve an MRR@10 of up to 0.27 compared to the best monolithic LLM baseline’s MRR@10 of 0.17, despite relying on earlier and smaller LLMs.1

Earth observation from satellite sensory data pose challenging problems, where machine learning is currently a key player. In recent years, Gaussian Process (GP) regression and other kernel methods have excelled in biophysical parameter estimation tasks from space. GP regression is based on solid Bayesian statistics, and generally yield efficient and accurate parameter estimates. However, GPs are typically used for inverse modeling based on concurrent observations and in situ measurements only. Very often a forward model encoding the well-understood physical relations is available though. In this work, we review three GP models that respect and learn the physics of the underlying processes in the context of inverse modeling. First, we will introduce a Joint GP (JGP) model that combines in situ measurements and simulated data in a single GP model. Second, we present a latent force model (LFM) for GP modeling that encodes ordinary differential equations to blend data-driven modeling and physical models of the system. The LFM performs multi-output regression, adapts to the signal characteristics, is able to cope with missing data in the time series, and provides explicit latent functions that allow system analysis and evaluation. Finally, we present an Automatic Gaussian Process Emulator (AGAPE) that approximates the forward physical model via interpolation, reducing the number of necessary nodes. Empirical evidence of the performance of these models will be presented through illustrative examples of vegetation monitoring and atmospheric modeling.

Bayesian optimization seeks the global optimum of a black-box, objective function f (x), in the fewest possible iterations. Recent work applied knowledge of the true value of the optimum to the Gaussian Process probabilistic model typically used in Bayesian optimization. This, together with a new acquisition function called Confidence Bound Minimization, resulted in a Gaussian probabilistic posterior in which the predictions were no greater than the known maximum (and no less than for minimum). Our novel work applies Confidence Bound Minimization to Bayesian optimization with Student's-t Processes, a probabilistic alternative which addresses known weaknesses in Gaussian Processes - outliers' probability and the calculation of posterior covariance. The new model is applied to the problem of hyperparameter tuning for an XGBoost classifier. Experiments show superior regret minimization and predictive accuracy, versus the popular Expected Improvement acquisition function. Combining Confidence Bound Minimization with a transformed Student's-t Process probabilistic model and known optima produces superior training regret minimization and posterior predictions for the Six-Hump Camel(2D) and Levy(4D) benchmark problems, which do not fall below true minima.

This paper describes a method combining Bayesian optimization (BO) and a lumped-capacitance thermal network model that is effective for speeding up the thermal design optimization of an electronic circuit board layout with transient heating chips. As electronic devices have become smaller and more complex, the importance of thermal design optimization to ensure heat dissipation performance has increased. However, such a thermal design optimization is difficult because various trade-offs associated with packaging and transient temperature changes of heat-generating components must be considered. This study aims to improve the performance of thermal design optimization by artificial intelligence. BO using a Gaussian process was combined with the lumped-capacitance thermal network model, and its performance was verified. As a result, BO successfully found the ideal circuit board layout as well as particle swarm optimization (PSO) and genetic algorithm (GA) could. The CPU time for BO was 1/5 and 1/4 of that for PSO and GA. In addition, BO found a non-intuitive optimal solution in approximately 7 min from 10 million layout patterns. It was estimated that this was 1/1000 of the CPU time required for analyzing all layout patterns.