Human-algorithm collaborative Bayesian optimization for engineering systems

Separation of the net ecosystem exchange of CO2 (F) into its component fluxes of net photosynthesis (FA) and nonfoliar respiration (FR) is important in understanding the physical and environmental controls on these fluxes, and how these fluxes may respond to environmental change. In this paper, we evaluate a partitioning method based on a combination of stable isotopes of CO2 and Bayesian optimization in the context of partitioning methods based on regressions with environmental variables. We combined high‐resolution measurements of stable carbon isotopes of CO2, ecosystem fluxes, and meteorological variables with a Bayesian parameter optimization approach to estimate FA and FR in a subalpine forest in Colorado, United States, over the course of 104 days during summer 2003. Results were generally in agreement with the independent environmental regression methods of Reichstein et al. (2005a) and Yi et al. (2004). Half‐hourly posterior parameter estimates of FA and FR derived from the Bayesian/isotopic method showed a strong diurnal pattern in both, consistent with established gross photosynthesis (GEE) and total ecosystem respiration (TER) relationships. Isotope‐derived FA was functionally dependent on light, but FR exhibited the expected temperature dependence only when the prior estimates for FR were temperature‐based. Examination of the posterior correlation matrix revealed that the available data were insufficient to independently resolve all the Bayesian‐estimated parameters in our model. This could be due to a small isotopic disequilibrium () between FA and FR, poor characterization of whole‐canopy photosynthetic discrimination or the isotopic flux (isoflux, analogous to net ecosystem exchange of 13CO2). The positive sign of indicates that FA was more enriched in 13C than FR. Possible reasons for this are discussed in the context of recent literature.

Abstarct Engineering design problems typically require optimizing a quality measure by finding the right combination of controllable input parameters. In Additive Manufacturing (AM), the output characteristics of the process can often be non-stationary functions of the process parameters. Bayesian Optimization (BO) is a methodology to optimize such “black-box” functions, i.e., the input–output relationship is unknown and expensive to compute. Optimization tasks involving “black-box” functions widely use BO with Gaussian Process (GP) regression surrogate model. Using GPs with standard kernels is insufficient for modeling non-stationary functions, while GPs with non-stationary kernels are typically over-parameterized. On the other hand, a Deep Gaussian Process (DGP) can overcome GPs’ shortcomings by considering a composition of multiple GPs. Inference in a DGP is challenging due to its structure resulting in a non-Gaussian posterior, and using DGP as a surrogate model for BO is not straightforward. Stochastic Imputation (SI)-based inference is promising in speed and accuracy for BO. This work proposes a bootstrap aggregation-based procedure to effectively utilize the SI-based inference for BO with a DGP surrogate model. The proposed BO algorithm DGP-SI-BO is faster and empirically better than the state-of-the-art BO method in optimizing non-stationary functions. Several analytical test functions and a case study in metal AM simulation demonstrate the applicability of the proposed method.

This study presents a data-driven approach for optimizing the operating altitude of Airborne Wind Energy (AWE) systems to maximize net energy production. Determining the optimal operating altitude of an AWE system is challenging, as the wind speed constantly varies with both time and altitude. Furthermore, without expensive auxiliary equipment, the wind speed is only measurable at the AWE system's operating altitude. The work presented in this paper shows how tools from machine learning can be blended with real-time control to optimize the AWE system's operating altitude efficiently, without the use of auxiliary wind profiling equipment. Specifically, Bayesian Optimization, which is a data-driven technique for finding the optimum of an unknown and expensive-to-evaluate objective function, is applied to the real-time control of an AWE system. The underlying objective function is modeled by a Gaussian Process (GP); then, Bayesian Optimization utilizes the predictive uncertainty information from the GP to decide the best subsequent operating altitude. In the AWE application, conventional Bayesian Optimization is extended to handle the time-varying nature of the wind shear profile (wind speed vs. time). Using real wind data, our method is validated against three baseline approaches. Our simulation results show that the Bayesian Optimization method is successful in dramatically increasing power production over these baselines.

Collision probability calculation is critical to space situational awareness or space traffic management. To determine the collision probability, the conjunction time needs to be determined first, but both collision probability and conjunction time usually exist large uncertainties. In this paper, a continuous representation of the polynomial chaos expansion (PCE)-based surrogate model is used to describe the uncertainty of space objects along with time. This PCE model facilitates the propagation of sampling points representing orbital uncertainty at any time. A machine learning-based Bayesian optimization method is employed to determine the conjunction time and then the collision probability. A data-driven Gaussian process regression model is built to approximate the Hausdorff distance, which is a distance metric between two space objects with uncertainty and is used as an objective function for Bayesian optimization. The proposed PCE model and Bayesian optimization do not require the Gaussian assumption and provide a general framework to calculate the conjunction time and collision probability. Two numerical experiments are used to show the effectiveness of the proposed algorithm and its close performance to the traditional Monte Carlo sampling-based method.

Gaussian Process based Bayesian Optimization is largely adopted for solving problems where the inputs are in Euclidean spaces. In this paper we associate the inputs to discrete probability distributions which are elements of the probability simplex. To search in the new design space, we need a distance between distributions. The optimal transport distance (aka Wasserstein distance) is chosen due to its mathematical structure and the computational strategies enabled by it. Both the GP and the acquisition function is generalized to an acquisition functional over the probability simplex. To optimize this functional two methods are proposed, one based on auto differentiation and the other based on proximal-point algorithm and the gradient flow. Finally, we report a preliminary set of computational results on a class of problems whose dimension ranges from 5 to 100. These results show that embedding the Bayesian optimization process in the probability simplex enables an effective algorithm whose performance over standard Bayesian optimization improves with the increase of problem dimensionality.

Driven by increased integration of renewable energy sources, the widespread decarbonization of power systems has led to energy price fluctuations that require greater adaptability and flexibility from grid users in order to maximize profits. Industrial loads equipped with flexible resources can optimize energy consumption rather than merely reacting to immediate events, thereby capitalizing on volatile energy prices. However, the absence of sufficient measured data in industrial processes limits the ability to fully harness this flexibility. To address this challenge, we presents a black-box optimization model for optimizing the energy consumption of cooling systems in the aquaculture industry using Extreme Gradient Boosting (XGBoost) and Bayesian Optimization (BO). XGBoost is employed to establish a nonlinear relationship between cooling system power consumption and available measured data. Based on this model, Bayesian Optimization with the Lower Confidence Bound (LCB) acquisition function is used to determine the optimal discharge temperature of water into breeding pools, minimizing day-ahead electricity costs. The proposed approach is validated using real-world data from a case study at the Port of Hirtshals, Denmark based on measurements from 2023. Our findings illustrate that leveraging the inherent flexibility of industrial processes can yield financial benefits while providing valuable signals for grid operators to adjust consumption behaviors through appropriate price mechanisms. Furthermore, machine learning techniques prove effective in optimizing energy consumption for industries with limited measured data, delivering accurate and practical estimations.

The simultaneous design of processes and enabling materials such as solvents, catalysts, and adsorbents is challenging because molecular‐ and process‐level decisions are strongly interdependent. Sequential approaches often yield suboptimal results since improvements in material properties may not translate into superior process performance. We propose a multiscale Bayesian Optimization (BO) framework that explicitly couples molecular‐ and process‐level design in a bi‐level structure. The inner loop solves deterministic process models, while the outer BO loop explores molecular descriptors, reducing dimensionality, improving sample efficiency, and handling process constraints. A case study on multiscale reactor and catalyst codesign demonstrates that this hybrid strategy achieves optimal performance with substantially lower computational effort than direct BO over the full decision space. BO, when aligned with the natural process—material hierarchy—thus provides an efficient methodology for multiscale design.

Bayesian optimization (BO) has demonstrated potential for optimizing control performance in data-limited settings, especially for systems with unknown dynamics or unmodeled performance objectives. The BO algorithm efficiently trades-off exploration and exploitation by leveraging uncertainty estimates using surrogate models. These surrogates are usually learned using data collected from the target dynamical system to be optimized. Intuitively, the convergence rate of BO is better for surrogate models that can accurately predict the target system performance. In classical BO, initial surrogate models are constructed using very limited data points, and therefore rarely yield accurate predictions of system performance. In this paper, we propose the use of meta-learning to generate an initial surrogate model based on data collected from performance optimization tasks performed on a variety of systems that are different to the target system. To this end, we employ deep kernel networks (DKNs) which are simple to train and which comprise encoded Gaussian process models that integrate seamlessly with classical BO. The effectiveness of our proposed DKN-BO approach for speeding up control system performance optimization is demonstrated using a well-studied nonlinear system with unknown dynamics and an unmodeled performance function.

Device sizing is a challenging problem for analog circuit design. Traditional methods depend on domain knowledge and intensive simulations to search for feasible parameters. Recent studies apply the Bayesian optimization (BO) and a Gaussian process (GP) model in analog circuit synthesis to improve efficiency. The BO framework automatically selects the parameter candidates by inferring the surrogate GP model. However, naive BO employs a sequential updating strategy which is inefficient in a multicore environment. Besides, the widely used GP model requires costly high fidelity data, which are obtained from fine simulations. In this article, we propose a constrained batch BO approach with a multifidelity (MF) model to solve the above difficulties. The batch BO exploits parallel computing and selects promising parameters by multiple acquisition function ensemble. In addition, the MF GP model adapts the low fidelity data obtained from coarse simulations. Specifically, the proposed method incorporates information gain in a weighted clustering algorithm to refine the parameter candidates. As a result, the proposed method maintains the candidates’ quality and diversity, which speeds up the optimization convergence. In the experiments, we demonstrate the efficiency of the proposed approach on three real-world circuits. The results show that our approach reduces the simulation costs by at least 54.6% compared to the state-of-the-art baselines.

In tackling frequent batch anomalies in manufacturing processes, this study introduces a novel semi-supervised in-process monitoring method that enables real-time anomaly detection during production rather than post-process inspection. By integrating physically-informed Bayesian feature optimization with semi-supervised learning, the proposed method captures batch defects under conditions of scarce anomaly samples. First, a Bayesian feature optimization strategy is proposed to extract critical indicators from redundant industrial signals, effectively bridging the gap between domain expertise and data-driven representations. Second, to address the semi-supervised challenge of scarce anomaly samples, an out-of-distribution detection model trained on normal samples is established, combined with conformal calibration to determine robust decision thresholds. The superiority of the proposed method is validated with real-world stamping and injection molding datasets, with ablation studies confirming robustness to hyperparameter variations. The results demonstrate the potential of the proposed method for anomaly monitoring in data-scarce industrial contexts.

While Bayesian Optimization (BO) is a very popular method for optimizing expensive black-box functions, it fails to leverage the experience of domain experts. This causes BO to waste function evaluations on bad design choices (e.g., machine learning hyperparameters) that the expert already knows to work poorly. To address this issue, we introduce Bayesian Optimization with a Prior for the Optimum (BOPrO). BOPrO allows users to inject their knowledge into the optimization process in the form of priors about which parts of the input space will yield the best performance, rather than BO’s standard priors over functions, which are much less intuitive for users. BOPrO then combines these priors with BO’s standard probabilistic model to form a pseudo-posterior used to select which points to evaluate next. We show that BOPrO is around \(6.67\times \) faster than state-of-the-art methods on a common suite of benchmarks, and achieves a new state-of-the-art performance on a real-world hardware design application. We also show that BOPrO converges faster even if the priors for the optimum are not entirely accurate and that it robustly recovers from misleading priors.

This paper proposes a Sequential Model Based Optimization framework for solving optimization problems characterized by a black-box, multi-extremal, expensive and partially defined objective function, under unknown constraints. This is a typical setting for simulation-optimization problems, where the objective function cannot be computed for some configurations of the decision/control variables due to the violation of some (unknown) constraint. The framework is organized in two consecutive phases, the first uses a Support Vector Machine classifier to approximate the boundary of the unknown feasible region within the search space, the second uses Bayesian Optimization to find a globally optimal (feasible) solution. A relevant difference with traditional Bayesian Optimization is that the optimization process is performed on the estimated feasibility region, only, instead of the entire search space. Some results on three 2D test functions and a real case study for the Pump Scheduling Optimization in Water Distribution Networks are reported. The proposed framework proved to be more effective and efficient than Bayesian Optimization approaches using a penalty for function evaluations outside the feasible region.

In Numerical Weather Prediction (NWP) models, such as the Weather Research and Forecasting (WRF) model, parameter uncertainty in physics parameterization schemes significantly impacts model output. Our study adopts a Bayesian probabilistic approach, building on prior research that identified temperature (T) and relative humidity (Rh) as sensitive to three key WRF parameters during southeast Australia's extreme heat events. Using Gaussian process regression‐based Bayesian Optimisation (G‐BO), we accurately estimated the optimal distributions for these parameters. Results show that the default values are outside their corresponding optimal distribution bounds for two of the three parameters, suggesting the need to reconsider these default values. Additionally, the robustness of the optimal parameter distributions is validated by their application to an independent extreme heat event, not included in the optimisation process. In this test, the optimized parameters substantially improved the simulation of T and Rh, highlighting their effectiveness in enhancing simulation accuracy during extreme heat conditions.

Data-driven design shows the promise of accelerating materials discovery but is challenging due to the prohibitive cost of searching the vast design space of chemistry, structure, and synthesis methods. Bayesian Optimization (BO) employs uncertainty-aware machine learning models to select promising designs to evaluate, hence reducing the cost. However, BO with mixed numerical and categorical variables, which is of particular interest in materials design, has not been well studied. In this work, we survey frequentist and Bayesian approaches to uncertainty quantification of machine learning with mixed variables. We then conduct a systematic comparative study of their performances in BO using a popular representative model from each group, the random forest-based Lolo model (frequentist) and the latent variable Gaussian process model (Bayesian). We examine the efficacy of the two models in the optimization of mathematical functions, as well as properties of structural and functional materials, where we observe performance differences as related to problem dimensionality and complexity. By investigating the machine learning models' predictive and uncertainty estimation capabilities, we provide interpretations of the observed performance differences. Our results provide practical guidance on choosing between frequentist and Bayesian uncertainty-aware machine learning models for mixed-variable BO in materials design.

In this research a Collaborative Optimization (CO) approach for multidisciplinary systems design is used to develop a decision based design framework for non-deterministic optimization. To date CO strategies have been developed for use in application to deterministic systems design problems. In this research the decision based design (DBD) framework proposed by Hazelrigg [1,2] is modified for use in a collaborative optimization framework. The Hazelrigg framework as originally proposed provides a single level optimization strategy that combines engineering decisions with business decisions in a single level optimization. By transforming this framework for use in collaborative optimization one can decompose the business and engineering decision making processes. In the new multilevel framework of Decision Based Collaborative Optimization (DBCO) the business decisions are made at the system level. These business decisions result in a set of engineering performance targets that disciplinary engineering design teams seek to satisfy as part of subspace optimizations. The Decision Based Collaborative Optimization framework more accurately models the existing relationship between business and engineering in multidisciplinary systems design.