Optimizing Closed-Loop Performance with Data from Similar Systems: A Bayesian Meta-Learning Approach

Hyper-Parameter Optimization (HPO) occupies a fundamental role in Deep Learning systems due to the number of hyper-parameters (HPs) to be set. The state-of-the-art of HPO methods are Grid Search, Random Search and Bayesian Optimization. The first two methods try all possible combinations and random combination of the HPs values, respectively. This is performed in a blind manner, without any information for choosing the new set of HPs values. Bayesian Optimization (BO), instead, keeps track of past results and uses them to build a probabilistic model mapping HPs into a probability density of the objective function. Bayesian Optimization builds a surrogate probabilistic model of the objective function, finds the HPs values that perform best on the surrogate model and updates it with new results. In this paper, we improve BO applied to Deep Neural Network (DNN) by adding an analysis of the results of the network on training and validation sets. This analysis is performed by exploiting rule-based programming, and in particular by using Probabilistic Logic Programming. The resulting system, called Symbolic DNN-Tuner, logically evaluates the results obtained from the training and the validation phase and, by applying symbolic tuning rules, fixes the network architecture, and its HPs, therefore improving performance. We also show the effectiveness of the proposed approach, by an experimental evaluation on literature and real-life datasets.

Abstarct Engineering design problems typically require optimizing a quality measure by finding the right combination of controllable input parameters. In Additive Manufacturing (AM), the output characteristics of the process can often be non-stationary functions of the process parameters. Bayesian Optimization (BO) is a methodology to optimize such “black-box” functions, i.e., the input–output relationship is unknown and expensive to compute. Optimization tasks involving “black-box” functions widely use BO with Gaussian Process (GP) regression surrogate model. Using GPs with standard kernels is insufficient for modeling non-stationary functions, while GPs with non-stationary kernels are typically over-parameterized. On the other hand, a Deep Gaussian Process (DGP) can overcome GPs’ shortcomings by considering a composition of multiple GPs. Inference in a DGP is challenging due to its structure resulting in a non-Gaussian posterior, and using DGP as a surrogate model for BO is not straightforward. Stochastic Imputation (SI)-based inference is promising in speed and accuracy for BO. This work proposes a bootstrap aggregation-based procedure to effectively utilize the SI-based inference for BO with a DGP surrogate model. The proposed BO algorithm DGP-SI-BO is faster and empirically better than the state-of-the-art BO method in optimizing non-stationary functions. Several analytical test functions and a case study in metal AM simulation demonstrate the applicability of the proposed method.

Surrogate models are used to dramatically improve the design efficiency of numerical aerodynamic shape optimization, where high-fidelity, expensive computational fluid dynamics (CFD) is often employed. Traditionally, in adaptation, only one single sample point is chosen to update the surrogate model during each updating cycle, after the initial surrogate model is built. To enable the selection of multiple new samples at each updating cycle, a few parallel infilling strategies have been developed in recent years, in order to reduce the optimization wall clock time. In this article, an alternative parallel infilling strategy for surrogate-based constrained optimization is presented and demonstrated by the aerodynamic shape optimization of transonic wings. Different from existing methods in which multiple sample points are chosen by a single infill criterion, this article uses a combination of multiple infill criteria, with each criterion choosing a different sample point. Constrained drag minimizations of the ONERA-M6 and DLR-F4 wings are exercised to demonstrate the proposed method, including low-dimensional (6 design variables) and higher-dimensional problems (up to 48 design variables). The results show that, for surrogate-based optimization of transonic wings, the proposed method is more effective than the existing parallel infilling strategies, when the number of initial sample points are in the range from Nv to 8Nv (Nv here denotes the number of design variables). Each case is repeated 50 times to eliminate the effect of randomness in our results.

Surrogate models have been used to replace computationally expensive analysis models in engineering design problems. However, time-dependent variables and historical data are usually ignored in the surrogate modeling process. For instance, in a dam network design, using hydraulic simulations to estimate the water flow is computationally expensive, and the data is in the form of time series. So, we need time-dependent surrogate models to replace these simulations and manage this computational complexity. In this paper, we describe surrogate models to predict the amount of water flow into a reservoir. The challenge is that the flow is a time-dependent variable, and we need to incorporate time series into surrogate models. Thus, there are three contributions: (1) using surrogate modeling to predict flow for dam network design, (2) incorporating time series analysis in surrogate models for water network design, (3) using an ensemble of surrogates to increase the accuracy of prediction. We also demonstrate how to integrate surrogate models and machine learning with time series analysis for more accurate and faster prediction. Due to the availability of data, we use the Buffalo Reservoir in the Red River Basin as an example. Based on the time series data for flow, evaporation, precipitation, and maximum and minimum temperature, three surrogate models are used to examine the impact of integrating time series into surrogate models. These are multivariate autoregressive integrated moving average (MARIMA), a classic time series analysis method; artificial neural network (ANN), and random forest (RF) methods, two machine learning surrogate models. We use seven different time lags as features within an RF model, as an ensemble of surrogate models, and predict the flow for seven-time steps ahead. We successfully incorporate the time series data and particularly the concept of the time lag within surrogate models. We show that RF as the ensemble of surrogates provides more accurate predictions than the other two surrogate models. Although this method has been demonstrated for the Red River Basin, it could also be applied to designing anything in which time-dependent flow is an issue, for example, in biomedical applications, the management of manufacturing processes and product sales as well as any products in which fluid flow is an issue.

Unlike circuit parameter and sizing optimizations, the automated design of analog circuit topologies poses significant challenges for learning-based approaches. One challenge arises from the combinatorial growth of the topology space with circuit size, which limits the topology optimization efficiency. Moreover, traditional circuit evaluation methods are time-consuming, while the presence of data discontinuity in the topology space makes the accurate prediction of circuit performance exceptionally difficult for unseen topologies. To tackle these challenges, we design a novel Graph-Transformer-based Network (GTN) as the surrogate model for circuit evaluation, offering a substantial acceleration in the speed of circuit topology optimization without sacrificing performance. Our GTN model architecture is designed to embed voltage changes in circuit loops and current flows in connected devices, enabling accurate performance predictions for circuits with unseen topologies. To address the cold start problem when scaling GTN to large-scale circuits, we further introduce a curriculum learning strategy that progressively trains GTN from small-scale to large-scale circuits. This approach enables the model to first learn fundamental physical principles from simpler topologies and gradually adapt to complex configurations, effectively bridging the circuit complexity gap and improving prediction accuracy. Taking the power converter circuit design as an experimental task, our GTN model significantly outperforms an analytical approach and baseline methods directly utilizing graph neural networks. Furthermore, GTN achieves less than 5% relative error and 196 × speed-up compared with high-fidelity simulation. Notably, our GTN surrogate model empowers an automatic circuit design framework to discover circuits of comparable quality to those identified through high-fidelity simulation while reducing the time required by up to 98.2%. With curriculum learning, the enhanced GTN achieves a 51% improvement for performance prediction of large-scale circuits compared to the GTN model without this strategy. These advancements establish GTN as a scalable framework for automated analog circuit design across varying circuit complexity levels.

Bayesian Optimization Analysis of Containment-Venting Operation in a Boiling Water Reactor Severe Accident

A two-stage ensemble Kalman filter based on multiscale model reduction for inverse problems in time fractional diffusion-wave equations

Device sizing is a challenging problem for analog circuit design. Traditional methods depend on domain knowledge and intensive simulations to search for feasible parameters. Recent studies apply the Bayesian optimization (BO) and a Gaussian process (GP) model in analog circuit synthesis to improve efficiency. The BO framework automatically selects the parameter candidates by inferring the surrogate GP model. However, naive BO employs a sequential updating strategy which is inefficient in a multicore environment. Besides, the widely used GP model requires costly high fidelity data, which are obtained from fine simulations. In this article, we propose a constrained batch BO approach with a multifidelity (MF) model to solve the above difficulties. The batch BO exploits parallel computing and selects promising parameters by multiple acquisition function ensemble. In addition, the MF GP model adapts the low fidelity data obtained from coarse simulations. Specifically, the proposed method incorporates information gain in a weighted clustering algorithm to refine the parameter candidates. As a result, the proposed method maintains the candidates’ quality and diversity, which speeds up the optimization convergence. In the experiments, we demonstrate the efficiency of the proposed approach on three real-world circuits. The results show that our approach reduces the simulation costs by at least 54.6% compared to the state-of-the-art baselines.

Electrochemical machining is a promising non-traditional manufacturing process to make high-quality parts. The benefits of minimal thermally and mechanically induced stresses, free of burr, and a low surface roughness are appealing for industry and research institutes. However, the combined chemical reaction, electric field, fluid mechanics, and material properties involve a significant number of independent parameters which are difficult to analyze in order to draw comprehensive conclusions. To our current knowledge, process responses such as the material removal rate, optimal feed rate, and cutting profile cannot be represented accurately by analytical solutions. In recent years, deep learning has had tremendous success in analyzing sophisticated systems. The improved computation efficiency and reduced size of the training dataset required for deep learning have enabled various prediction models in the manufacturing industry. In this paper, a new approach is developed using the deep convolutional network with the Bayesian optimization algorithm to predict the diameters of the drilled hole from an electrochemical machining process. The Keras application programming interface (API) was used to build the deep convolutional network; the feed rate, pulse-on time, and voltage were used as input parameters to provide a fair comparison with a neural network from previous research. Random dropout layers were added to prevent overfitting of the network. Instead of tuning the network parameter by trial and error, the Bayesian parameter optimization algorithm was implemented to find the optimal set of parameters of the deep convolutional network that yields the minimum mean square error. The proposed algorithm was compared with a previously developed neural network with partially embedded physical knowledge. Improved training speed and accuracy were observed in comparison with the traditional neural network. The prediction model using the proposed deep learning algorithm demonstrated better prediction accuracy and provided a more systematic way to select the hyperparameter for the deep convolutional network.

Multivariate conditional regression using a Bayesian estimation approach for the development of a surrogate dissolution model as part of a comprehensive Process Analytical Technology (PAT) release strategy is the subject of this chapter. An appropriate experimental design provides the basis for constructing a process model in the first step, relating dissolution variables as the response to the critical process parameters (CPPs). Dissolution variables are either: Case 1: in vitro release measurements at one or several specific dissolution time points, or Case 2: the three parameters of the Weibull function describing the full profile at the tablet or batch or run average level. The dissolution variables are augmented with the measured API Content-by-NIR of the tablet to form the multivariate response vector in the process model. Then a conditional regression of the dissolution variables given API Content-by-NIR content and process parameters settings in the second step yields a predictive surrogate model of the dissolution variables. In the first case, the surrogate model directly yields predictions of the in vitro release measurement at the specific time point(s). In the second case, Weibull parameters are predicted, yielding a mean dissolution prediction at any time point on the dissolution profile. The Bayesian estimation approach to the conditional regression model permits simulations that can characterize the analytical performance of the surrogate model with respect to accuracy and precision for model qualification purposes. Two examples are given which show the statistical modeling details of both cases.

This paper presents a research methodology for identifying the Pratt & Whitney Canada PW127G turboprop engine from simulation data using optimized feedforward neural networks (FNN). A set of measurable variables - ground speed, throttle lever angle, pressure altitude ​, high-pressure spool speed , and propeller speed ​ - is used to predict normalized engine torque, providing a surrogate engine model suitable for integration into flight simulators. The methodology follows a two-stage strategy. First, a baseline L-BFGS–trained FNN is combined with two architecture-search methods, Extended Great Deluge (EGD) and Bayesian Optimization (BO). On the turboprop dataset, BO achieves a lower test RMSE than EGD and is therefore selected as the preferred architecture optimization strategy. Second, BO is fixed and used to optimize two FNN configurations: Baseline FNN with inputs (ground speed, throttle lever angle, pressure altitude , propeller speed ​) and Core-Enhanced FNN additionally including high-pressure spool speed . The optimized Core-Enhanced FNN significantly reduces the root mean square error from 1.066 to 0.4834 on testing data, corresponding to an average error reduction of about 55% compared with Baseline FNN, and also decreases mean relative error and error variance, confirming the importance of core-speed information for high-fidelity torque prediction. The results demonstrate that L-BFGS–trained FNNs, combined with BO-based architecture search and simulation-derived data, provide an effective and computationally efficient surrogate engine model for turboprop torque (and indirectly thrust) estimation in advanced flight simulation and training applications.

Activity-based transportation models simulate demand and supply as a complex system and therefore large set of parameters need to be adjusted. One such model is Preday activity-based model that requires adjusting a large set of parameters for its calibration on new urban environments. Hence, the calibration process is time demanding, and due to costly simulations, various optimisation methods with dimensionality reduction and stochastic approximation are adopted. This study adopts Bayesian Optimisation for Likelihood-free Inference (BOLFI) method for calibrating the Preday activity-based model to a new urban area. Unlike the traditional variant of the method that uses Gaussian Process as a surrogate model for approximating the likelihood function through modelling discrepancy, we apply a composite surrogate model that encompasses Random Forest surrogate model for modelling the discrepancy and Gaussian Mixture Model for estimating the its density. The results show that the proposed method benefits the extension and improves the general applicability to high-dimensional settings without losing the efficiency of the Bayesian Optimisation in sampling new samples towards the global optima.

Bayesian optimization (BO) based on the Gaussian process (GP) surrogate model has attracted extensive attention in the field of optimization and design of experiments (DoE). It usually faces two problems: the unstable GP prediction due to the ill-conditioned Gram matrix of the kernel and the difficulty of determining the trade-off parameter between exploitation and exploration. To solve these problems, we investigate the K-optimality, aiming at minimizing the condition number. Firstly, the Sequentially Bayesian K-optimal design (SBKO) is proposed to ensure the stability of the GP prediction, where the K-optimality is given as the acquisition function. We show that the SBKO reduces the integrated posterior variance and maximizes the hyper-parameters’ information gain simultaneously. Secondly, a K-optimal enhanced Bayesian Optimization (KO-BO) approach is given for the optimization problems, where the K-optimality is used to define the trade-off balance parameters which can be output automatically. Specifically, we focus our study on the K-optimal enhanced Expected Improvement algorithm (KO-EI). Numerical examples show that the SBKO generally outperforms the Monte Carlo, Latin hypercube sampling, and sequential DoE approaches by maximizing the posterior variance with the highest precision of prediction. Furthermore, the study of the optimization problem shows that the KO-EI method beats the classical EI method due to its higher convergence rate and smaller variance.

Gaussian process regression is a well-established Bayesian machine learning method. We propose a new approach to Gaussian process regression using quantum kernels based on parameterized quantum circuits. By employing a hardware-efficient feature map and careful regularization of the Gram matrix, we demonstrate that the variance information of the resulting quantum Gaussian process can be preserved. We also show that quantum Gaussian processes can be used as a surrogate model for Bayesian optimization, a task that critically relies on the variance of the surrogate model. To demonstrate the performance of this quantum Bayesian optimization algorithm, we apply it to the hyperparameter optimization of a machine learning model which performs regression on a real-world dataset. We benchmark the quantum Bayesian optimization against its classical counterpart and show that quantum version can match its performance.

Rather than optimizing expensive objective functions such as complex engineering simulations directly, Bayesian optimization methodologies fit a surrogate model (typically Kriging or a Gaussian Process) on evaluations of the objective function(s). To determine the next evaluation, an acquisition function is optimized (also referred to as infill criterion or sampling policy) which incorporates the model prediction and uncertainty and balances exploration and exploitation. Therefore, Bayesian optimization methodologies replace a single optimization of the objective function by a sequence of optimization problems: this makes sense as the acquisition function is cheap-to-evaluate whereas the objective is not. Depending on the goal different acquisition functions are available: multi-objective acquisition functions are relatively new and this chapter gives a state-of-the-art overview and illustrates some approaches based on hypervolume improvement. It is shown that the quality of the model is crucial for the performance of Bayesian optimization and illustrate this by using the more flexible Student-t processes as surrogate models.