Multi-objective optimization using Deep Gaussian Processes: Application to Aerospace Vehicle Design

Abstarct Engineering design problems typically require optimizing a quality measure by finding the right combination of controllable input parameters. In Additive Manufacturing (AM), the output characteristics of the process can often be non-stationary functions of the process parameters. Bayesian Optimization (BO) is a methodology to optimize such “black-box” functions, i.e., the input–output relationship is unknown and expensive to compute. Optimization tasks involving “black-box” functions widely use BO with Gaussian Process (GP) regression surrogate model. Using GPs with standard kernels is insufficient for modeling non-stationary functions, while GPs with non-stationary kernels are typically over-parameterized. On the other hand, a Deep Gaussian Process (DGP) can overcome GPs’ shortcomings by considering a composition of multiple GPs. Inference in a DGP is challenging due to its structure resulting in a non-Gaussian posterior, and using DGP as a surrogate model for BO is not straightforward. Stochastic Imputation (SI)-based inference is promising in speed and accuracy for BO. This work proposes a bootstrap aggregation-based procedure to effectively utilize the SI-based inference for BO with a DGP surrogate model. The proposed BO algorithm DGP-SI-BO is faster and empirically better than the state-of-the-art BO method in optimizing non-stationary functions. Several analytical test functions and a case study in metal AM simulation demonstrate the applicability of the proposed method.

We develop a novel framework to accelerate Gaussian process regression (GPR). In particular, we consider localization kernels at each data point to down-weigh the contributions from other data points that are far away, and we derive the GPR model stemming from the application of such localization operation. Through a set of experiments, we demonstrate the competitive performance of the proposed approach compared to full GPR, other localized models, and deep Gaussian processes. Crucially, these performances are obtained with considerable speedups compared to standard global GPR due to the sparsification effect of the Gram matrix induced by the localization operation.

In multi-objective Bayesian optimization, an infill criterion is an important part, as it is the indicator to evaluate how much good a new set of solutions is, compared to a Pareto-front approximation set. This paper presents a deterministic algorithm for computing the Expected R2 Indicator for bi-objective problems and studies its use as an infill criterion in Bayesian Global Optimization. The R2-Indicator was introduced in 1998 by M. Hansen and A. Jaszkiewicz for performance assessment in multi-objective optimization and is more recently also used in indicator-based multi-criterion evolutionary algorithms (IBEAs). In Bayesian Global Optimization, we propose the Expected R2-indicator Improvement (ER2I) as an infill criterion. It is defined as the expected decrease of the R2 indicator by a point that is sampled from a predictive Gaussian distribution. The ER2I can also be used as a pre-selection criterion in surrogate-assisted IBEAs. It provides an alternative to the Expected Hypervolume-Indicator Improvement (EHVI) that requires a reference point, bounding the Pareto front from above. In contrast, the ER2I works with a utopian reference point that bounds the Pareto front from below. In addition, the ER2I supports preference modelling with utility functions and its computation time grows only linearly with the number of considered weight combinations. It is straightforward to approximate the ER2I by Monte Carlo Integration, but so far a deterministic algorithm to solve the non-linear integral remained unknown. We outline a deterministic algorithm for the computation of the bi-objective ER2I with Chebychev utility functions. Moreover, we study monotonicity properties of the ER2I w.r.t. parameters of the predictive distribution and numerical simulations demonstrate fast convergence to Pareto fronts of different shapes and the ability of the ER2I Bayesian optimization to fill gaps in the Pareto front approximation.

Background: Immune checkpoint inhibitors (ICIs) targeting PD-1 or PD-L1 have dramatically improved survival of patients with gastric cancer (GC). However, recent clinical trials, including KEYNOTE-059, KEYNOTE-061, and ATTRACTION-2, show that the objective response rate (ORR) for ICIs is still less than 20% in patients with advanced GC. Therefore, predicting who will respond to ICIs in advance is essential to make better treatment plans for GC patients. There are potential biomarkers to predict ICI response, but most of them can only be applied to a limited population of patients. For example, Kim ST., et al., (PMID: 30013197) show that patients with MSI-high or EBV-positive responded well to ICIs, but these subtypes account for less than 30 percent of GC patients (PMID:25079317). In this study, we propose a prediction model of ICI response using the 32 GC-specific genes that we identified from mutation data (PMID: 26635139) and showed their close association with survival and ICI response in GC (PMID: 35140202; PMID: 36508166). In addition, since the number of available GC-ICI samples is small, we design our prediction model in Bayesian and ensemble learning frameworks which are robust to small sample size problems. Methods: We collected mRNA sequencing data from GC-ICI treated patients from our institutions and publications (PMID: 30013197; PMID: 35254400; PMID: 33846173). We used 108 samples previously included in our previous work (PMID: 36508166) as the training data and 27 samples (13 responders and 14 non-responders) from (PMID: 3384617) as the test data. We built and trained a classifier to separate responders from non-responders using the expressions of the 32 genes. We proposed to use a Deep Gaussian process (GP) model, a multi-layered extension of a GP that is itself a Bayesian method. We optimized the parameters of each GP model in variational inference. To implement ensemble learning, we trained multiple DGP models with different initializations of the model parameters and data splitting (training/validation sets). For each training run, the best model was selected based on the prediction performance on the validation set, measured by AUC. The final prediction for a test sample was made by combining the prediction results of all the DGP models by taking the average. Results: We implemented the DGP-based prediction model using PyTorch in Python, and the codes will be publicly available on the GitHub repository (https://github.com/hwanglab/GC_ICI_response_prediction). We achieved a prediction performance of 0.813 AUC on the test data (sensitivity = 0.769 and specificity = 0.714). Conclusion: We propose the prediction model of ICI response, which can handle small sample size problems in the framework of Bayesian and ensemble learning. The experimental results using the GC-ICI data demonstrate that the 32 genes can be used as predictive markers for ICI response in GC. Citation Format: Sunho Park, Minji Kim, Changjin Hong, Tae Hyun Hwang. A deep Gaussian process (DGP)-based prediction model for immunecheckpoint inhibitor (ICI) response in gastric cancer (GC) [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2024; Part 1 (Regular Abstracts); 2024 Apr 5-10; San Diego, CA. Philadelphia (PA): AACR; Cancer Res 2024;84(6_Suppl):Abstract nr 2316.

Pressure swing adsorption (PSA) is an energy-efficient technology for gas separation, while the multi-objective optimization of PSA is a challenging task. To tackle this, we propose a hybrid optimization framework, which integrates three steps. In the first step, we establish surrogate models for the constraints using Gaussian processes (GPs) and employ multi-objective Bayesian optimization to search for feasible points that satisfy the constraints. In the second step, we establish surrogate models for the objective function and constraints using GPs and utilize constrained multi-objective Bayesian optimization to search for an approximate Pareto front. In the third step, we perform a local search based on the approximate Pareto front. By employing the trust region filter method, we construct quadratic models for each constraint and objective function and refine the Pareto front to achieve local optimality. This framework demonstrates the efficiency of Bayesian optimization and the local optimality of the trust region method. A comparison with the popular evolutionary algorithm, Nondominated Sorting Genetic Algorithm II (NSGA-II), showed that this framework had a higher hypervolume of the Pareto front while halving the runtime and reducing the number of simulations by a factor of 20.

Deep Gaussian processes (DGPs) are increasingly popular as predictive models in machine learning for their nonstationary flexibility and ability to cope with abrupt regime changes in training data. Here, we explore DGPs as surrogates for computer simulation experiments whose response surfaces exhibit similar characteristics. In particular, we transport a DGP’s automatic warping of the input space and full uncertainty quantification, via a novel elliptical slice sampling Bayesian posterior inferential scheme, through to active learning strategies that distribute runs nonuniformly in the input space—something an ordinary (stationary) GP could not do. Building up the design sequentially in this way allows smaller training sets, limiting both expensive evaluation of the simulator code and mitigating cubic costs of DGP inference. When training data sizes are kept small through careful acquisition, and with parsimonious layout of latent layers, the framework can be both effective and computationally tractable. Our methods are illustrated on simulation data and two real computer experiments of varying input dimensionality. We provide an open source implementation in the deepgp package on CRAN.

Despite numerous works over the past two decades, friction-induced vibrations, especially braking noises, are a major issue for transportation manufacturers as well as for the scientific community. To study these fugitive phenomena, the engineers need numerical methods to efficiently predict the mode coupling instabilities in a multiparametric context. The objective of this paper is to approximate the unstable frequencies and the associated damping rates extracted from a complex eigenvalue analysis under variability. To achieve this, a deep Gaussian process is considered to fit the non-linear and non-stationary evolutions of the real and imaginary parts of complex eigenvalues. The current challenge is to build an efficient surrogate modelling, considering a small training set. A discussion about the sample distribution density effect, the training set size and the kernel function choice is proposed. The results are compared to those of a Gaussian process and a deep neural network. A focus is made on several deceptive predictions of surrogate models, although the better settings were well chosen in theory. Finally, the deep Gaussian process is investigated in a multiparametric analysis to identify the best number of hidden layers and neurons, allowing a precise approximation of the behaviours of complex eigensolutions.

Music is a powerful force that evokes human emotions. Several investigations of music emotion recognition (MER) have been conducted in recent years. This paper proposes a system for detecting emotion in music that is based on a deep Gaussian process (GP). The system consists of two parts-feature extraction and classification. In the feature extraction part, five types of features that are associated with emotions are selected for representing the music signal; these are rhythm, dynamics, timbre, pitch and tonality. A music clip is decomposed into frames and these features are extracted from each frame. Next, statistical values, such as mean and standard deviation, of frame-based features are calculated to generate a 38-dimensional feature vector. In the classification part, a deep GP is utilized for emotion recognition. We treat classification problem from the perspective of regression. Finally, 9 classes of emotion are categorized by 9 one-versus-all classifiers. The experimental results demonstrate that the proposed system performs well in emotion recognition.

Financial Applications of Gaussian Processes and Bayesian Optimization

Gaussian process regression is a well-established Bayesian machine learning method. We propose a new approach to Gaussian process regression using quantum kernels based on parameterized quantum circuits. By employing a hardware-efficient feature map and careful regularization of the Gram matrix, we demonstrate that the variance information of the resulting quantum Gaussian process can be preserved. We also show that quantum Gaussian processes can be used as a surrogate model for Bayesian optimization, a task that critically relies on the variance of the surrogate model. To demonstrate the performance of this quantum Bayesian optimization algorithm, we apply it to the hyperparameter optimization of a machine learning model which performs regression on a real-world dataset. We benchmark the quantum Bayesian optimization against its classical counterpart and show that quantum version can match its performance.

Deep Gaussian processes (DGPs) are the natural extension of Gaussian processes (GPs) to a multi-layer architecture. DGPs are powerful probabilistic models that have shown better results than standard GPs in terms of generalization performance and prediction uncertainty estimation. Nevertheless, exact inference in DGPs is intractable, making these models hard to train. For this task, current approaches in the literature use approximate inference techniques such as variational inference or approximate expectation propagation. In this work, we present a new method for inference in DGPs using an approximate inference technique based on Monte Carlo methods and the expectation propagation algorithm. Our experiments show that our method scales well to large datasets and that its performance is comparable or better than other state of the art methods. Furthermore, our training method leads to interesting properties in the predictive distribution of the DGP. More precisely, it is able to capture output noise that is dependent on the input and it can generate multimodal predictive distributions. These two properties, which are not shared by other state-of-the-art approximate inference methods for DGPs, are analyzed in detail in our experiments.

Due to the adversarial limitations between the output solutions in the multi-objective optimization stage, the energy efficiency optimization of large pumping stations has the disadvantage of difficult coordination between power consumption and efficiency. Therefore, a multi-objective Bayesian hyperparameter optimization algorithm design for energy efficiency optimization of large pumping stations has been carried out. A multi-objective optimization model for energy efficiency of large pumping stations was constructed by combining the input power of the main motor, transmission energy loss, transformer loss, and energy consumption of auxiliary equipment, with efficiency as the constraint. We extract sequence features that affect pump water demand, and use feature labels as hyperparameters for the multi-objective optimization model of energy efficiency in large pumping stations. We transform the multi-objective optimization problem into a hyperparameter optimization problem, avoiding adversarial output solutions. We iterate the query points collected by EI, so the output model reaches a convergent optimal solution. The test results show that the designed algorithm can improve the pumping efficiency of the pump station while controlling the power consumption of the pump station, achieving collaborative optimization of efficiency.

Deep Gaussian process regression for lithium-ion battery health prognosis and degradation mode diagnosis

In this paper, we present DeepMap, a deep Gaussian process for indoor radio map construction and location estimation. To address the shortcomings of existing Gaussian process based approaches, we present a DeepMap system, which employs deep Gaussian process for constructing received signal strength (RSS) radio maps and a Bayesian algorithm for online localization. We design a two-layer deep Gaussian process model to capture the relationship between the RSS space and the location space and provide an offline Bayesian training method to determine model parameters. A Bayesian fusion method using multiple APs is proposed for accurate location estimation. Experimental results verify the performances of DeepMap in a large indoor environment and validate its robustness with moderate training data.

Mechanical property prediction plays a crucial role in the steel industry, enabling better materials selection and enhancing production efficiency. In this study, we propose a novel framework that facilitates optimal materials selection for steel’s alloying elements, based on accurate predictions of the Jominy hardness curve. Leveraging Gaussian Process (GP) regression, we provide probabilistic predictions of steel hardenability characteristics from alloy element composition. Taking it a step further, our framework incorporates these accurate predictions into a constrained optimization process, yielding optimal compositions that reduce overall spending while meeting performance specifications. Through data obtained from 1080 steel samples, our GP regression model exhibits high accuracy, achieving an RMSE of 1.37 and showcasing significant improvements in the field. Moreover, our constrained optimization utilizing the GP model and historical market data reveals an average cost reduction of 18% on alloying element expenses, highlighting the tangible cost-saving potential of this approach. By leveraging Gaussian Process (GP) regression, we not only achieve accurate predictions of the Jominy hardness curve based on alloy element composition, but we also introduce a crucial element of uncertainty quantification. This empowers us to place trust in the results of our optimization process, ensuring robust and reliable materials selection. The integration of GP regression and optimization provides a powerful tool for achieving cost-effective materials selection and marks a significant advancement compared to existing studies. This research underscores the promise of machine learning in the steel industry, demonstrating its ability to yield substantial cost savings and enhance decision-making in materials selection.