Hierarchical Gaussian Process Models for Improved Metamodeling

Simulations are often used for the design of complex systems as they allow to explore the design space without the need to build several prototypes. Over the years, the simulation accuracy, as well as the associated computational cost has increased significantly, limiting the overall number of simulations during the design process. Therefore, metamodeling aims to approximate the simulation response with a cheap-to-evaluate mathematical approximation, learned from a limited set of simulator evaluations. Kernel-based methods using stationary kernels are nowadays wildly used. However, using stationary kernels for non-stationary responses can be inappropriate and result in poor models when combined with sequential design. We present the application of a novel kernel-based technique, known as Deep Gaussian Processes, which is better able to cope with these difficulties. We evaluate the method for non-stationary regression on a series of real-world problems, showing that it outperforms the standard Gaussian Processes with stationary kernels.

Abstarct Engineering design problems typically require optimizing a quality measure by finding the right combination of controllable input parameters. In Additive Manufacturing (AM), the output characteristics of the process can often be non-stationary functions of the process parameters. Bayesian Optimization (BO) is a methodology to optimize such “black-box” functions, i.e., the input–output relationship is unknown and expensive to compute. Optimization tasks involving “black-box” functions widely use BO with Gaussian Process (GP) regression surrogate model. Using GPs with standard kernels is insufficient for modeling non-stationary functions, while GPs with non-stationary kernels are typically over-parameterized. On the other hand, a Deep Gaussian Process (DGP) can overcome GPs’ shortcomings by considering a composition of multiple GPs. Inference in a DGP is challenging due to its structure resulting in a non-Gaussian posterior, and using DGP as a surrogate model for BO is not straightforward. Stochastic Imputation (SI)-based inference is promising in speed and accuracy for BO. This work proposes a bootstrap aggregation-based procedure to effectively utilize the SI-based inference for BO with a DGP surrogate model. The proposed BO algorithm DGP-SI-BO is faster and empirically better than the state-of-the-art BO method in optimizing non-stationary functions. Several analytical test functions and a case study in metal AM simulation demonstrate the applicability of the proposed method.

Robotic Information Gathering (RIG) is a foundational research topic that answers how a robot (team) collects informative data to efficiently build an accurate model of an unknown target function under robot embodiment constraints. RIG has many applications, including but not limited to autonomous exploration and mapping, 3D reconstruction or inspection, search and rescue, and environmental monitoring. A RIG system relies on a probabilistic model’s prediction uncertainty to identify critical areas for informative data collection. Gaussian processes (GPs) with stationary kernels have been widely adopted for spatial modeling. However, real-world spatial data is typically non-stationary—different locations do not have the same degree of variability. As a result, the prediction uncertainty does not accurately reveal prediction error, limiting the success of RIG algorithms. We propose a family of non-stationary kernels named Attentive Kernel (AK), which is simple and robust and can extend any existing kernel to a non-stationary one. We evaluate the new kernel in elevation mapping tasks, where AK provides better accuracy and uncertainty quantification over the commonly used stationary kernels and the leading non-stationary kernels. The improved uncertainty quantification guides the downstream informative planner to collect more valuable data around the high-error area, further increasing prediction accuracy. A field experiment demonstrates that the proposed method can guide an Autonomous Surface Vehicle (ASV) to prioritize data collection in locations with significant spatial variations, enabling the model to characterize salient environmental features.

Deep Gaussian processes (DGPs) are the natural extension of Gaussian processes (GPs) to a multi-layer architecture. DGPs are powerful probabilistic models that have shown better results than standard GPs in terms of generalization performance and prediction uncertainty estimation. Nevertheless, exact inference in DGPs is intractable, making these models hard to train. For this task, current approaches in the literature use approximate inference techniques such as variational inference or approximate expectation propagation. In this work, we present a new method for inference in DGPs using an approximate inference technique based on Monte Carlo methods and the expectation propagation algorithm. Our experiments show that our method scales well to large datasets and that its performance is comparable or better than other state of the art methods. Furthermore, our training method leads to interesting properties in the predictive distribution of the DGP. More precisely, it is able to capture output noise that is dependent on the input and it can generate multimodal predictive distributions. These two properties, which are not shared by other state-of-the-art approximate inference methods for DGPs, are analyzed in detail in our experiments.

Background: Immune checkpoint inhibitors (ICIs) targeting PD-1 or PD-L1 have dramatically improved survival of patients with gastric cancer (GC). However, recent clinical trials, including KEYNOTE-059, KEYNOTE-061, and ATTRACTION-2, show that the objective response rate (ORR) for ICIs is still less than 20% in patients with advanced GC. Therefore, predicting who will respond to ICIs in advance is essential to make better treatment plans for GC patients. There are potential biomarkers to predict ICI response, but most of them can only be applied to a limited population of patients. For example, Kim ST., et al., (PMID: 30013197) show that patients with MSI-high or EBV-positive responded well to ICIs, but these subtypes account for less than 30 percent of GC patients (PMID:25079317). In this study, we propose a prediction model of ICI response using the 32 GC-specific genes that we identified from mutation data (PMID: 26635139) and showed their close association with survival and ICI response in GC (PMID: 35140202; PMID: 36508166). In addition, since the number of available GC-ICI samples is small, we design our prediction model in Bayesian and ensemble learning frameworks which are robust to small sample size problems. Methods: We collected mRNA sequencing data from GC-ICI treated patients from our institutions and publications (PMID: 30013197; PMID: 35254400; PMID: 33846173). We used 108 samples previously included in our previous work (PMID: 36508166) as the training data and 27 samples (13 responders and 14 non-responders) from (PMID: 3384617) as the test data. We built and trained a classifier to separate responders from non-responders using the expressions of the 32 genes. We proposed to use a Deep Gaussian process (GP) model, a multi-layered extension of a GP that is itself a Bayesian method. We optimized the parameters of each GP model in variational inference. To implement ensemble learning, we trained multiple DGP models with different initializations of the model parameters and data splitting (training/validation sets). For each training run, the best model was selected based on the prediction performance on the validation set, measured by AUC. The final prediction for a test sample was made by combining the prediction results of all the DGP models by taking the average. Results: We implemented the DGP-based prediction model using PyTorch in Python, and the codes will be publicly available on the GitHub repository (https://github.com/hwanglab/GC_ICI_response_prediction). We achieved a prediction performance of 0.813 AUC on the test data (sensitivity = 0.769 and specificity = 0.714). Conclusion: We propose the prediction model of ICI response, which can handle small sample size problems in the framework of Bayesian and ensemble learning. The experimental results using the GC-ICI data demonstrate that the 32 genes can be used as predictive markers for ICI response in GC. Citation Format: Sunho Park, Minji Kim, Changjin Hong, Tae Hyun Hwang. A deep Gaussian process (DGP)-based prediction model for immunecheckpoint inhibitor (ICI) response in gastric cancer (GC) [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2024; Part 1 (Regular Abstracts); 2024 Apr 5-10; San Diego, CA. Philadelphia (PA): AACR; Cancer Res 2024;84(6_Suppl):Abstract nr 2316.

The Gaussian process (GP) is a popular statistical technique for stochastic function approximation and uncertainty quantification from data. GPs have been adopted into the realm of machine learning (ML) in the last two decades because of their superior prediction abilities, especially in data-sparse scenarios, and their inherent ability to provide robust uncertainty estimates. Even so, their performance highly depends on intricate customizations of the core methodology, which often leads to dissatisfaction among practitioners when standard setups and off-the-shelf software tools are being deployed. Arguably, the most important building block of a GP is the kernel function, which assumes the role of a covariance operator. Stationary kernels of the Matérn class are used in the vast majority of applied studies; poor prediction performance and unrealistic uncertainty quantification are often the consequences. Non-stationary kernels show improved performance but are rarely used due to their more complicated functional form and the associated effort and expertise needed to define and tune them optimally. In this perspective, we want to help ML practitioners make sense of some of the most common forms of non-stationarity for Gaussian processes. We show a variety of kernels in action using representative datasets, carefully study their properties, and compare their performances. Based on our findings, we propose a new kernel that combines some of the identified advantages of existing kernels.

Despite numerous works over the past two decades, friction-induced vibrations, especially braking noises, are a major issue for transportation manufacturers as well as for the scientific community. To study these fugitive phenomena, the engineers need numerical methods to efficiently predict the mode coupling instabilities in a multiparametric context. The objective of this paper is to approximate the unstable frequencies and the associated damping rates extracted from a complex eigenvalue analysis under variability. To achieve this, a deep Gaussian process is considered to fit the non-linear and non-stationary evolutions of the real and imaginary parts of complex eigenvalues. The current challenge is to build an efficient surrogate modelling, considering a small training set. A discussion about the sample distribution density effect, the training set size and the kernel function choice is proposed. The results are compared to those of a Gaussian process and a deep neural network. A focus is made on several deceptive predictions of surrogate models, although the better settings were well chosen in theory. Finally, the deep Gaussian process is investigated in a multiparametric analysis to identify the best number of hidden layers and neurons, allowing a precise approximation of the behaviours of complex eigensolutions.

Music is a powerful force that evokes human emotions. Several investigations of music emotion recognition (MER) have been conducted in recent years. This paper proposes a system for detecting emotion in music that is based on a deep Gaussian process (GP). The system consists of two parts-feature extraction and classification. In the feature extraction part, five types of features that are associated with emotions are selected for representing the music signal; these are rhythm, dynamics, timbre, pitch and tonality. A music clip is decomposed into frames and these features are extracted from each frame. Next, statistical values, such as mean and standard deviation, of frame-based features are calculated to generate a 38-dimensional feature vector. In the classification part, a deep GP is utilized for emotion recognition. We treat classification problem from the perspective of regression. Finally, 9 classes of emotion are categorized by 9 one-versus-all classifiers. The experimental results demonstrate that the proposed system performs well in emotion recognition.

In this paper, we study node-level graph regression, which aims to predict an output vector for each node on a given graph. This task has a broad range of applications, including spatio-temporal forecasting and computational biology. We propose a model called Deep Gaussian Processes over Graphs (DGPG), which is composed of hierarchical Gaussian processes and learns the mapping between input-output signals in graph domains. DGPG possesses several distinctive advantages, such as the ability in capturing uncertainty, effectiveness on small datasets, and requiring fewer efforts for selecting model architectures and hyperparameters. It is also more favorable than traditional Gaussian process models in terms of expressiveness and scalability, due to the hierarchical deep structure and the variational inference framework. Moreover, we generalize DGPG to a more challenging setting where the graph structure is time-varying. Our theoretical analysis shows that the graph information can improve convergence by reducing sampling variances when optimizing the evidence lower bound, and the challenge of time-varying graph structure can be addressed by a time-weighted sampling scheme. The performance of DGPG is demonstrated through extensive experiments in various synthetic and real-world datasets. Some appealing characteristics of DGPG are further discussed, such as its ability to capture prediction uncertainty and learn graph structures.

By detecting the changing trend, Change point detection (CPD) can describe the underlying behavior of the system with time series (e.g., equipment failure detection, auxiliary medical diagnosis, and climate change detection, etc.). However, in the current big data environment, how quickly obtaining time-series data dependencies and accurately detecting change points is still a challenge. We propose Sailfish, an unsupervised change point detection framework based on the Gaussian process for time series data. In comparison with existing CPD designs, Sailfish has two novel features: 1) using the deep Gaussian process as a hidden variable transformer and 2) integrating the Gaussian process into a bi-LSTM cell for capturing past and future embedded feature trends. Our extensive experiments show that the Sailfish significantly outperforms five state-of-the-art CPD methods with a faster speed and higher accuracy on three public real-time-series datasets. Especially on the large volume of datasets, Sailfish can achieve up to a 60% reduction in training time and 13% F1-score improvement with little overhead compared to the state-of-the-arts.KeywordsChange point detectionGaussian processBi-Long short term memoryBayesian neural network

Hierarchical Bayesian approach to experimental data fusion: Application to strength prediction of high entropy alloys from hardness measurements

Active touch sensing can benefit from the representation of uncertainty in order to guide sensing movements and to drive sensing strategies that operate to reduce uncertainty with respect to the task at hand. Here we explore learning approaches that can acquire task knowledge quickly and with relatively small datasets and with the potential to be exploited for active sensing in robots and as models of biological sensory systems. Specifically, we explore the utility of deep (hierarchical) Gaussian Process models (Deep GPs) that have shown promise as models of episodic memory processes due to their low-dimensionality (compactness), generative capability, and ability to explicitly represent uncertainty. Using data obtained in a robotic active touch task (contour following), we show that both single-layer and Deep GP models are capable of providing robust function approximations from tactile data to angle and sensor position, with Deep GPs showing some advantages in terms of accuracy and uncertainty quantification in angle discrimination.KeywordsActive touchDeep Gaussian processContour followingTactile sensing

Deep Gaussian process regression for lithium-ion battery health prognosis and degradation mode diagnosis

We develop a novel framework to accelerate Gaussian process regression (GPR). In particular, we consider localization kernels at each data point to down-weigh the contributions from other data points that are far away, and we derive the GPR model stemming from the application of such localization operation. Through a set of experiments, we demonstrate the competitive performance of the proposed approach compared to full GPR, other localized models, and deep Gaussian processes. Crucially, these performances are obtained with considerable speedups compared to standard global GPR due to the sparsification effect of the Gram matrix induced by the localization operation.

International audience