Minimum variance lower bound estimation with Gaussian Process models

This contribution presents a new development in the design of control system based on evolving Gaussian process (GP) models . GP models provide a probabilistic, nonparametric modelling approach for black-box identification of nonlinear dynamic systems. They can highlight areas of the input space where prediction quality is poor, due to the lack of data or its complexity, by indicating the higher variance around the predicted mean. GP models contain noticeably less coefficients to be optimised than commonly used parametric models. While GP models are Bayesian models, their output is normal distribution, expressed in terms of mean and variance. Latter can be interpreted as a confidence in prediction and used in many fields, especially in control system. Evolving GP model is the concept approach within which various ways of model adaptations can be used. Successful control system needs as much as possible data about process to be controlled . If the prior knowledge about the system to be controlled is scarce or the system varies with time or operating region, this control problem can be solved with an iterative method which adapts model with information obtained with streaming data and concurrently optimises hyperparameter values. This contribution provides: a survey of adaptive control algorithms for dynamic systems described in publications where GP models have been used for control design, a novel and improved closed-loop controller with evolving GP models and an example for the illustration of proposed control algorithm.

Quantitative prediction of subjective pain intensity from whole-brain fMRI data using Gaussian processes

Elevated temperatures limit the peak performance of systems because of frequent interventions by thermal throttling. Non-uniform thermal states across system nodes also cause performance variation within seemingly equivalent nodes leading to significant degradation of overall performance. In this paper we present a framework for creating a lightweight thermal prediction system suitable for run-time management decisions. We pursue two avenues to explore optimized lightweight thermal predictors. First, we use feature selection algorithms to improve the performance of previously designed machine learning methods. Second, we develop alternative methods using neural network and linear regression-based methods to perform a comprehensive comparative study of prediction methods. We show that our optimized models achieve improved performance with better prediction accuracy and lower overhead as compared with the Gaussian process model proposed previously. Specifically we present a reduced version of the Gaussian process model, a neural network–based model, and a linear regression–based model. Using the optimization methods, we are able to reduce the average prediction errors in the Gaussian process from $4.2^\circ$ C to $2.9^\circ$ C. We also show that the newly developed models using neural network and Lasso linear regression have average prediction errors of $2.9^\circ$ C and $3.8^\circ$ C respectively. The prediction overheads are 0.22, 0.097, and 0.026 ms per prediction for reduced Gaussian process, neural network, and Lasso linear regression models, respectively, compared with 0.57 ms per prediction for the previous Gaussian process model. We have implemented our proposed thermal prediction models on a two-node system configuration to help identify the optimal task placement. The task placement identified by the models reduces the average system temperature by up to $11.9^\circ$ C without any performance degradation. Furthermore, these models respectively achieve 75, 82.5, and 74.17 percent success rates in correctly pointing to those task placements with better thermal response, compared with 72.5 percent success for the original model in achieving the same objective. Finally, we extended our analysis to a 16-node system and we were able to train models and execute them in real time to guide task migration and achieve on average 17 percent reduction in the overall system cooling power.

For the challenging task of modeling actual complex motions, we propose a new class of Gaussian process (GP) models that are data-driven and also take into account prior knowledge of the motion intention. As a theoretical basis, we show that the GP regression is mathematically equivalent to regularized least-squares estimation for random functions with known prior means. Compared with the popular GP models in machine learning literature, the proposed GP motion model priors with conditional kernels have at least two advantages: 1) they are nonstationary and more applicable to represent complex motions by integrating the basic kinematic principles; 2) conditional kernels are further devised by incorporating the motion intent so that the resultant GP models are more versatile and would expectedly entail more accurate trajectory prediction. A superior property of the GP models with conditional kernels is found to improve the computational efficiency. Finally, illustrative examples are provided to show the superiority of the proposed motion models and to verify the theoretical results given in the paper.

Earth observation from satellite sensory data pose challenging problems, where machine learning is currently a key player. In recent years, Gaussian Process (GP) regression and other kernel methods have excelled in biophysical parameter estimation tasks from space. GP regression is based on solid Bayesian statistics, and generally yield efficient and accurate parameter estimates. However, GPs are typically used for inverse modeling based on concurrent observations and in situ measurements only. Very often a forward model encoding the well-understood physical relations is available though. In this work, we review three GP models that respect and learn the physics of the underlying processes in the context of inverse modeling. First, we will introduce a Joint GP (JGP) model that combines in situ measurements and simulated data in a single GP model. Second, we present a latent force model (LFM) for GP modeling that encodes ordinary differential equations to blend data-driven modeling and physical models of the system. The LFM performs multi-output regression, adapts to the signal characteristics, is able to cope with missing data in the time series, and provides explicit latent functions that allow system analysis and evaluation. Finally, we present an Automatic Gaussian Process Emulator (AGAPE) that approximates the forward physical model via interpolation, reducing the number of necessary nodes. Empirical evidence of the performance of these models will be presented through illustrative examples of vegetation monitoring and atmospheric modeling.

Prediction of pricing and hedging errors for equity linked warrants with Gaussian process models☆

Device sizing is a challenging problem for analog circuit design. Traditional methods depend on domain knowledge and intensive simulations to search for feasible parameters. Recent studies apply the Bayesian optimization (BO) and a Gaussian process (GP) model in analog circuit synthesis to improve efficiency. The BO framework automatically selects the parameter candidates by inferring the surrogate GP model. However, naive BO employs a sequential updating strategy which is inefficient in a multicore environment. Besides, the widely used GP model requires costly high fidelity data, which are obtained from fine simulations. In this article, we propose a constrained batch BO approach with a multifidelity (MF) model to solve the above difficulties. The batch BO exploits parallel computing and selects promising parameters by multiple acquisition function ensemble. In addition, the MF GP model adapts the low fidelity data obtained from coarse simulations. Specifically, the proposed method incorporates information gain in a weighted clustering algorithm to refine the parameter candidates. As a result, the proposed method maintains the candidates’ quality and diversity, which speeds up the optimization convergence. In the experiments, we demonstrate the efficiency of the proposed approach on three real-world circuits. The results show that our approach reduces the simulation costs by at least 54.6% compared to the state-of-the-art baselines.

This article describes a method for modelling non-linear dynamic systems from measurement data. The method merges the linear local model blending approach in the velocity-based linearisation form with Bayesian Gaussian process (GP) modelling. The new Fixed-Structure GP (FSGP) model has a predetermined linear model structure with varying and probabilistic parameters represented by GP models. These models have several advantages for the modelling of local model parameters as they give us adequate results, even with small data sets. Furthermore, they provide a measure of the confidence in the prediction of the varying parameters and information about the dependence of the parameters on individual inputs. The FSGP model can be applied for the extended local linear equivalence class of non-linear systems. The obtained non-linear system model can be, for example, used for control-system design. The proposed modelling method is illustrated with a simple example of non-linear system modelling for control design.

For multi-input and multi-output (MIMO) systems, its nonlinearity and strong coupling make the control problem more complicated. This paper introduces a new PID control strategy based on Gaussian process (GP) model to solve the problem of dynamic nonlinearity and strong coupling in complex industrial fields. This processing method is a combination of partial least squares (PLS) as an external framework and GP for internal processing. Firstly, the dynamic PLS (DPLS) is used to extract the input and output variables to eliminate the correlation, and the obtained feature matrix is used as the input of the GP to establish the model. Therefore, the MIMO control problem in the original space is transformed into multiple single-input and single-output (SISO) control problems in the latent space. Finally, a numerical example are used to illustrate the efficiency of the proposed control method.

A new data-driven system identification method, called KL-GP, is proposed for spatiotemporal system. It combines Karhunen-Loève (KL) decomposition and Gaussian process (GP) models. As the nonlinear spatial-temporal spatiotemporal system has strong spatiotemporal characteristics, KL decomposition with good characteristics is employed for time/space separation and dimension reduction. Then the spatiotemporal output is expanded onto a low-dimensional KL space with temporal coefficients. GP models are employed to build up the temporal relation using these coefficients. In addition, a healthy spatial-temporal model that has accuracy predictions is always unknown in practice. GP provides an estimate of the variance of its predicted output. Using this characteristic, active data in the spatiotemporal system region can be found out for the model improvement. This enables the spatiotemporal system model to be updated without high computational demand. Simulation results of spatiotemporal system are presented to demonstrate the effectiveness of this KLGP modeling method.

The Gaussian process (GP) model provides a powerful methodology for calibrating a computer model in the presence of model uncertainties. However, if the data contain systematic experimental errors, then the GP model can lead to an unnecessarily complex adjustment of the computer model. In this work, we introduce an adjustment procedure that brings the computer model closer to the data by making minimal changes to it. This is achieved by applying a lasso-based variable selection on the systematic experimental error terms while fitting the GP model. Two real examples and simulations are presented to demonstrate the advantages of the proposed approach. This article has supplementary material available online.

Dynamic reliability is defined as the probability that an engineered system successfully performs the predefined functionality over a certain period of time considering time-variant operation condition and component deterioration. In practice, it is still a major challenge to conduce dynamic reliability analysis due to the prohibitively high computational costs. In this study, a confidence-based meta-modeling approach is proposed for efficient sensitivity-free dynamic reliability analysis, referred to as double-loop adaptive sampling (DLAS). In DLAS a Gaussian process (GP) model is constructed to approximate extreme system responses over time, so that Monte Carlo simulation (MCS) can be employed directly to estimate dynamic reliability. A qualitative confidence measure is proposed to evaluate the accuracy of dynamic reliability estimation while using the MCS approach based on developed GP models. To improve the confidence, a double-loop adaptive sampling scheme is developed to efficiently update the GP model in a sequential manner, by considering system input variables and time concurrently in double sampling loops. The model updating process can be terminated once the user defined confidence target is satisfied. The DLAS approach does not require computationally expensive sensitivity analysis, thus substantially improves the efficiency of dynamic reliability assessment. Two case studies are used to demonstrate the effectiveness of DLAS for dynamic reliability analysis.

Storage system performance modeling is crucial for efficient use of heterogeneous shared resources on leadership class computers. Variability in application performance, particularly variability arising from concurrent applications sharing I/O resources, is a major hurdle in the development of accurate performance models. We adopt a deep learning approach based on conditional variational auto encoders (CVAE) for I/O performance modeling, and use it to quantify performance variability. We illustrate our approach using the data collected on Edison, a production supercomputing system at the National Energy Research Scientific Computing Center (NERSC). The CVAE approach is investigated by comparing it to a previously proposed sensitivity-based Gaussian process (GP) model. We find that the CVAE model performs slightly better than the GP model in cases where training and testing data come from different applications, since CVAE can inherently leverage the whole data from multiple applications whereas GP partitions the data and builds separate models for each partition. Hence, the CVAE offers an alternative modeling approach that does not need preprocessing; it has enough flexibility to handle data from a wide variety of applications without changing the inference approach.

In this work, we propose a novel framework for large-scale Gaussian process (GP) modeling. Contrary to the global, and local approximations proposed in the literature to address the computational bottleneck with exact GP modeling, we employ a combined global-local approach in building the approximation. Our framework uses a subset-of-data approach where the subset is a union of a set of global points designed to capture the global trend in the data, and a set of local points specific to a given testing location to capture the local trend around the testing location. The correlation function is also modeled as a combination of a global, and a local kernel. The predictive performance of our framework, which we refer to as TwinGP, is comparable to the state-of-the-art GP modeling methods, but at a fraction of their computational cost.

This study compared hierarchical Bayesian, mixed-effects Gaussian process regression, and random forest models for predicting height to crown base (HCB) in Qinghai spruce (Picea crassifolia Kom.) forests using LiDAR-derived data. Both modeling approaches were applied to a dataset of 510 trees from 16 plots in northern China. The models incorporated tree-level variables (height, diameter at breast height, crown projection area) and plot-level spatial competition indices. Model performance was evaluated using leave-one-plot-out cross-validation. The Gaussian mixed-effects process model (with an RMSE of 1.59 and MAE of 1.25) slightly outperformed the hierarchical Bayesian model and the random forest model. Both models identified LiDAR-derived tree height, DBH, and LiDAR-derived crown projection area as primary factors influencing HCB. The spatial competition index (SCI) emerged as the most effective random effect, with the lowest AIC and BIC values, highlighting the importance of local competition dynamics in HCB formation. Uncertainty analysis revealed consistent patterns across the predicted values, with an average relative uncertainty of 33.89% for the Gaussian process model. These findings provide valuable insights for forest management and suggest that incorporating spatial competition indices can enhance HCB predictions.