Fixed-structure Gaussian process model

This contribution presents a new development in the design of control system based on evolving Gaussian process (GP) models . GP models provide a probabilistic, nonparametric modelling approach for black-box identification of nonlinear dynamic systems. They can highlight areas of the input space where prediction quality is poor, due to the lack of data or its complexity, by indicating the higher variance around the predicted mean. GP models contain noticeably less coefficients to be optimised than commonly used parametric models. While GP models are Bayesian models, their output is normal distribution, expressed in terms of mean and variance. Latter can be interpreted as a confidence in prediction and used in many fields, especially in control system. Evolving GP model is the concept approach within which various ways of model adaptations can be used. Successful control system needs as much as possible data about process to be controlled . If the prior knowledge about the system to be controlled is scarce or the system varies with time or operating region, this control problem can be solved with an iterative method which adapts model with information obtained with streaming data and concurrently optimises hyperparameter values. This contribution provides: a survey of adaptive control algorithms for dynamic systems described in publications where GP models have been used for control design, a novel and improved closed-loop controller with evolving GP models and an example for the illustration of proposed control algorithm.

A new data-driven system identification method, called KL-GP, is proposed for spatiotemporal system. It combines Karhunen-Loève (KL) decomposition and Gaussian process (GP) models. As the nonlinear spatial-temporal spatiotemporal system has strong spatiotemporal characteristics, KL decomposition with good characteristics is employed for time/space separation and dimension reduction. Then the spatiotemporal output is expanded onto a low-dimensional KL space with temporal coefficients. GP models are employed to build up the temporal relation using these coefficients. In addition, a healthy spatial-temporal model that has accuracy predictions is always unknown in practice. GP provides an estimate of the variance of its predicted output. Using this characteristic, active data in the spatiotemporal system region can be found out for the model improvement. This enables the spatiotemporal system model to be updated without high computational demand. Simulation results of spatiotemporal system are presented to demonstrate the effectiveness of this KLGP modeling method.

Quantitative prediction of subjective pain intensity from whole-brain fMRI data using Gaussian processes

Elevated temperatures limit the peak performance of systems because of frequent interventions by thermal throttling. Non-uniform thermal states across system nodes also cause performance variation within seemingly equivalent nodes leading to significant degradation of overall performance. In this paper we present a framework for creating a lightweight thermal prediction system suitable for run-time management decisions. We pursue two avenues to explore optimized lightweight thermal predictors. First, we use feature selection algorithms to improve the performance of previously designed machine learning methods. Second, we develop alternative methods using neural network and linear regression-based methods to perform a comprehensive comparative study of prediction methods. We show that our optimized models achieve improved performance with better prediction accuracy and lower overhead as compared with the Gaussian process model proposed previously. Specifically we present a reduced version of the Gaussian process model, a neural network–based model, and a linear regression–based model. Using the optimization methods, we are able to reduce the average prediction errors in the Gaussian process from $4.2^\circ$ C to $2.9^\circ$ C. We also show that the newly developed models using neural network and Lasso linear regression have average prediction errors of $2.9^\circ$ C and $3.8^\circ$ C respectively. The prediction overheads are 0.22, 0.097, and 0.026 ms per prediction for reduced Gaussian process, neural network, and Lasso linear regression models, respectively, compared with 0.57 ms per prediction for the previous Gaussian process model. We have implemented our proposed thermal prediction models on a two-node system configuration to help identify the optimal task placement. The task placement identified by the models reduces the average system temperature by up to $11.9^\circ$ C without any performance degradation. Furthermore, these models respectively achieve 75, 82.5, and 74.17 percent success rates in correctly pointing to those task placements with better thermal response, compared with 72.5 percent success for the original model in achieving the same objective. Finally, we extended our analysis to a 16-node system and we were able to train models and execute them in real time to guide task migration and achieve on average 17 percent reduction in the overall system cooling power.

Earth observation from satellite sensory data pose challenging problems, where machine learning is currently a key player. In recent years, Gaussian Process (GP) regression and other kernel methods have excelled in biophysical parameter estimation tasks from space. GP regression is based on solid Bayesian statistics, and generally yield efficient and accurate parameter estimates. However, GPs are typically used for inverse modeling based on concurrent observations and in situ measurements only. Very often a forward model encoding the well-understood physical relations is available though. In this work, we review three GP models that respect and learn the physics of the underlying processes in the context of inverse modeling. First, we will introduce a Joint GP (JGP) model that combines in situ measurements and simulated data in a single GP model. Second, we present a latent force model (LFM) for GP modeling that encodes ordinary differential equations to blend data-driven modeling and physical models of the system. The LFM performs multi-output regression, adapts to the signal characteristics, is able to cope with missing data in the time series, and provides explicit latent functions that allow system analysis and evaluation. Finally, we present an Automatic Gaussian Process Emulator (AGAPE) that approximates the forward physical model via interpolation, reducing the number of necessary nodes. Empirical evidence of the performance of these models will be presented through illustrative examples of vegetation monitoring and atmospheric modeling.

For the challenging task of modeling actual complex motions, we propose a new class of Gaussian process (GP) models that are data-driven and also take into account prior knowledge of the motion intention. As a theoretical basis, we show that the GP regression is mathematically equivalent to regularized least-squares estimation for random functions with known prior means. Compared with the popular GP models in machine learning literature, the proposed GP motion model priors with conditional kernels have at least two advantages: 1) they are nonstationary and more applicable to represent complex motions by integrating the basic kinematic principles; 2) conditional kernels are further devised by incorporating the motion intent so that the resultant GP models are more versatile and would expectedly entail more accurate trajectory prediction. A superior property of the GP models with conditional kernels is found to improve the computational efficiency. Finally, illustrative examples are provided to show the superiority of the proposed motion models and to verify the theoretical results given in the paper.

Prediction of pricing and hedging errors for equity linked warrants with Gaussian process models☆

Device sizing is a challenging problem for analog circuit design. Traditional methods depend on domain knowledge and intensive simulations to search for feasible parameters. Recent studies apply the Bayesian optimization (BO) and a Gaussian process (GP) model in analog circuit synthesis to improve efficiency. The BO framework automatically selects the parameter candidates by inferring the surrogate GP model. However, naive BO employs a sequential updating strategy which is inefficient in a multicore environment. Besides, the widely used GP model requires costly high fidelity data, which are obtained from fine simulations. In this article, we propose a constrained batch BO approach with a multifidelity (MF) model to solve the above difficulties. The batch BO exploits parallel computing and selects promising parameters by multiple acquisition function ensemble. In addition, the MF GP model adapts the low fidelity data obtained from coarse simulations. Specifically, the proposed method incorporates information gain in a weighted clustering algorithm to refine the parameter candidates. As a result, the proposed method maintains the candidates’ quality and diversity, which speeds up the optimization convergence. In the experiments, we demonstrate the efficiency of the proposed approach on three real-world circuits. The results show that our approach reduces the simulation costs by at least 54.6% compared to the state-of-the-art baselines.

The Gaussian process (GP) model provides a powerful methodology for calibrating a computer model in the presence of model uncertainties. However, if the data contain systematic experimental errors, then the GP model can lead to an unnecessarily complex adjustment of the computer model. In this work, we introduce an adjustment procedure that brings the computer model closer to the data by making minimal changes to it. This is achieved by applying a lasso-based variable selection on the systematic experimental error terms while fitting the GP model. Two real examples and simulations are presented to demonstrate the advantages of the proposed approach. This article has supplementary material available online.

Dynamic reliability is defined as the probability that an engineered system successfully performs the predefined functionality over a certain period of time considering time-variant operation condition and component deterioration. In practice, it is still a major challenge to conduce dynamic reliability analysis due to the prohibitively high computational costs. In this study, a confidence-based meta-modeling approach is proposed for efficient sensitivity-free dynamic reliability analysis, referred to as double-loop adaptive sampling (DLAS). In DLAS a Gaussian process (GP) model is constructed to approximate extreme system responses over time, so that Monte Carlo simulation (MCS) can be employed directly to estimate dynamic reliability. A qualitative confidence measure is proposed to evaluate the accuracy of dynamic reliability estimation while using the MCS approach based on developed GP models. To improve the confidence, a double-loop adaptive sampling scheme is developed to efficiently update the GP model in a sequential manner, by considering system input variables and time concurrently in double sampling loops. The model updating process can be terminated once the user defined confidence target is satisfied. The DLAS approach does not require computationally expensive sensitivity analysis, thus substantially improves the efficiency of dynamic reliability assessment. Two case studies are used to demonstrate the effectiveness of DLAS for dynamic reliability analysis.

For multi-input and multi-output (MIMO) systems, its nonlinearity and strong coupling make the control problem more complicated. This paper introduces a new PID control strategy based on Gaussian process (GP) model to solve the problem of dynamic nonlinearity and strong coupling in complex industrial fields. This processing method is a combination of partial least squares (PLS) as an external framework and GP for internal processing. Firstly, the dynamic PLS (DPLS) is used to extract the input and output variables to eliminate the correlation, and the obtained feature matrix is used as the input of the GP to establish the model. Therefore, the MIMO control problem in the original space is transformed into multiple single-input and single-output (SISO) control problems in the latent space. Finally, a numerical example are used to illustrate the efficiency of the proposed control method.

In this paper, we utilize the probability density function of the data to estimate the minimum variance lower bound (MVLB) of a nonlinear system. For this purpose, the Gaussian Process (GP) model has been used. With this model, given a new input and based on past observations, we naturally obtained the variance of the predictive distribution of the future output, which enabled us to estimate MVLB as well as estimation uncertainty. Also, an advantage of the proposed method over others is its ability to estimate MVLB recursively. The application of this method to the real-life dynamic process (experimental four-tank process) indicates that this approach gives very credible estimates of the MVLB.

Storage system performance modeling is crucial for efficient use of heterogeneous shared resources on leadership class computers. Variability in application performance, particularly variability arising from concurrent applications sharing I/O resources, is a major hurdle in the development of accurate performance models. We adopt a deep learning approach based on conditional variational auto encoders (CVAE) for I/O performance modeling, and use it to quantify performance variability. We illustrate our approach using the data collected on Edison, a production supercomputing system at the National Energy Research Scientific Computing Center (NERSC). The CVAE approach is investigated by comparing it to a previously proposed sensitivity-based Gaussian process (GP) model. We find that the CVAE model performs slightly better than the GP model in cases where training and testing data come from different applications, since CVAE can inherently leverage the whole data from multiple applications whereas GP partitions the data and builds separate models for each partition. Hence, the CVAE offers an alternative modeling approach that does not need preprocessing; it has enough flexibility to handle data from a wide variety of applications without changing the inference approach.

We propose an iterative method to safely learn the unmodeled dynamics of a control-affine nonlinear system using Bayesian Gaussian process (GP) models with polynomial kernel functions. The method maintains safety by ensuring that the system state stays within the region of attraction (ROA) of a stabilizing control policy while collecting data. A quadratic programming based exploration control policy is computed to keep the exploration trajectory inside an inner-approximation of the ROA and to maximize the information gained from the trajectory. A prior GP model, which incorporates prior information about the unknown dynamics, is used to construct an initial stabilizing policy. As the GP model is updated with data, it is used to synthesize a new policy and a larger ROA, which increases the range of safe exploration. The use of polynomial kernels allows us to compute ROA inner-approximations and stabilizing control laws for the model using sum-of-squares programming. We also provide a probabilistic guarantee of safety which ensures that the policy computed using the learned model stabilizes the true dynamics with high confidence.

Parameter estimation for nonlinear dynamic system models, represented by ordinary differential equations (ODEs), using noisy and sparse data, is a vital task in many fields. We propose a fast and accurate method, manifold-constrained Gaussian process inference (MAGI), for this task. MAGI uses a Gaussian process model over time series data, explicitly conditioned on the manifold constraint that derivatives of the Gaussian process must satisfy the ODE system. By doing so, we completely bypass the need for numerical integration and achieve substantial savings in computational time. MAGI is also suitable for inference with unobserved system components, which often occur in real experiments. MAGI is distinct from existing approaches as we provide a principled statistical construction under a Bayesian framework, which incorporates the ODE system through the manifold constraint. We demonstrate the accuracy and speed of MAGI using realistic examples based on physical experiments.