Variational Fourier features for Gaussian processes

Gaussian process has been widely used in areas including geostatistics and uncertainty quantification due to its parsimonious yet flexible representation of a stochastic process. However, analyzing a large data set with Gaussian process can be challenging due to its O(n3) computational complexity, where n denotes the size of the data set. The recently proposed Nearest Neighbor Gaussian Process (NNGP) aims to approximate a Gaussian process with a target covariance function by using a series of conditional distributions and then exploiting the sparse precision matrices. We demonstrate that NNGP has the potential to be used for uncertainty quantification. We discover that when using NNGP to approximate a Gaussian process with strong smoothness, e.g., the squared-exponential covariance function, Bayesian inference needs to be carried out carefully with marginalizing over the random effects in NNGP. Using simulated and real data, we investigate empirically the performance of NNGP to approximate the squared-exponential covariance function as well as its ability to handle change-of-support effect, a common phenomenon in geostatistics and uncertainty quantification when only aggregated data over space are available.

This paper describes a real-time data collection framework and an adaptive machining learning method for constructing a real-time energy prediction model for a machine tool. To effectively establish the energy consumption pattern of a machine tool over time, the energy prediction model is continuously updated with new measurement data to account for time-varying effects of the machine tool, such as tool wear and machine tool deterioration. In this work, a real-time data collection and processing framework is developed to retrieve raw data from a milling machine tool and its sensors and convert them into relevant input features. The extracted input features are then used to construct the energy prediction model using Gaussian Process (GP) regression. To update the GP regression model with real-time streaming data, we investigate the use of sparse representation of the covariance matrix to reduce the computational and storage demands of the GP regression. We compare computational efficiency of sparse GP to that of full GP regression model and show the effectiveness of the sparse GP regression model for tracking the variation in the energy consumption pattern of the target machine.

Gaussian processes (GPs) are a popular class of Bayesian nonparametric models, but its training can be computationally burdensome for massive training datasets. While there has been notable work on scaling up these models for big data, existing methods typically rely on a stationary GP assumption for approximation, and can thus perform poorly when the underlying response surface is nonstationary, that is, it has some regions of rapid change and other regions with little change. Such non-stationarity is, however, ubiquitous in real-world problems, including our motivating application for surrogate modeling of computer experiments. We propose a new Product of Sparse GP (ProSpar-GP) method for scalable GP modeling with massive nonstationary data. The ProSpar-GP makes use of a carefully-constructed product-of-experts formulation of sparse GP experts, where different experts are placed within local regions of non-stationarity. These GP experts are fit via a novel variational inference approach, which capitalizes on mini-batching and GPU acceleration for efficient optimization of inducing points and length-scale parameters for each expert. We further show that the ProSpar-GP is Kolmogorov-consistent, in that its generative distribution defines a valid stochastic process over the prediction space; such a property provides essential stability for variational inference, particularly in the presence of non-stationarity. We then demonstrate the improved performance of the ProSpar-GP over the state-of-the-art, in a suite of numerical experiments and an application for surrogate modeling of a satellite drag simulator. Supplemental materials for this article are available online.

Gaussian processes (GPs) for machine learning have been studied systematically over the past two decades. However, kernel design for GPs and the associated hyper-parameters optimization are still difficult, and to a large extent open problems. We consider GP regression for time series modeling and analysis. The underlying stationary kernel is approximated closely by a new grid spectral mixture (GSM) kernel, which is a linear combination of low-rank sub-kernels. In the case where a large number of the involved sub-kernels are used, either the Nyström or the random Fourier feature approximations can be adopted to reduce the required computer storage. The unknown GP hyper-parameters consist of the nonnegative weights of all sub-kernels as well as the noise variance, and they are determined through the maximum-likelihood estimation method. Two optimization methods for solving the unknown hyper-parameters are introduced, including a sequential majorization-minimization (MM) method and a nonlinearly constrained alternating direction method of multipliers (ADMM). Experimental results, based on various time series datasets, corroborate that the proposed GSM kernel-based GP regression model outperforms several benchmarks in terms of prediction accuracy and numerical stability.

Uncertainty propagation using infinite mixture of Gaussian processes and variational Bayesian inference

Recursive estimation for sparse Gaussian process regression

Gaussian processes (GPs) are natural generalisations of multivariate Gaussian random variables to infinite (countably or continuous) index sets. GPs have been applied in a large number of fields to a diverse range of ends, and very many deep theoretical analyses of various properties are available. This paper gives an introduction to Gaussian processes on a fairly elementary level with special emphasis on characteristics relevant in machine learning. It draws explicit connections to branches such as spline smoothing models and support vector machines in which similar ideas have been investigated. Gaussian process models are routinely used to solve hard machine learning problems. They are attractive because of their flexible non-parametric nature and computational simplicity. Treated within a Bayesian framework, very powerful statistical methods can be implemented which offer valid estimates of uncertainties in our predictions and generic model selection procedures cast as nonlinear optimization problems. Their main drawback of heavy computational scaling has recently been alleviated by the introduction of generic sparse approximations.13,78,31 The mathematical literature on GPs is large and often uses deep concepts which are not required to fully understand most machine learning applications. In this tutorial paper, we aim to present characteristics of GPs relevant to machine learning and to show up precise connections to other "kernel machines" popular in the community. Our focus is on a simple presentation, but references to more detailed sources are provided.

SummaryA continuous parameter, stationary Gaussian process is introduced as a first approach to the probabilistic representation of the phenotype inheritance process. With some specific assumptions about the components of the covariance function, it may describe the temporal behaviour of the “cancer-proneness phenotype” (CPF) as a quantitative continuous trait. Upcrossing a fixed level (“threshold”) u and reaching level zero are the extremes of the Gaussian process considered; it is assumed that they might be interpreted as the transformation of CPF into a “neoplastic disease phenotype” or as the non-proneness to cancer, respectively.

In many machine learning problems, one has to work with data of different types, including continuous, discrete, and categorical data. Further, it is often the case that many of these data are missing from the database. This paper proposes a Gaussian process framework that efficiently captures the information from mixed numerical and categorical data that effectively incorporates missing variables. First, we propose a generative model for the mixed-type data. The generative model exploits Gaussian processes with kernels constructed from the latent vectors. We also propose a method for inference of the unknowns, and in its implementation, we rely on a sparse spectrum approximation of the Gaussian processes and variational inference. We demonstrate the performance of the method for both supervised and unsupervised tasks. First, we investigate the imputation of missing variables in an unsupervised setting, and then we show the results of joint imputation and classification on IBM employee data.

Sparse Gaussian process (GP) models provide an efficient way to perform regression on large data sets. The key idea is to select a representative subset of the available training data, which induces the sparse GP model approximation. In the past, a variety of selection criteria for GP approximation have been proposed, but they either lack accuracy or suffer from high computational costs. In this paper, we introduce a novel and straightforward criterion for successive selection of training points used for GP model approximation. The proposed algorithm allows a fast and efficient selection of training points, while being competitive in learning performance. As evaluation, we employ our approach in learning inverse dynamics models for robot control using very large data sets (e.g. 500.000 samples). It is demonstrated in experiments that our approximated GP model is sufficiently fast for real-time prediction in robot control. Comparisons with other state-of-the-art approximation techniques show that our proposed approach is significantly faster, while being competitive to generalization accuracy.

We discuss the representation of thermodynamic processes in diagrams. Although the idea that it is impossible to represent non-quasistatic processes is correct and consensual, it is surprising that we sometimes find in the literature such processes represented somehow, in contradiction with, for instance, the concept of equilibrium state. At other times, the justification given for process representation does not help clarify the difference between reversible and quasistatic processes. In this paper we address the errors inherent to such procedures and also, paradoxically, propose the representation of non-quasistatic processes using the concept of identical processes.

We analyze surface wave data taken in Currituck Sound, North Carolina, during a storm on 4 February 2002. Our focus is on the application of nonlinear Fourier analysis (NLFA) methods (Osborne 2010) to analyze the data set: The approach spectrally decomposes a nonlinear wave field into sine waves, Stokes waves, and phase-locked Stokes waves otherwise known as breather trains. Breathers are nonlinear beats, or packets which “breathe” up and down smoothly over cycle times of minutes to hours. The maximum amplitudes of the packets during the cycle have a largest central wave whose properties are often associated with the study of “rogue waves.” The mathematical physics of the nonlinear Schrodinger (NLS) equation is assumed and the methods of algebraic geometry are applied to give the nonlinear spectral representation. The distinguishing characteristic of the NLFA method is its ability to spectrally decompose a time series into its nonlinear coherent structures (Stokes waves and breathers) rather than just sine waves. This is done by the implementation of multidimensional, quasi-periodic Fourier series, rather than ordinary Fourier series. To determine preliminary estimates of nonlinearity, we use the significant wave height Hs, the peak period Tp, and the length of the time series T. The time series analyzed here have 8192 points and T =1677.72 s = 27.96 min. Near the peak of the storm, we find Hs ≈ 0.55 m, Tp ≈ 2.4 s so that for the wave steepness of a near Gaussian process, \({S} = \left (\pi ^{5/2}/g\right )H_{s}/{T}_{p}^{2}\), we find S ≈ 0.17, quite high for ocean waves. Likewise, we estimate the Benjamin-Feir (BF) parameter for a near Gaussian process, \({I_{BF}} = \left (\pi ^{5/2}/g \right ) H_{s} T/{T}_{p}^{3}\), and we find IBF ≈ 119. Since the BF parameter describes the nonlinear behavior of the modulational instability, leading to the formation of breather packets in a measured wave train, we find the IBF for these storm waves to be a surprisingly high number. This is because IBF, as derived here, roughly estimates the number of breather trains in a near Gaussian time series. The BF parameter suggests that there are roughly 119 breather trains in a time series of length 28 min near the peak of the storm, meaning that we would have average breather packets of about 14 s each with about 5-6 waves in each packet. Can these surprising results, estimated from simple parameters, be true from the point of view of the complex nonlinear wave dynamics of the BF instability and the NLS equation? We analyze the data set with the NLFA to verify, from a nonlinear spectral point of view, the presence of large numbers of breather trains and we determine many of their properties, including the rise time for the breathers to grow to their maximum amplitudes from a quiescent initial state. Energetically, about 95% of the NLFA components are found to consist of breather trains; the remaining small amplitude components are sine and Stokes waves. The presence of a large number of densely packed breather trains suggests an interpretation of the data in terms of breather turbulence, highly nonlinear integrable turbulence theoretically predicted for the NLS equation, providing an interesting paradigm for the nonlinear wave motion, in contrast to the random phase Gaussian approximation often considered in the analysis of data.

This paper proposed a variational Bayesian approach for the SVM regression based on the likelihood model of an infinite mixture of Gaussians. To evaluate this approach the method was applied to synthetic datasets. We compared this new approximation approach with the standard SVM algorithm as well as other well established methods such as Gaussian Process.

An efficient numerical method for non-ergodic ground-motion inference and prediction is proposed that alleviates the large computational and memory requirements associated with the traditional approach based on Gaussian Processes described in Landwehr et al. (Bull Seismol Soc Am, https://doi.org/10.1785/0120160118, 2016). The method uses the latest developments in Gaussian Processes and Machine Learning from Wilson and Nickisch (in: International Conference on Machine Learning, pp 1775–1784, 2015) (SKI) and Gardner et al. (in: International Conference on Artificial Intelligence and Statistics, pp. 1407–1416, 2018) (SKIP) and uses sparse approximations combined with efficient matrix decompositions to accurately approximate the large covariance matrices involved in the calculations. This efficient method can be used for both inference of hyperparameters of the non-ergodic ground-motion models and for forward predictions of non-ergodic median ground-motion. The application to predictions are presented. For large datasets of 100,000 to 1,000,000 ground motion values, the proposed method increases the computation speed by factors of 100 to 1000, reducing run times from days to minutes. In addition, the memory requirements are reduced from hundreds of GB to a few GB only, which makes the development of non-ergodic ground-motion models practical using traditional desktop computers.

In this chapter we will show that every stationary process can be approximated by a sum of harmonic oscillations with random and uncorrelated amplitudes. The limit of these sums leads to an integral representation, the so called spectral representation of stationary processes. This spectral representation is a generalization of the Fourier representation of deterministic sequences to stationary processes. This is of key importance for the theory of stationary processes as well as for their interpretation. The spectral representation defines a bijective isometry between the time domain and the so called frequency domain of the process. Linear dynamic transformations of stationary processes (see chapter 4) can be performed and interpreted in this frequency domain in a particularly simple manner. The Fourier representation of the covariance function corresponding to the spectral representation of the process permits an equivalent, and nice to interpret, description of the linear dependence structure of the process. In this section, we show the essential steps of the proof of the spectral representation of stationary processes. Despite the fact that the structure of the proof provides essential insights, readers may skip technical details when reading it for the first time.