Uncertainty Quantification Using the Nearest Neighbor Gaussian Process

The growing field of large-scale time domain astronomy requires methods for probabilistic data analysis that are computationally tractable, even with large data sets. Gaussian processes (GPs) are a popular class of models used for this purpose, but since the computational cost scales, in general, as the cube of the number of data points, their application has been limited to small data sets. In this paper, we present a novel method for GPs modeling in one dimension where the computational requirements scale linearly with the size of the data set. We demonstrate the method by applying it to simulated and real astronomical time series data sets. These demonstrations are examples of probabilistic inference of stellar rotation periods, asteroseismic oscillation spectra, and transiting planet parameters. The method exploits structure in the problem when the covariance function is expressed as a mixture of complex exponentials, without requiring evenly spaced observations or uniform noise. This form of covariance arises naturally when the process is a mixture of stochastically driven damped harmonic oscillators—providing a physical motivation for and interpretation of this choice—but we also demonstrate that it can be a useful effective model in some other cases. We present a mathematical description of the method and compare it to existing scalable GP methods. The method is fast and interpretable, with a range of potential applications within astronomical data analysis and beyond. We provide well-tested and documented open-source implementations of this method in C++, Python, and Julia.

Quality measures for the evaluation of machine learning architectures on the quantification of epistemic and aleatoric uncertainties in complex dynamical systems

In nonparametric Bayesian approaches, Gaussian stochastic processes can serve as priors on real-valued function spaces. Existing literature on the posterior convergence rates under Gaussian process priors shows that it is possible to achieve optimal or near-optimal posterior contraction rates if the smoothness of the Gaussian process matches that of the target function. Among those priors, Gaussian process with a parametric Matérn covariance function is particularly notable in that its degree of smoothness can be determined by a dedicated smoothness parameter. Ma and Bhadra (2023) recently introduced a new family of covariance functions called the Confluent Hypergeometric (CH) class that simultaneously possess two parameters: one controls the tail index of the polynomially decaying covariance function, and the other parameter controls the degree of mean-squared smoothness analogous to the Matérn class. In this paper, we show that with proper choice of rescaling parameters in the Matérn and CH covariance functions, it is possible to obtain the minimax optimal posterior contraction rate for η-regular functions for nonparametric regression model with fixed design. Unlike the previous results for unrescaled cases, the smoothness parameter of the covariance function need not equal η for achieving the optimal minimax rate, for either rescaled Matérn or rescaled CH covariances, illustrating a key benefit of rescaling. We also consider a fully Bayesian treatment of the rescaling parameters and show the resulting posterior distributions still contract at the minimax-optimal rate. The resultant hierarchical Bayesian procedure is fully adaptive to the unknown true smoothness. The theoretical properties of the rescaled and hierarchical Matérn and CH classes are further verified via extensive simulations and an illustration on a geospatial data set is presented.

The class of Gaussian stochastic processes plays an important role in stochastic modeling and is widely used as a rich source of models in many areas of applied probability, including actuarial and financial mathematics. There are many practical and theoretical reasons for using Gaussian processes to model actuarial problems. For instance, the family of Gaussian processes covers a large class of correlation structures and enables explicit analysis of models for which classical renewal‐type tools do not work. On the other hand, theoretical results, mostly based on central limit theorem‐type argumentation, formally justify the use of Gaussian processes.

Gaussian Process Regression (GPR) is a nonparametric machine learning model that provides uncertainty quantification in making predictions. GPR utilizes several covariance functions (CFs) in the process of developing models to ensure high accuracy. There are five common CFs in GPR, which are the Radial Basis Function (RBF), Rational Quadratic (RQ), Periodic (Per), Matérn 3/2 (Mat 3/2), and Matérn 5/2 (Mat 5/2), where each covariance function (CF) has different characteristics and behaviors. This paper is to investigate the comparative performances of each CF when applied to the Global Wheat Prices dataset. Error metric measurement like Mean Square Error (MSE) and Root Mean Square Error (RMSE), while Coefficient of Variation (CV) for uncertainty quantification measurement is computed for each CF, and comparisons was made among the CFs to conclude the best CF for this application. The lowest values among them will be the best CF for the data. It should be noted that the CV for each model should be less than 5%, and the CF with the smallest value of CV is considered reliable. The five CFs were fit to the Global Wheat Prices dataset, and it was found that the Mat 3/2 produced the best performances with the lowest values of MSE, RMSE, and CV. Mat 3/2 is the most efficient CF for making predictions since it gives the lowest value of error metric measurement and the lowest value of CV under 5% among the other CFs, making it more reliable for modeling. Overall, the outcome shows that Mat 3/2 is the best CF to be used in developing a GPR model to predict Global Wheat Prices dataset.

The advancement in artificial intelligence and nanotechnology has provided new solutions for tackling problems in enhanced engineering design. This research focuses on both AI assisted observational methodologies and Bayesian uncertainty quantification (BUQ) for improving the predictive models, material properties, and design procedures. Four complex techniques of estimating and managing uncertainty are the following: Bayesian Neural Networks (BNN), Gaussian Processes (GP), Monte Carlo Dropout (MCD), Ensemble Learning (EL). Numerical studies revealed that the forecast accuracy of the proposed framework is 94.6% with BNN and 93.1% with GP, which makes excellent improvements over prior arts of up to 15% in uncertainty quantification. Besides, the computational resources are less by 20% with EL compared to standalone approaches, while the incorporation of nanoscale information increase AT and RT by 17%. To demonstrate that proposed AI-driven BUQ framework addresses the limitations of existing approaches, a comparative discussion is provided. The results reinforce its viability of providing sustainable and efficient engineering solutions under conditions of risk. This work may be used as a platform for subsequent research in the synergies between AI, nanotechnology, and the uncertainty quantification of advanced materials and complex systems to drive progress in material as well as engineering design.

Gaussian processes with built-in dimensionality reduction: Applications to high-dimensional uncertainty propagation

Some scenarios require the computation of a predictive distribution of a new value evaluated on an objective function conditioned on previous observations. We are interested on using a model that makes valid assumptions on the objective function whose values we are trying to predict. Some of these assumptions may be smoothness or stationarity. Gaussian process (GPs) are probabilistic models that can be interpreted as flexible distributions over functions. They encode the assumptions through covariance functions, making hypotheses about new data through a predictive distribution by being fitted to old observations. We can face the case where several GPs are used to model different objective functions. GPs are non-parametric models whose complexity is cubic on the number of observations. A measure that represents how similar is one GP predictive distribution with respect to another would be useful to stop using one GP when they are modelling functions of the same input space. We are really inferring that two objective functions are correlated, so one GP is enough to model both of them by performing a transformation of the prediction of the other function in case of inverse correlation. We show empirical evidence in a set of synthetic and benchmark experiments that GPs predictive distributions can be compared and that one of them is enough to predict two correlated functions in the same input space. This similarity metric could be extremely useful used to discard objectives in Bayesian many-objective optimization.

Surface registration, the task of aligning several multidimensional point sets, is a necessary task in many scientific fields. In this work, a novel statistical approach is developed to solve the problem of nonrigid registration. While the application of an affine transformation results in rigid registration, using a general nonlinear function to achieve nonrigid registration is necessary when the point sets require deformations that change over space. The use of a local likelihood-based approach using windowed Gaussian processes provides a flexible way to accurately estimate the nonrigid deformation. This strategy also makes registration of massive data sets feasible by splitting the data into many subsets. The estimation results yield spatially-varying local rigid registration parameters. Gaussian process surface models are then fit to the parameter fields, allowing prediction of the transformation parameters at unestimated locations, specifically at observation locations in the unregistered data set. Applying these transformations results in a global, nonrigid registration. A penalty on the transformation parameters is included in the likelihood objective function. Combined with smoothing of the local estimates from the surface models, the nonrigid registration model can prevent the problem of overfitting. The efficacy of the nonrigid registration method is tested in two simulation studies, varying the number of windows and number of points, as well as the type of deformation. The nonrigid method is applied to a pair of massive remote sensing elevation data sets exhibiting complex geological terrain, with improved accuracy and uncertainty quantification in a cross validation study versus two rigid registration methods.

Simulations of future climate contain variability arising from a number of sources, including internal stochasticity and external forcings. However, to the best of our abilities climate models and the true observed climate depend on the same underlying physical processes. In this paper, we simultaneously study the outputs of multiple climate simulation models and observed data, and we seek to leverage their mean structure as well as interdependencies that may reflect the climate’s response to shared forcings. Bayesian modeling provides a fruitful ground for the nuanced combination of multiple climate simulations. We introduce one such approach whereby a Gaussian process is used to represent a mean function common to all simulated and observed climates. Dependent random effects encode possible information contained within and between the plurality of climate model outputs and observed climate data. We propose an empirical Bayes approach to analyze such models in a computationally efficient way. This methodology is amenable to the CMIP6 model ensemble, and we demonstrate its efficacy at forecasting global average near-surface air temperature. Results suggest that this model and the extensions it engenders may provide value to climate prediction and uncertainty quantification.

This article presents a study of Gaussian process (GP) models applied to the problems of modeling and data fusion in the context of large-scale terrain modeling. The proposed model naturally provides a multiresolution representation of space, incorporates and handles uncertainties aptly, and copes with incompleteness of sensory information. These attributes are considered essential to support most field robotics applications, including autonomous mining. GP regression techniques are applied to estimate and interpolate (to fill gaps in occluded areas) elevation information across the field. GP approximation methods are introduced to enable the application of the proposed techniques to large data sets. To obtain a comprehensive model of complex terrain, typically, multiple sensory modalities and multiple data sets are required. The GP modeling approach is consequently extended to fuse multiple, multimodal data sets to obtain a best estimate of the elevation given the individual data sets. Two different GP-based concepts are applied to perform data fusion-heteroscedastic GPs and dependent GPs (DGPs). Thus, this article presents a report on an ongoing study of the use of GPs and several GPbased concepts to the problem of large-scale terrain modeling in the context of mining automation.

Gaussian Processes (GPs) are a powerful modelling framework incorporating kernels and Bayesian inference, and are recognised as stateof-the-art for many machine learning tasks. Despite this, GPs have seen few applications in natural language processing (notwithstanding several recent papers by the authors). We argue that the GP framework offers many benefits over commonly used machine learning frameworks, such as linear models (logistic regression, least squares regression) and support vector machines. Moreover, GPs are extremely flexible and can be incorporated into larger graphical models, forming an important additional tool for probabilistic inference. Notably, GPs are one of the few models which support analytic Bayesian inference, avoiding the many approximation errors that plague approximate inference techniques in common use for Bayesian models (e.g. MCMC, variational Bayes).1 GPs accurately model not just the underlying task, but also the uncertainty in the predictions, such that uncertainty can be propagated through pipelines of probabilistic components. Overall, GPs provide an elegant, flexible and simple means of probabilistic inference and are well overdue for consideration of the NLP community. This tutorial will focus primarily on regression and classification, both fundamental techniques of wide-spread use in the NLP community. Within NLP, linear models are near ubiquitous, because they provide good results for many tasks, support efficient inference (including dynamic programming in structured prediction) and support simple parameter interpretation. However, linear models are inherently limited in the types of relationships between variables they can model. Often

Analysis of geospatial data has traditionally been model-based, with a mean model, customarily specified as a linear regression on the covariates, and a Gaussian process covariance model, encoding the spatial dependence. While nonlinear machine learning algorithms like neural networks are increasingly being used for spatial analysis, current approaches depart from the model-based setup and cannot explicitly incorporate spatial covariance. We propose NN-GLS, embedding neural networks directly within the traditional Gaussian process (GP) geostatistical model to accommodate nonlinear mean functions while retaining all other advantages of GP, like explicit modeling of the spatial covariance and predicting at new locations via kriging. In NN-GLS, estimation of the neural network parameters for the nonlinear mean of the Gaussian Process explicitly accounts for the spatial covariance through use of the generalized least squares (GLS) loss, thus, extending the linear case. We show that NN-GLS admits a representation as a special type of graph neural network (GNN). This connection facilitates the use of standard neural network computational techniques for irregular geospatial data, enabling novel and scalable mini-batching, backpropagation, and kriging schemes. We provide methodology to obtain uncertainty bounds for estimation and predictions from NN-GLS. Theoretically, we show that NN-GLS will be consistent for irregularly observed spatially correlated data processes. We also provide a finite sample concentration rate, which quantifies the need to accurately model the spatial covariance in neural networks for dependent data. To our knowledge, these are the first large-sample results for any neural network algorithm for irregular spatial data. We demonstrate the methodology through numerous simulations and an application to air pollution modeling. We develop a software implementation of NN-GLS in the Python package geospaNN. Supplementary materials for this article are available online, including a standardized description of the materials available for reproducing the work.

The Gaussian process (GP) is a popular statistical technique for stochastic function approximation and uncertainty quantification from data. GPs have been adopted into the realm of machine learning (ML) in the last two decades because of their superior prediction abilities, especially in data-sparse scenarios, and their inherent ability to provide robust uncertainty estimates. Even so, their performance highly depends on intricate customizations of the core methodology, which often leads to dissatisfaction among practitioners when standard setups and off-the-shelf software tools are being deployed. Arguably, the most important building block of a GP is the kernel function, which assumes the role of a covariance operator. Stationary kernels of the Matérn class are used in the vast majority of applied studies; poor prediction performance and unrealistic uncertainty quantification are often the consequences. Non-stationary kernels show improved performance but are rarely used due to their more complicated functional form and the associated effort and expertise needed to define and tune them optimally. In this perspective, we want to help ML practitioners make sense of some of the most common forms of non-stationarity for Gaussian processes. We show a variety of kernels in action using representative datasets, carefully study their properties, and compare their performances. Based on our findings, we propose a new kernel that combines some of the identified advantages of existing kernels.

Comparing cell population balance model simulation through Gaussian processes and discretisation