Neural Network Gaussian Processes by Increasing Depth.

This talk addresses the challenge of generating high-resolution wind climatology maps for Switzerland, a region characterized by sparse measurement stations and complex mountainous terrain. Accurately mapping wind patterns in such areas is inherently difficult due to the nonlinear and rapidly changing nature of wind flow, compounded by diverse and abrupt relief. Existing numerical models also lack the spatial granularity and temporal resolution necessary for comprehensive wind mapping.Our objective is twofold. Firstly, we aim to create more detailed wind maps that closely align with observations. Leveraging comprehensive orography maps, topographic descriptors, and numerical model outputs, we seek to downscale wind maps to achieve finer spatial resolution. Secondly, we intend to utilize these downscaled wind maps to compute wind climatology maps, focusing on maximum wind gusts and mean wind velocity, both hourly and daily. These wind climatology maps are invaluable for a plethora of stakeholders such as renewable energy planning, infrastructure development, and environmental monitoring, aiding in informed resource allocation and decision-making.We propose employing machine learning techniques, specifically Gaussian Processes (GPs) and Neural Processes (NPs), to address these challenges. Both present a compelling approach for wind map downscaling as both learn from small datasets and sparse observations to interpolate wind data and generalize to unseen locations. GPs offer a probabilistic framework to model complex relationships between inputs and outputs. By pairing sparse observations, high-resolution topographic descriptors and knowledge of the dynamics via a numerical model output, GPs can infer wind patterns across the territory with improved accuracy and spatial detail. Additionally, GPs inherently provide uncertainty estimates crucial for determining confidence in predictions. Finally, GPs rely on explicit prior knowledge and modeling assumptions, making their predictions interpretable. NPs, on the other hand, leverage neural networks to learn complex, non-linear relationships between inputs and outputs. Furthermore, NPs are scalable and highly efficient in handling large-scale datasets, and their model-less nature allow for more flexibility in modelling complex distributions.Integration of these machine learning techniques with domain-specific knowledge and data sources will enable the development of robust and accurate models for generating high-resolution wind climatology maps. These maps will closely align with observational data, advancing our understanding of local wind patterns and their impact on various applications. This approach aims to foster interdisciplinary collaboration and innovation in this field.Note: Drafting the initial version of this abstract has been aided by AI tools.

The single-layer perceptron, introduced by Rosenblatt in 1958, is one of the earliest and simplest neural network models. However, it is incapable of classifying linearly inseparable patterns. A new era of neural network research started in 1986, when the backpropagation (BP) algorithm was rediscovered for training the multilayer perceptron (MLP) model. An MLP with a large number of hidden nodes can function as a universal approximator. To date, the MLP model is the most fundamental and important neural network model. It is also the most investigated neural network model. Even in this AI or deep learning era, the MLP is still among the few most investigated and used neural network models. Numerous new results have been obtained in the past three decades. This survey paper gives a comprehensive and state-of-the-art introduction to the perceptron model, with emphasis on learning, generalization, model selection and fault tolerance. The role of the perceptron model in the deep learning era is also described. This paper provides a concluding survey of perceptron learning, and it covers all the major achievements in the past seven decades. It also serves a tutorial for perceptron learning.

Predictive analytics can forecast trends and determines statistical probabilities and to act upon fraud and security threats for big data applications such as business trading, fraud detection, crime investigation, banking, insurance, enterprise security, government, healthcare, e-commerce, and telecommunications. Predictive analytics as a service (PAaaS) framework is proposed in our earlier work. One solution based upon ensemble model that uses Gaussian process with varying hyper-parameters is also given in our earlier works. Test results proved that the third hyper-parameter values yielded a good result with less error rate and more variance which is reliable for a predictive model. This paper presents solution based upon ensemble model that uses best out of prediction algorithms such as artificial neural networks (ANN), auto-regression algorithm (ARX) and Gaussian process (GP). Feature engineering methods such as recursive feature elimination that uses random forest algorithm is used for attribute selection. Performance measures NRMSE and COD are used to analyze the model. Test results proved that neural networks performed well when compared to regression and Gaussian process.

Prediction of pricing and hedging errors for equity linked warrants with Gaussian process models☆

ABSTRACTInterferometric gravitational-wave observatories have opened a new era in astronomy. The rich data produced by an international network enable detailed analysis of the curved space-time around black holes. With nearly 100 signals observed so far and thousands expected in the next decade, their population properties enable insights into stellar evolution and the expansion of our Universe. However, the detectors are afflicted by transient noise artefacts known as ‘glitches’ which contaminate the signals and bias inferences. Of the 90 signals detected to date, 18 were contaminated by glitches. This feasibility study explores a new approach to transient gravitational-wave data analysis using Gaussian processes, which model the underlying physics of the glitch-generating mechanism rather than the explicit realization of the glitch itself. We demonstrate that if the Gaussian process kernel function can adequately model the glitch morphology, we can recover the parameters of simulated signals. Moreover, we find that the Gaussian processes kernels used in this work are well suited to modelling long-duration glitches which are most challenging for existing glitch-mitigation approaches. Finally, we show how the time-domain nature of our approach enables a new class of time-domain tests of General Relativity, performing a re-analysis of the inspiral-merger-ringdown test on the first observed binary black hole merger. Our investigation demonstrates the feasibility of the Gaussian processes as an alternative to the traditional framework but does not yet establish them as a replacement. Therefore, we conclude with an outlook on the steps needed to realize the full potential of the Gaussian process approach.

Analysis of geospatial data has traditionally been model-based, with a mean model, customarily specified as a linear regression on the covariates, and a Gaussian process covariance model, encoding the spatial dependence. While nonlinear machine learning algorithms like neural networks are increasingly being used for spatial analysis, current approaches depart from the model-based setup and cannot explicitly incorporate spatial covariance. We propose NN-GLS, embedding neural networks directly within the traditional Gaussian process (GP) geostatistical model to accommodate nonlinear mean functions while retaining all other advantages of GP, like explicit modeling of the spatial covariance and predicting at new locations via kriging. In NN-GLS, estimation of the neural network parameters for the nonlinear mean of the Gaussian Process explicitly accounts for the spatial covariance through use of the generalized least squares (GLS) loss, thus, extending the linear case. We show that NN-GLS admits a representation as a special type of graph neural network (GNN). This connection facilitates the use of standard neural network computational techniques for irregular geospatial data, enabling novel and scalable mini-batching, backpropagation, and kriging schemes. We provide methodology to obtain uncertainty bounds for estimation and predictions from NN-GLS. Theoretically, we show that NN-GLS will be consistent for irregularly observed spatially correlated data processes. We also provide a finite sample concentration rate, which quantifies the need to accurately model the spatial covariance in neural networks for dependent data. To our knowledge, these are the first large-sample results for any neural network algorithm for irregular spatial data. We demonstrate the methodology through numerous simulations and an application to air pollution modeling. We develop a software implementation of NN-GLS in the Python package geospaNN. Supplementary materials for this article are available online, including a standardized description of the materials available for reproducing the work.

We introduce squared neural Poisson point processes (SNEPPPs) by parameterising the intensity function by the squared norm of a two layer neural network. When the hidden layer is fixed and the second layer has a single neuron, our approach resembles previous uses of squared Gaussian process or kernel methods, but allowing the hidden layer to be learnt allows for additional flexibility. In many cases of interest, the integrated intensity function admits a closed form and can be computed in quadratic time in the number of hidden neurons. We enumerate a far more extensive number of such cases than has previously been discussed. Our approach is more memory and time efficient than naive implementations of squared or exponentiated kernel methods or Gaussian processes. Maximum likelihood and maximum a posteriori estimates in a reparameterisation of the final layer of the intensity function can be obtained by solving a (strongly) convex optimisation problem using projected gradient descent. We demonstrate SNEPPPs on real, and synthetic benchmarks, and provide a software implementation.

Separating relevant and irrelevant information is key to any modeling process or scientific inquiry. Theoretical physics offers a powerful tool for achieving this in the form of the renormalization group (RG). Here we demonstrate a practical approach to performing Wilsonian RG in the context of Gaussian Process (GP) Regression. We systematically integrate out the unlearnable modes of the GP kernel, thereby obtaining an RG flow of the GP in which the data sets the IR scale. In simple cases, this results in a universal flow of the ridge parameter, which becomes input-dependent in the richer scenario in which non-Gaussianities are included. In addition to being analytically tractable, this approach goes beyond structural analogies between RG and neural networks by providing a natural connection between RG flow and learnable vs. unlearnable modes. Studying such flows may improve our understanding of feature learning in deep neural networks, and enable us to identify potential universality classes in these models.

Prediction of material behavior using machine learning (ML) requires consistent, accurate, and, representative large data for training. However, such consistent and reliable experimental datasets are not always available for materials. To address this challenge, we synergistically integrate ML with high-throughput reactive molecular dynamics (MD) simulations to elucidate the constitutive relationship of calcium–silicate–hydrate (C–S–H) gel—the primary binding phase in concrete formed via the hydration of ordinary portland cement. Specifically, a highly consistent dataset on the nine elastic constants of more than 300 compositions of C–S–H gel is developed using high-throughput reactive simulations. From a comparative analysis of various ML algorithms including neural networks (NN) and Gaussian process (GP), we observe that NN provides excellent predictions. To interpret the predicted results from NN, we employ SHapley Additive exPlanations (SHAP), which reveals that the influence of silicate network on all the elastic constants of C–S–H is significantly higher than that of water and CaO content. Additionally, the water content is found to have a more prominent influence on the shear components than the normal components along the direction of the interlayer spaces within C–S–H. This result suggests that the in-plane elastic response is controlled by water molecules whereas the transverse response is mainly governed by the silicate network. Overall, by seamlessly integrating MD simulations with ML, this paper can be used as a starting point toward accelerated optimization of C–S–H nanostructures to design efficient cementitious binders with targeted properties.

In the deep learning era, improving the neural network performance in jet physics is a rewarding task, as it directly contributes to more accurate physics measurements at the LHC. Recent research has proposed various network designs in consideration of the full Lorentz symmetry, but its benefit is still not systematically asserted, given that there remain many successful networks without taking it into account. We conduct a detailed study on the Lorentz-symmetric design. We propose two generalized approaches for modifying a network—these methods are experimented on Particle Flow Network, ParticleNet, and LorentzNet and exhibit a general performance gain. We also reveal that the notable improvement attributed to the “pairwise mass” feature in the network is due to its introduction of a structure that fully complies with Lorentz symmetry. We confirm that Lorentz-symmetry preservation serves as a strong inductive bias of jet physics, hence calling for attention to such general recipes in future network designs. Published by the American Physical Society 2024

Elevated temperatures limit the peak performance of systems because of frequent interventions by thermal throttling. Non-uniform thermal states across system nodes also cause performance variation within seemingly equivalent nodes leading to significant degradation of overall performance. In this paper we present a framework for creating a lightweight thermal prediction system suitable for run-time management decisions. We pursue two avenues to explore optimized lightweight thermal predictors. First, we use feature selection algorithms to improve the performance of previously designed machine learning methods. Second, we develop alternative methods using neural network and linear regression-based methods to perform a comprehensive comparative study of prediction methods. We show that our optimized models achieve improved performance with better prediction accuracy and lower overhead as compared with the Gaussian process model proposed previously. Specifically we present a reduced version of the Gaussian process model, a neural network–based model, and a linear regression–based model. Using the optimization methods, we are able to reduce the average prediction errors in the Gaussian process from $4.2^\circ$ C to $2.9^\circ$ C. We also show that the newly developed models using neural network and Lasso linear regression have average prediction errors of $2.9^\circ$ C and $3.8^\circ$ C respectively. The prediction overheads are 0.22, 0.097, and 0.026 ms per prediction for reduced Gaussian process, neural network, and Lasso linear regression models, respectively, compared with 0.57 ms per prediction for the previous Gaussian process model. We have implemented our proposed thermal prediction models on a two-node system configuration to help identify the optimal task placement. The task placement identified by the models reduces the average system temperature by up to $11.9^\circ$ C without any performance degradation. Furthermore, these models respectively achieve 75, 82.5, and 74.17 percent success rates in correctly pointing to those task placements with better thermal response, compared with 72.5 percent success for the original model in achieving the same objective. Finally, we extended our analysis to a 16-node system and we were able to train models and execute them in real time to guide task migration and achieve on average 17 percent reduction in the overall system cooling power.

We propose a theoretical understanding of neural networks in terms of Wilsonian effective field theory. The correspondence relies on the fact that many asymptotic neural networks are drawn from Gaussian processes (GPs), the analog of non-interacting field theories. Moving away from the asymptotic limit yields a non-Gaussian process (NGP) and corresponds to turning on particle interactions, allowing for the computation of correlation functions of neural network outputs with Feynman diagrams. Minimal NGP likelihoods are determined by the most relevant non-Gaussian terms, according to the flow in their coefficients induced by the Wilsonian renormalization group. This yields a direct connection between overparameterization and simplicity of neural network likelihoods. Whether the coefficients are constants or functions may be understood in terms of GP limit symmetries, as expected from ’t Hooft’s technical naturalness. General theoretical calculations are matched to neural network experiments in the simplest class of models allowing the correspondence. Our formalism is valid for any of the many architectures that becomes a GP in an asymptotic limit, a property preserved under certain types of training.

Bayesian optimization (BO) can accelerate materials discovery by guiding expensive experiments toward the most promising processing conditions. We systematically compare five BO surrogate and framework combinations (Gaussian processes in Ax, Gaussian processes and Monte-Carlo neural networks in BayBE, random forests in Lolopy, and tree-structured Parzen (TPE) estimators in Hyperopt) on three benchmarks that mimic common materials design tasks (a discrete solid-electrolyte composition space, a hybrid discrete/continuous laminate-composite design problem solved with micromechanics modeling, and the continuous Ishigami analytic function which is a standard optimization benchmark). Each BO surrogate is paired with posterior mean, probability of improvement, and expected improvement acquisition functions and run for 100 trials from randomized initial samples with uniform random search providing a control. Across five random seeds per setting, BayBE’s Gaussian-process surrogate with expected improvement consistently reached ≥95 % of the known optimum in the fewest evaluations, while Lolopy’s random forest matched or exceeded GP performance on purely categorical or mixed spaces at a higher computational cost. Posterior mean alone often stagnated at local optima, underscoring the need for exploration, whereas probability and expected improvement balanced exploration and exploitation leading to better optimization in fewer trials. Execution times ranged from milliseconds for TPE to minutes for neural-network and random-forest surrogates. These results establish baseline expectations for BO in automated materials laboratories and highlight expected improvement with Gaussian processes as a reliable first choice, with random forests offering a strong alternative when categorical variables dominate. The benchmark suite and code are released to facilitate future surrogate, acquisition, and constraint-handling research in data-driven materials optimization.

Prediction of Length-of-day Variations Based on Gaussian Processes

The computational limitations of complex numerical models have led to adoption of statistical emulators across a variety of problems in science and engineering disciplines to circumvent the high computational costs associated with numerical simulations. In flood modelling, many hydraulic and hydrodynamic numerical models, especially when operating at high spatiotemporal resolutions, have prohibitively high computational costs for tasks requiring the instantaneous generation of very large numbers of simulation results. This study examines the appropriateness and robustness of Gaussian Process (GP) models to emulate the results from a hydraulic inundation model. The developed GPs produce real-time predictions based on the simulation output from LISFLOOD-FP numerical model. An efficient dimensionality reduction scheme is developed to tackle the high dimensionality of the output space and is combined with the GPs to investigate the predictive performance of the proposed emulator for estimation of the inundation depth. The developed GP-based framework is capable of robust and straightforward quantification of the uncertainty associated with the predictions, without requiring additional model evaluations and simulations. Further, this study explores the computational advantages of using a GP-based emulator over alternative methodologies such as neural networks, by undertaking a comparative analysis. For the case study data presented in this paper, the GP model was found to accurately reproduce water depths and inundation extent by classification and produce computational speedups of approximately 10,000 times compared with the original simulator, and 80 times for a neural network-based emulator.