Navigating Small Datasets with Machine Learning: Gaussian Process Modeling for Colloidal Gelation.

Subglacial drainage models sensitively depend on the values of numerous uncertain parameters. However, the computation time associated with running these models makes it difficult to quantify the associated uncertainty in model outputs and to use field data to calibrate parameter values. To overcome these computational limitations, we construct a Gaussian Process (GP) emulator that accelerates subglacial drainage modelling by ~1000x. The GP predicts spatiotemporally resolved water pressure as a function of eight model parameters and is trained using ensembles of up to 512 simulations with the Glacier Drainage System (GlaDS) model applied to the Kangerlussuaq sector of the western Greenland Ice Sheet. The GP reproduces the spatial patterns and daily temporal variations simulated by GlaDS within ~4%, with locally higher errors near moulins and during the early melt season. As an application of the GP, we compute the sensitivity of basal water pressure to each of the eight parameters and find that three parameters (ice-flow coefficient, bed bump aspect ratio and the subglacial cavity system conductivity) explain 90% of the variance in model outputs. Next, we explore using a borehole water-pressure timeseries to calibrate the eight uncertain parameters. We take a Bayesian perspective to quantify the uncertainty in parameter estimates and use the GP in place of the physics-based model to make Markov Chain Monte Carlo sampling computationally feasible. We find meaningful constraints relative to the prior assumptions on most parameters and a factor-of-three reduction in uncertainty of the calibrated model predictions. However, significant differences between the calibrated model and the borehole data suggest that structural limitations of the model, rather than poorly constrained parameters or computational cost, remain the most important constraint on subglacial drainage modelling.

This talk addresses the challenge of generating high-resolution wind climatology maps for Switzerland, a region characterized by sparse measurement stations and complex mountainous terrain. Accurately mapping wind patterns in such areas is inherently difficult due to the nonlinear and rapidly changing nature of wind flow, compounded by diverse and abrupt relief. Existing numerical models also lack the spatial granularity and temporal resolution necessary for comprehensive wind mapping.Our objective is twofold. Firstly, we aim to create more detailed wind maps that closely align with observations. Leveraging comprehensive orography maps, topographic descriptors, and numerical model outputs, we seek to downscale wind maps to achieve finer spatial resolution. Secondly, we intend to utilize these downscaled wind maps to compute wind climatology maps, focusing on maximum wind gusts and mean wind velocity, both hourly and daily. These wind climatology maps are invaluable for a plethora of stakeholders such as renewable energy planning, infrastructure development, and environmental monitoring, aiding in informed resource allocation and decision-making.We propose employing machine learning techniques, specifically Gaussian Processes (GPs) and Neural Processes (NPs), to address these challenges. Both present a compelling approach for wind map downscaling as both learn from small datasets and sparse observations to interpolate wind data and generalize to unseen locations. GPs offer a probabilistic framework to model complex relationships between inputs and outputs. By pairing sparse observations, high-resolution topographic descriptors and knowledge of the dynamics via a numerical model output, GPs can infer wind patterns across the territory with improved accuracy and spatial detail. Additionally, GPs inherently provide uncertainty estimates crucial for determining confidence in predictions. Finally, GPs rely on explicit prior knowledge and modeling assumptions, making their predictions interpretable. NPs, on the other hand, leverage neural networks to learn complex, non-linear relationships between inputs and outputs. Furthermore, NPs are scalable and highly efficient in handling large-scale datasets, and their model-less nature allow for more flexibility in modelling complex distributions.Integration of these machine learning techniques with domain-specific knowledge and data sources will enable the development of robust and accurate models for generating high-resolution wind climatology maps. These maps will closely align with observational data, advancing our understanding of local wind patterns and their impact on various applications. This approach aims to foster interdisciplinary collaboration and innovation in this field.Note: Drafting the initial version of this abstract has been aided by AI tools.

We present a novel computational approach for extracting localized signals from smooth background distributions. We focus on datasets that can be naturally presented as binned integer counts, demonstrating our procedure on the CERN open dataset with the Higgs boson signature, from the ATLAS collaboration at the Large Hadron Collider. Our approach is based on Gaussian Process (GP) regression — a powerful and flexible machine learning technique which has allowed us to model the background without specifying its functional form explicitly and separately measure the background and signal contributions in a robust and reproducible manner. Unlike functional fits, our GP-regression-based approach does not need to be constantly updated as more data becomes available. We discuss how to select the GP kernel type, considering trade-offs between kernel complexity and its ability to capture the features of the background distribution. We show that our GP framework can be used to detect the Higgs boson resonance in the data with more statistical significance than a polynomial fit specifically tailored to the dataset. Finally, we use Markov Chain Monte Carlo (MCMC) sampling to confirm the statistical significance of the extracted Higgs signature.

Physics-constrained machine learning is emerging as an important topic in the field of machine learning for physics. One of the most significant advantages of incorporating physics constraints into machine learning methods is that the resulting model requires significantly less data to train. By incorporating physical rules into the machine learning formulation itself, the predictions are expected to be physically plausible. Gaussian process (GP) is perhaps one of the most common methods in machine learning for small datasets. In this paper, we investigate the possibility of constraining a GP formulation with monotonicity on three different material datasets, where one experimental and two computational datasets are used. The monotonic GP is compared against the regular GP, where a significant reduction in the posterior variance is observed. The monotonic GP is strictly monotonic in the interpolation regime, but in the extrapolation regime, the monotonic effect starts fading away as one goes beyond the training dataset. Imposing monotonicity on the GP comes at a small accuracy cost, compared to the regular GP. The monotonic GP is perhaps most useful in applications where data are scarce and noisy, and monotonicity is supported by strong physical evidence.

Pressure gradient (PG) in liquid-liquid flow is one of the key components to design an energy-efficient transportation system for wellbores. This study aims to develop five robust machine learning (ML) algorithms and their fusions for a wide range of flow patterns (FP) regimes. The MLs include Support Vector Machine (SVM), Gaussian Process (GP), Random Forest (RF), Artificial Neural Network (ANN), k-Nearest Neighbor (kNN), and fusions of these five MLs. A total of eleven hundred experimental data points for nine FPs (two stratified and seven dispersed patterns) in horizontal wellbores are used to develop the MLs. The MLs' performance is evaluated using the metrics including mean absolute percentage error (MAPE), median absolute percentage error (MdAPE), coefficient of variation of root mean squared error (CV-RMSE), and adjusted coefficient of determination. The evaluation metrics are cross-validated using a repeated train-test split strategy. Seven important predictor variables are identified using a supervised feature selection approach: oil and water velocities, FP, input diameter, oil and water density, and oil viscosity. The results show that the high PG prediction accuracy can be achieved using GP compared to other MLs except for the ML-fusions (p < 0.05). A Friedman's test and Wilcoxon Sign-Rank post hoc analysis with Bonferroni correction show that PG prediction errors using GP are significantly lower than using the ANN model (p < 0.05). The values are 18.44 % and 23.9 % for CV-RMSE, 11.6 % and 10.06 % for MAPE, and 7.5 % and 6.75 % for MdAPE, using ANN and GP, respectively. While the previous studies mostly used ANN to demonstrate the capability of MLs to predict PG over the mechanistic or correlation-based models, the present research has shown that GP is even better than ANN using a wide range of FPs and a large data set. • Machine-learning algorithms are developed to predict the pressure gradient. • The Gaussian Process model produces significantly lower prediction error. • The median absolute percentage error using Gaussian Process is 6.75 %. • More than 95 % variation of pressure gradient can be explained using seven variables.

This paper presents a consistent Bayesian solution for data integration and history matching for oil reservoirs while accounting for both model and parameter uncertainties. The developed method uses Gaussian Process Regression to build a permeability map conforming to collected data at well bores. Following that, an augmented Markov Chain Monte Carlo sampler is used to condition the permeability map to dynamic production data. The selected proposal distribution for the Markov Chain Monte Carlo conforms to the Gaussian process regression output. The augmented Markov Chain Monte Carlo sampler allows transition steps between different models of the covariance function, and hence both the parameter and model space are effectively explored. In contrast to single model Markov Chain Monte Carlo samplers, the proposed augmented Markov Chain Monte Carlo sampler eliminates the selection bias of certain covariance structures of the inferred permeability field. The proposed algorithm can be used to account for general model and parameter uncertainties.

Comprehensive evaluation and systematic comparison of Gaussian process (GP) modelling applications in peptide quantitative structure-activity relationship

This article describes a method for modelling non-linear dynamic systems from measurement data. The method merges the linear local model blending approach in the velocity-based linearisation form with Bayesian Gaussian process (GP) modelling. The new Fixed-Structure GP (FSGP) model has a predetermined linear model structure with varying and probabilistic parameters represented by GP models. These models have several advantages for the modelling of local model parameters as they give us adequate results, even with small data sets. Furthermore, they provide a measure of the confidence in the prediction of the varying parameters and information about the dependence of the parameters on individual inputs. The FSGP model can be applied for the extended local linear equivalence class of non-linear systems. The obtained non-linear system model can be, for example, used for control-system design. The proposed modelling method is illustrated with a simple example of non-linear system modelling for control design.

Earth observation from satellite sensory data pose challenging problems, where machine learning is currently a key player. In recent years, Gaussian Process (GP) regression and other kernel methods have excelled in biophysical parameter estimation tasks from space. GP regression is based on solid Bayesian statistics, and generally yield efficient and accurate parameter estimates. However, GPs are typically used for inverse modeling based on concurrent observations and in situ measurements only. Very often a forward model encoding the well-understood physical relations is available though. In this work, we review three GP models that respect and learn the physics of the underlying processes in the context of inverse modeling. First, we will introduce a Joint GP (JGP) model that combines in situ measurements and simulated data in a single GP model. Second, we present a latent force model (LFM) for GP modeling that encodes ordinary differential equations to blend data-driven modeling and physical models of the system. The LFM performs multi-output regression, adapts to the signal characteristics, is able to cope with missing data in the time series, and provides explicit latent functions that allow system analysis and evaluation. Finally, we present an Automatic Gaussian Process Emulator (AGAPE) that approximates the forward physical model via interpolation, reducing the number of necessary nodes. Empirical evidence of the performance of these models will be presented through illustrative examples of vegetation monitoring and atmospheric modeling.

In this work, we discuss practical methods for the assessment, comparison, and selection of complex hierarchical Bayesian models. A natural way to assess the goodness of the model is to estimate its future predictive capability by estimating expected utilities. Instead of just making a point estimate, it is important to obtain the distribution of the expected utility estimate because it describes the uncertainty in the estimate. The distributions of the expected utility estimates can also be used to compare models, for example, by computing the probability of one model having a better expected utility than some other model. We propose an approach using cross-validation predictive densities to obtain expected utility estimates and Bayesian bootstrap to obtain samples from their distributions. We also discuss the probabilistic assumptions made and properties of two practical cross-validation methods, importance sampling and k-fold cross-validation. As illustrative examples, we use multilayer perceptron neural networks and gaussian processes with Markov chain Monte Carlo sampling in one toy problem and two challenging real-world problems.

Gaussian Processes (GPs) are a powerful modelling framework incorporating kernels and Bayesian inference, and are recognised as stateof-the-art for many machine learning tasks. Despite this, GPs have seen few applications in natural language processing (notwithstanding several recent papers by the authors). We argue that the GP framework offers many benefits over commonly used machine learning frameworks, such as linear models (logistic regression, least squares regression) and support vector machines. Moreover, GPs are extremely flexible and can be incorporated into larger graphical models, forming an important additional tool for probabilistic inference. Notably, GPs are one of the few models which support analytic Bayesian inference, avoiding the many approximation errors that plague approximate inference techniques in common use for Bayesian models (e.g. MCMC, variational Bayes).1 GPs accurately model not just the underlying task, but also the uncertainty in the predictions, such that uncertainty can be propagated through pipelines of probabilistic components. Overall, GPs provide an elegant, flexible and simple means of probabilistic inference and are well overdue for consideration of the NLP community. This tutorial will focus primarily on regression and classification, both fundamental techniques of wide-spread use in the NLP community. Within NLP, linear models are near ubiquitous, because they provide good results for many tasks, support efficient inference (including dynamic programming in structured prediction) and support simple parameter interpretation. However, linear models are inherently limited in the types of relationships between variables they can model. Often

This paper provides an overview of the following three modeling approaches – Physics-based Modeling, Machine Learning, and Hybrid Modeling (Physics-based Modeling &amp; Machine Learning combined), and addresses their applicability for multiphase flow simulations in specific use-cases. Physics-based modeling builds on well understood concepts in, e.g., thermodynamics, fluid dynamics, fluid modeling and optimization techniques. It requires deep domain knowledge as well as accurate fluid data and may incur significant computational cost. Machine Learning systems are based on learning algorithms, which find relationships between sensor data and output variables in a training dataset. The approach requires a good understanding of the learning algorithms and statistics. Small datasets or changes in operational conditions limit the suitability of this approach. Hybrid Models combine Physics with Machine Learning, and these models are on a sliding scale between pure Physics-based Models and pure Machine Learning Models. The individual use-case defines how the Hybrid Model is configured and where it sits on the sliding scale. Here, we will investigate three different use-cases: Physics supporting Machine Learning: Use physics to generate data to support training of machine learning models when data is sparse. Machine Learning supporting Physics: Use machine learning to provide additional input to physics modeling when available data does not suffice to achieve accurate numerical solutions. True Hybrid: Close the circle and use physics for feature engineering to improve the machine learning models, which then provide synthetic data as input to physics-based simulations. We observe that there is not one universal solution. On the one hand, flow simulations may address situations which are stochastic in nature and not deterministic. Here, a machine learning model based on physics-based simulations fits the purpose well. On the other hand, there are situations when machine learning can deduce relations between parameters such that you can provide additional input into physics-based models. One such example is to address the oil-water split at very high water cuts. By adopting the appropriate combination, the hybrid approach results in superior accuracy and offers the ability to address a broader range of applications than physics or machine learning alone.

Energy storage technology helps make possible many of the freedoms and personal mobility enjoyed by today's society. In consumer electronics for example, devices such as cell phones and laptops have experienced a great surge in popularity in large-part thanks to advances in battery energy density. In addition to providing convenience, energy storage technology is also a critical enabler on the pathway to the low carbon lifestyle required for future sustainable societies. For these reasons, worldwide interest in energy storage research continues to grow. In conventional vehicle applications, lead acid batteries remain the preferred low-cost solution, providing good cranking power capability to start internal combustion engines at low temperatures. But for electric traction applications, more energy and power-dense chemistries are required. The nickel-metal hydride chemistry, with roughly double the power and energy of the lead acid chemistry, has functioned for some years as the battery of choice for mild and medium hybrid electric vehicles. But to effectively transition our vehicles to run on electricity rather than fossil-based fuels requires substantial further advancement in energy dense, long life, low cost energy storage. The lithium ion chemistry is presently poised to enter transportation markets. Lithium ion battery safety remains a concern, however, given the energetic nature of the chemistry. Another compelling technology for power and cycling-intensive transportation applications are electrochemical ultracapacitors that function based on a double layer charge separation, rather than faradic mechanism. Given the intermittency of sun and wind availability, widespread penetration of renewable energy also requires some means to store it, often for many hours. Today's lithium ion chemistries are too expensive for such large-scale applications. Less-expensive traditional technologies such as lead acid suffer from short life. In certain geographic locations, compressed air and pumped hydraulic power may have a role in renewable energy storage. Novel electrochemical technologies, such as the vanadium redox flow battery, readily scalable and with promising life, may also have a role. Battery science and engineering spans diverse subject areas ranging from electrochemistry to materials science, mechanical and electrical engineering. Battery research, by definition, is a multidisciplinary endeavor. Taking lithium ion safety as an example, safety improvements are sought on multiple fronts. Good design and selection of stable materials, careful quality control during manufacture, proper thermal and electrical design of battery packs under normal and abusive conditions, and sound design of monitoring electronics, algorithms and controls are all needed to produce a safe, high-energy battery. For all storage chemistries, breakthrough advances will no doubt come from research groups that merge the boundaries of several of these traditional science and engineering disciplines. In the theme of multidisciplinary energy storage research, this special issue of International Journal of Energy Research compiles overview and research articles from a broad array of leading researchers in industry, government laboratories and academia. We trust that the findings compiled herein will motivate further investigation in the rapidly evolving field of energy storage.

Shear viscosity, though being a fundamental property of all fluids, is computationally expensive to calculate from equilibrium molecular dynamics simulations. Recently, machine learning (ML) methods have been used to augment molecular simulations in many contexts, thus showing promise to estimate viscosity too in a relatively inexpensive manner. However, ML methods face significant challenges—such as overfitting, when the size of the data set is small, as is the case with viscosity. In this work, we train seven ML models to predict the shear viscosity of a Lennard–Jones fluid, with particular emphasis on addressing issues arising from a small data set. Specifically, the issues related to model selection, performance estimation and uncertainty quantification were investigated. First, we show that the widely used performance estimation procedure of using a single unseen data set shows a wide variability—in estimating the errors on—small data sets. In this context, the common practice of using cross validation (CV) to select the hyperparameters (model selection) can be adapted to estimate the generalization error (performance estimation) as well. We compare two simple CV procedures for their ability to do both model selection and performance estimation, and find that k-fold CV based procedure shows a lower variance of error estimates. Also, these CV procedures naturally lead to an ensemble of trained ML models. We discuss the role of performance metrics in training and evaluation and propose a method to rank the ML models based on multiple metrics. Finally, two methods for uncertainty quantification—Gaussian process regression (GPR) and ensemble method—were used to estimate the uncertainty on individual predictions. The uncertainty estimates from GPR were also used to construct an applicability domain using which the ML models provided even more reliable predictions on an independent viscosity data set generated in this work. Overall, the procedures prescribed in this work, together, lead to robust ML models for small data sets.

Gaussian Processes (GPs) are widely-used tools in spatial statistics and machine learning and the formulae for the mean function and covariance kernel of a GP [Formula: see text] that is the image of another GP [Formula: see text] under a linear transformation [Formula: see text] acting on the sample paths of [Formula: see text] are well known, almost to the point of being folklore. However, these formulae are often used without rigorous attention to technical details, particularly when [Formula: see text] is an unbounded operator such as a differential operator, which is common in many modern applications. This note provides a self-contained proof of the claimed formulae for the case of a closed, densely-defined operator [Formula: see text] acting on the sample paths of a square-integrable (not necessarily Gaussian) stochastic process. Our proof technique relies upon Hille’s theorem for the Bochner integral of a Banach-valued random variable.