Bayesian Optimization of Expected Quadratic Loss for Multiresponse Computer Experiments with Internal Noise

Subglacial drainage models sensitively depend on the values of numerous uncertain parameters. However, the computation time associated with running these models makes it difficult to quantify the associated uncertainty in model outputs and to use field data to calibrate parameter values. To overcome these computational limitations, we construct a Gaussian Process (GP) emulator that accelerates subglacial drainage modelling by ~1000x. The GP predicts spatiotemporally resolved water pressure as a function of eight model parameters and is trained using ensembles of up to 512 simulations with the Glacier Drainage System (GlaDS) model applied to the Kangerlussuaq sector of the western Greenland Ice Sheet. The GP reproduces the spatial patterns and daily temporal variations simulated by GlaDS within ~4%, with locally higher errors near moulins and during the early melt season. As an application of the GP, we compute the sensitivity of basal water pressure to each of the eight parameters and find that three parameters (ice-flow coefficient, bed bump aspect ratio and the subglacial cavity system conductivity) explain 90% of the variance in model outputs. Next, we explore using a borehole water-pressure timeseries to calibrate the eight uncertain parameters. We take a Bayesian perspective to quantify the uncertainty in parameter estimates and use the GP in place of the physics-based model to make Markov Chain Monte Carlo sampling computationally feasible. We find meaningful constraints relative to the prior assumptions on most parameters and a factor-of-three reduction in uncertainty of the calibrated model predictions. However, significant differences between the calibrated model and the borehole data suggest that structural limitations of the model, rather than poorly constrained parameters or computational cost, remain the most important constraint on subglacial drainage modelling.

Bayesian calibration has been used to transform the prior distributions of unknown inputs inherited in a building energy simulation model into the trustworthy posterior distributions. It obtains the posterior distributions using a joint distribution composed of a likelihood function and prior distributions of unknown inputs in the Bayesian paradigm. In other words, it provides higher benefits in terms of a stochastic approach than the deterministic calibration and the feasibility of their calibrated results was sufficiently discussed. However, challenging issues in Bayesian calibration still remains as follows: (1) inappropriate selection of prior distributions, (2) truncated sample dataset of the likelihood functions. The aforementioned issues can increase the risks of Bayesian calibration. This paper aims to inform the risks of Bayesian calibration associated with the aforementioned issues through a reference case study. For this study, the Gaussian Process (GP) emulator, which can be regarded as a meta-model of Building Performance Simulation (BPS) tools, was used to reduce the simulation run-time. Bayesian calibration using the GP emulator was implemented with what-if scenarios considering the aforementioned issues. And then the validated models were used for a stochastic retrofit analysis of glazing systems. With the results of the estimated posterior distributions, validation, and stochastic retrofit, this paper presents Bayesian calibration issues regarding the selection of prior distributions and sample dataset of the likelihood functions.

Earth observation from satellite sensory data pose challenging problems, where machine learning is currently a key player. In recent years, Gaussian Process (GP) regression and other kernel methods have excelled in biophysical parameter estimation tasks from space. GP regression is based on solid Bayesian statistics, and generally yield efficient and accurate parameter estimates. However, GPs are typically used for inverse modeling based on concurrent observations and in situ measurements only. Very often a forward model encoding the well-understood physical relations is available though. In this work, we review three GP models that respect and learn the physics of the underlying processes in the context of inverse modeling. First, we will introduce a Joint GP (JGP) model that combines in situ measurements and simulated data in a single GP model. Second, we present a latent force model (LFM) for GP modeling that encodes ordinary differential equations to blend data-driven modeling and physical models of the system. The LFM performs multi-output regression, adapts to the signal characteristics, is able to cope with missing data in the time series, and provides explicit latent functions that allow system analysis and evaluation. Finally, we present an Automatic Gaussian Process Emulator (AGAPE) that approximates the forward physical model via interpolation, reducing the number of necessary nodes. Empirical evidence of the performance of these models will be presented through illustrative examples of vegetation monitoring and atmospheric modeling.

This contribution presents a new development in the design of control system based on evolving Gaussian process (GP) models . GP models provide a probabilistic, nonparametric modelling approach for black-box identification of nonlinear dynamic systems. They can highlight areas of the input space where prediction quality is poor, due to the lack of data or its complexity, by indicating the higher variance around the predicted mean. GP models contain noticeably less coefficients to be optimised than commonly used parametric models. While GP models are Bayesian models, their output is normal distribution, expressed in terms of mean and variance. Latter can be interpreted as a confidence in prediction and used in many fields, especially in control system. Evolving GP model is the concept approach within which various ways of model adaptations can be used. Successful control system needs as much as possible data about process to be controlled . If the prior knowledge about the system to be controlled is scarce or the system varies with time or operating region, this control problem can be solved with an iterative method which adapts model with information obtained with streaming data and concurrently optimises hyperparameter values. This contribution provides: a survey of adaptive control algorithms for dynamic systems described in publications where GP models have been used for control design, a novel and improved closed-loop controller with evolving GP models and an example for the illustration of proposed control algorithm.

This book on experimental design is somewhat similar in flavor to the classic book by G.E

Abstract. Subglacial drainage models represent water flow at the ice–bed interface through coupled distributed and channelized systems to determine water pressure, discharge, and drainage system geometry. While they are used to understand processes such as the relationship between surface melt and ice flow, the number of uncertain model parameters and the computational cost of running models makes it difficult to adequately explore the high-dimensional parameter space and evaluate uncertainty in model predictions. Here, we develop Gaussian process (GP) emulators that make fast predictions with associated uncertainty of subglacial drainage model outputs. Using a truncated principal component (PC) basis representation, we construct a GP emulator for diurnally averaged subglacial water pressure. We also explore emulation of scalar variables describing drainage efficiency and configuration. We train the emulators using ensembles of up to 512 simulations varying eight parameters of the Glacier Drainage System (GlaDS) model on a synthetic domain intended to represent an ice-sheet margin. The emulators make predictions ∼ 1000 times faster than GlaDS simulations, with errors <3 % for the water pressure field and ∼ 5 %–9 % for drainage efficiency and configuration. We apply the emulators to explore the eight-dimensional parameter space by computing variance-based parameter sensitivity indices, finding that three parameters (ice flow coefficient, bed bump aspect ratio, and the subglacial cavity system conductivity) explain 90 % of the variance in modelled water pressure in response to parameter changes. The GP emulator approach described here is well suited to integrating observational data with models to make calibrated, credible predictions of subglacial drainage.

The computational limitations of complex numerical models have led to adoption of statistical emulators across a variety of problems in science and engineering disciplines to circumvent the high computational costs associated with numerical simulations. In flood modelling, many hydraulic and hydrodynamic numerical models, especially when operating at high spatiotemporal resolutions, have prohibitively high computational costs for tasks requiring the instantaneous generation of very large numbers of simulation results. This study examines the appropriateness and robustness of Gaussian Process (GP) models to emulate the results from a hydraulic inundation model. The developed GPs produce real-time predictions based on the simulation output from LISFLOOD-FP numerical model. An efficient dimensionality reduction scheme is developed to tackle the high dimensionality of the output space and is combined with the GPs to investigate the predictive performance of the proposed emulator for estimation of the inundation depth. The developed GP-based framework is capable of robust and straightforward quantification of the uncertainty associated with the predictions, without requiring additional model evaluations and simulations. Further, this study explores the computational advantages of using a GP-based emulator over alternative methodologies such as neural networks, by undertaking a comparative analysis. For the case study data presented in this paper, the GP model was found to accurately reproduce water depths and inundation extent by classification and produce computational speedups of approximately 10,000 times compared with the original simulator, and 80 times for a neural network-based emulator.

\nFlood risk analysis often involves the integration of multivariate probability distributions over a domain defined by a consequence function. Often, solutions of this risk integral involves Monte-Carlo sampling techniques, whereby 1000’s of potential flood events are generated. It is necessary to evaluate the consequence of flooding for each sampled event. A significant computational time is required in running flood related physical process models, making it computationally impractical to evaluate flood risk using this approach. To overcome the computational challenges, this paper focusses on the Gaussian Process Emulator (GPE) meta-modelling approach. Traditionally, a “look-up table” method is used when a large number of simulations from a numerical model are required. This approach typically involves simulating conditions defined across a regular matrix, and then linearly interpolating intermediate conditions. In this paper we compare a traditional “look-up table” approach to the GPE and analyse their performance in approximating SWAN wave transformation model. In both cases, selecting an appropriate training design set is important and is taken into consideration in the analysis. The analysis shows that the GPE approach requires significantly fewer SWAN runs to obtain similar (or better) accuracies, enabling a substantial reduction in computation time, hence aiding the practicality of Monte-Carlo sampling techniques in advanced flood risk modelling.\n

Abstract. Sensitivity analysis of atmospheric models is necessary to identify the processes that lead to uncertainty in model predictions, to help understand model diversity through comparison of driving processes, and to prioritise research. Assessing the effect of parameter uncertainty in complex models is challenging and often limited by CPU constraints. Here we present a cost-effective application of variance-based sensitivity analysis to quantify the sensitivity of a 3-D global aerosol model to uncertain parameters. A Gaussian process emulator is used to estimate the model output across multi-dimensional parameter space, using information from a small number of model runs at points chosen using a Latin hypercube space-filling design. Gaussian process emulation is a Bayesian approach that uses information from the model runs along with some prior assumptions about the model behaviour to predict model output everywhere in the uncertainty space. We use the Gaussian process emulator to calculate the percentage of expected output variance explained by uncertainty in global aerosol model parameters and their interactions. To demonstrate the technique, we show examples of cloud condensation nuclei (CCN) sensitivity to 8 model parameters in polluted and remote marine environments as a function of altitude. In the polluted environment 95 % of the variance of CCN concentration is described by uncertainty in the 8 parameters (excluding their interaction effects) and is dominated by the uncertainty in the sulphur emissions, which explains 80 % of the variance. However, in the remote region parameter interaction effects become important, accounting for up to 40 % of the total variance. Some parameters are shown to have a negligible individual effect but a substantial interaction effect. Such sensitivities would not be detected in the commonly used single parameter perturbation experiments, which would therefore underpredict total uncertainty. Gaussian process emulation is shown to be an efficient and useful technique for quantifying parameter sensitivity in complex global atmospheric models.

The decision to terminate a controlled clinical trial at the time of an interim analysis is perhaps best made by weighing the value of the likely additional information to be gained if further subjects are enrolled against the various costs of that further enrollment. The most commonly used statistical plans for interim analysis (eg, O'Brien-Fleming), however, are based on a frequentist approach that makes no such comparison. A two-armed Bayesian decision-theoretic clinical trial design is developed for a disease with two possible outcomes, incorporating a quadratic decision loss function and using backward induction to quantify the cost of future enrollment. Monte Carlo simulation is used to compare frequentist error rates and mean required sample sizes for these Bayesian designs with the two-tailed frequentist group-sequential designs of, O'Brien-Fleming and Pocock. When the terminal decision loss function is chosen to yield typical frequentist error rates, the mean sample sizes required by the Bayesian designs are smaller than those of the corresponding O'Brien-Fleming frequentist designs, largely due to the more frequent interim analyses typically used with the Bayesian designs and the ability of the Bayesian designs to terminate early and conclude equivalence. Adding stochastic curtailment to the frequentist designs and using the same number of interim analyses results in largely equivalent trials. An example of a Bayesian design for the data safety monitoring of a clinical trial is given. Our design assumes independence of the probabilities of success in the two trial arms. Additionally, we have chosen non-informative priors and selected loss functions to produce trials with appealing frequentist error rates, rather than choosing priors that reflect realistic prior information and loss functions that reflect true costs. Our Bayesian designs allow interpretation of the final results along either Bayesian or frequentist lines. For the Bayesian, they minimize the total cost and allow the direct calculation of the probability density function for the difference in efficacy. For the frequentist, they have well-characterized type I and II error rates and in some cases lead to a reduction in the mean sample size.

Gaussian processes (Kriging) are interpolating data-driven models that are frequently applied in various disciplines. Often, Gaussian processes are trained on datasets and are subsequently embedded as surrogate models in optimization problems. These optimization problems are nonconvex and global optimization is desired. However, previous literature observed computational burdens limiting deterministic global optimization to Gaussian processes trained on few data points. We propose a reduced-space formulation for deterministic global optimization with trained Gaussian processes embedded. For optimization, the branch-and-bound solver branches only on the free variables and McCormick relaxations are propagated through explicit Gaussian process models. The approach also leads to significantly smaller and computationally cheaper subproblems for lower and upper bounding. To further accelerate convergence, we derive envelopes of common covariance functions for GPs and tight relaxations of acquisition functions used in Bayesian optimization including expected improvement, probability of improvement, and lower confidence bound. In total, we reduce computational time by orders of magnitude compared to state-of-the-art methods, thus overcoming previous computational burdens. We demonstrate the performance and scaling of the proposed method and apply it to Bayesian optimization with global optimization of the acquisition function and chance-constrained programming. The Gaussian process models, acquisition functions, and training scripts are available open-source within the “MeLOn—MachineLearning Models for Optimization” toolbox (https://git.rwth-aachen.de/avt.svt/public/MeLOn).

The Department of Energy relies on complex physics simulations for prediction in domains like cosmology, nuclear theory, and materials science. These simulations are often extremely computationally intensive, with some requiring days or weeks for a single simulation. In order to assure their accuracy, these models are calibrated against observational data in order to estimate inputs and systematic biases. Because of their great computational complexity, this process typically requires the construction of an emulator, a fast approximation to the simulation. In this paper, two emulator approaches are compared: Gaussian process regression and neural networks. Their emulation accuracy and calibration performance on three real problems of Department of Energy interest is considered. On these problems, the Gaussian process emulator tends to be more accurate with narrower, but still well‐calibrated uncertainty estimates. The neural network emulator is accurate, but tends to have large uncertainty on its predictions. As a result, calibration with the Gaussian process emulator produces more constrained posteriors that still perform well in prediction.

Quantitative prediction of subjective pain intensity from whole-brain fMRI data using Gaussian processes

To analyze the performance of various estimators, this paper compares Bayesian and classical estimate approaches across a variety of loss functions. The work focuses on informative versus non-informative priors in Bayesian estimation. The calculations are based on a number of loss functions, including the Square Error Loss Function (SELF), Quadratic Loss Function (QLF), Precautionary Loss Function (PLF), and Entropy Loss Function (ELF). The methodology uses simulation techniques as well as real-world datasets to test the performance and robustness of the proposed estimators. The Bayesian estimators, classical estimators, and related Bayes hazards using empirical analysis and extensive simulations to computed. The findings show that the Minimum Mean Square Error estimator (MiniMSE) is more accurate and reliable than other estimators in a variety of settings. Furthermore, for both types of priors, it is demonstrated that the Bayes risk under the Quadratic Loss Function (QLF) is the lowest among all loss functions considered, including SELF, PLF, and ELF. This suggests that, when compared to other loss functions, QLF is a more effective and balanced criterion for making estimation decisions. The study’s findings provide vital new insights into how to choose the optimum estimate techniques and loss functions for statistical decision theory and practical data processing.

For the challenging task of modeling actual complex motions, we propose a new class of Gaussian process (GP) models that are data-driven and also take into account prior knowledge of the motion intention. As a theoretical basis, we show that the GP regression is mathematically equivalent to regularized least-squares estimation for random functions with known prior means. Compared with the popular GP models in machine learning literature, the proposed GP motion model priors with conditional kernels have at least two advantages: 1) they are nonstationary and more applicable to represent complex motions by integrating the basic kinematic principles; 2) conditional kernels are further devised by incorporating the motion intent so that the resultant GP models are more versatile and would expectedly entail more accurate trajectory prediction. A superior property of the GP models with conditional kernels is found to improve the computational efficiency. Finally, illustrative examples are provided to show the superiority of the proposed motion models and to verify the theoretical results given in the paper.