Uncertainty analysis of hydrological multi-model ensembles based on CBP-BMA method

Purpose Low-dose radiation risks are generally extrapolated from groups exposed at high levels of dose. Measurement error substantially alters dose-response shape and hence extrapolated risk. Much attention has been paid to methods of dealing with shared errors, common in many datasets, in particular using Bayesian model averaging (BMA) methods. Materials and methods We test two types of BMA model using simulated data. The first we term the quasi-two-dimensional Monte Carlo with BMA (quasi-2DMC + BMA) method, similar to the BMA method proposed by Hoeting et al. but distinct from the 2DMC + BMA method of Kwon et al. The second we term the marginal-quasi-2DMC + BMA method, which uses a more complicated marginal calculation, and may be closer to the 2DMC + BMA method of Kwon et al. Results Assuming a true linear model of dose response, the coverage probabilities for the linear coefficient are 90–95%, for the quasi-2DMC + BMA method, but lower than this, 52–60%, for the marginal-quasi-2DMC + BMA method. Assuming a true linear-quadratic model the coverage probabilities of both linear/quadratic excess relative risk (ERR) coefficients for quasi-2DMC + BMA are too low, and when shared Berkson error is large (50%) the probabilities do not exceed 5%. By comparison, the coverage probabilities for both linear and quadratic coefficients for the marginal-quasi-2DMC + BMA method are generally too high, about 100%. For the linear model quasi-2DMC + BMA yields good estimates of the ERR coefficient, but for marginal-quasi-2DMC + BMA these are upwardly biased. For the linear-quadratic model both quasi-2DMC + BMA and marginal-quasi-2DMC + BMA methods yield substantially biased estimates. Conclusions The performance of both quasi-2DMC + BMA and marginal-quasi-2DMC + BMA methods is bad, with bias and poor coverage.

A hydrologic uncertainty processor (HUP) that the prior density and likelihood function are explicitly expressed is introduced based on a copula function. The results show that the copula-based HUP is comparable to the meta-Gaussian HUP regarding the posterior median forecasts. Besides, the probabilistic forecasts produced by the copula-based HUP have slightly higher reliability and lower resolution, as compared to those of the meta-Gaussian HUP. A Copula Bayesian processor associated with the Bayesian model averaging (CBP-BMA) method is used with ensemble lumped hydrological models. Compared with the BMA and Copula-BMA methods, the CBP-BMA method relaxes any assumption on the distribution of conditional PDFs. The case study results demonstrate that the CBP-BMA method can improve hydrological forecasting precision with higher coverage ratios. The copula-based HUP and CBP-BMA methods provide alternative approaches to uncertainty analysis of hydrological forecasts.

This paper briefly discusses the nature, the causes and the role of predictive uncertainty in flood forecasting and proposes a novel approach to its estimation. Following the definition of predictive uncertainty, its importance in the decision process is highlighted in relation to the different sources of errors (model, parameter, observations, boundary conditions) that affect flood forecasting. Moreover, the paper briefly analyses the importance of using a full predictive uncertainty, obtained by marginalising the parameter uncertainty, instead of the predictive uncertainty conditional to a single parameter set. Finally, a new Model Conditional Processor (MCP) for the assessment of predictive uncertainty is then proposed as an alternative to the Hydrologic Uncertainty Processor (HUP) introduced by Krzysztofowicz as well as to the Bayesian Model Averaging (BMA) approach due to Raftery et al. The new MCP approach, which aims at assessing, and possibly reducing, predictive uncertainty, allows combination of the observations with one or several models’ forecasts in a multi‐Normal space, by transforming observations and model forecasts in a multivariate Normal space by means of the Normal Quantile Transform. The results of the new approach are shown for the case of the River Po in Italy, and compared with the results obtainable both with the HUP and the BMA.

<p>In recent years, the number of large-scale hydrological forecasting systems has been steadily growing. This may lead to regions having numerous models spatially overlapping each other. Some of these regions have what we will refer to as a regional, more specialized, model for the area that performs generally better than their large-scale counterpart, considering the coarser spatial resolution and sometimes lack of calibration of the latter. Our work explored the possibility of using simple methods to retrieve hydrological information from a large-scale model, namely the National Surface and River Prediction System (NSRPS) that will eventually cover the Canadian territory, in order to improve the forecasts from a local system, namely the Système de Prévision Hydrologique (SPH) that covers most of the province of Quebec. Outputs from the two forecasting systems were thus combined using methods including the simple mean, a weighted average in which the weights are optimized either using the Kling-Gupta Efficiency (KGE) or the Continuous Ranked Probability Score (CRPS) as cost functions, or weights calculated from the residual errors of the forecasts. Bayesian Model Averaging (BMA) was also explored to combine the ensemble forecasts from both systems. The results show that it is possible to improve the local hydrological forecasts by using simple weighted combinations with forecasts from the large-scale system. Performance was assessed using many well-known criteria such as the Nash-Sutcliffe Efficiency (NSE), KGE and CRPS. Results were averaged over the 61 available gauging stations and analyzed at lead times ranging from 3 to 120 hours. We observed improvements in all criteria for lead times over 60 hours as well as no loss in performance at any lead times. Finally, the methods were also used in a leave-one-out setup to simulate performance on ungauged basins. The performance gain for ungauged basins is similar to that of the gauged basins, hinting at the fact that these simple methods could also improve forecasts in more remote territories where no measurements are available.</p>

Despite the same purpose, each satellite product has different value because of its inescapable uncertainty. Also the satellite products have been calculated for a long time, and the kinds of the products are various and enormous. So the efforts for reducing the uncertainty and dealing with enormous data will be necessary. In this paper, we create an ensemble Sea Surface Temperature (SST) using MODIS Aqua, MODIS Terra and COMS (Communication Ocean and Meteorological Satellite). We used Bayesian Model Averaging (BMA) as ensemble method. The principle of the BMA is synthesizing the conditional probability density function (PDF) using posterior probability as weight. The posterior probability is estimated using EM algorithm. The BMA PDF is obtained by weighted average. As the result, the ensemble SST showed the lowest RMSE and MAE, which proves the applicability of BMA for satellite data ensemble. As future work, parallel processing techniques using Hadoop framework will be adopted for more efficient computation of very big satellite data.

Using high-quality dataset from 12 flux towers in north China, the performance of four evapotranspiration (ET) models and the multi-model ensemble approaches including the simple averaging (SA) and Bayesian model average (BMA) were systematically evaluated in this study. The four models were the single-layer Penman–Monteith (P–M) model, the two-layer Shuttleworthe–Wallace (S–W) model, the advection–aridity (A–A) model, and a modified Priestley–Taylor (PT-JPL). Based on the mean value of Taylor skill (S) and the regression slope between measured and simulated ET values across all sites, the order of overall performance of the individual models from the best to the worst were: S–W (0.88, 0.87), PT-JPL (0.80, 1.17), P–M (0.63, 1.73) and A–A (0.60, 1.68) [statistics stated as (Taylor skill, regression slope)]. Here, all models used the same values of parameters, LAI and fractional vegetation cover as well as the forcing meteorological data. Thus, the differences in model performance were mainly attributed to errors in model structure. To the ensemble approach, the BMA method has the advantage of generating more skillful and reliable predictions than the SA scheme. However, successful implementation of BMA requires accurate estimates of its parameters, and some degradation in performance were observed when the BMA parameters generated from the training period were used for the validation period. Thus, it is necessary to explore the seasonal variations of the BMA parameters according the different growth stages. Finally, the optimal conditional density function of half-hourly ET approximated well by the double-exponential distribution. Copyright © 2016 John Wiley & Sons, Ltd.

To study the impact of training sample heterogeneity on the performance of Bayesian model averaging (BMA), two BMA experiments were performed on probabilistic quantitative precipitation forecasts (PQPFs) in the northern China region in July and August of 2010 generated from an 11-member short-range ensemble forecasting system. One experiment, as in many conventional BMA studies, used an overall training sample that consisted of all available cases in the training period, while the second experiment used stratified sampling BMA by first dividing all available training cases into subsamples according to their ensemble spread, and then performing BMA on each subsample. The results showed that ensemble spread is a good criterion to divide ensemble precipitation cases into subsamples, and that the subsamples have different statistical properties. Pooling the subsamples together forms a heterogeneous overall sample. Conventional BMA is incapable of interpreting heterogeneous samples, and produces unreliable PQPF. It underestimates the forecast probability at high-threshold PQPF and local rainfall maxima in BMA percentile forecasts. BMA with stratified sampling according to ensemble spread overcomes the problem reasonably well, producing sharper predictive probability density functions and BMA percentile forecasts, and more reliable PQPF than the conventional BMA approach. The continuous ranked probability scores, Brier skill scores, and reliability diagrams of the two BMA experiments were examined for all available forecast days, along with a logistic regression experiment. Stratified sampling BMA outperformed the raw ensemble and conventional BMA in all verifications, and also showed better skill than logistic regression in low-threshold forecasts.

Characterization of aquifer heterogeneity is inherently difficult because of the insufficiency of data, the inflexibility of parameterization methods, and non-uniqueness of parameterization methods. Groundwater predictions are greatly affected by multiple interpretations of aquifer properties and the uncertainties of model parameters. This study introduces a Bayesian model averaging (BMA) method along with multiple generalized parameterization (GP) methods to identify hydraulic conductivity and along with multiple simulation models to predict groundwater head and quantify the prediction uncertainty. Two major issues about BMA are discussed. The first problem is with using Occam’s window in usual BMA applications. Occam’s window only accepts models in a very narrow range, tending to single out the best method and discard other good methods. A variance window is proposed to replace Occam’s window to cope with this problem. The second problem is with using the Kashyap information criterion (KIC) in the approximation of posterior model probabilities, which tends to prefer highly uncertain model by considering the Fisher information matrix. The Bayesian information criterion (BIC) is recommended because it is able to avoid controversial results and it is computationally efficient. Numerical examples are designed to test the Bayesian model averaging method on hydraulic conductivity identification and groundwater head prediction. The proposed methodologies are then applied to the hydraulic conductivity identification of the Alamitos Gap area, and the hydraulic conductivity estimation and groundwater head prediction of the “1,500-foot” sand in East Baton Rouge Parish, Louisiana. The results show that the GP method provides great flexibility in parameterization with small conditional variance. The use of the variance window is necessary to avoid a dominant model when many models perform equally well. Compared to KIC, BIC is able to give an unbiased posterior model probability. It is also concluded that the uncertainty increases by including multiple models under the BMA framework, but risks are reduced by avoiding overconfidence in the solution from one model.

The main approach to probabilistic weather forecasting has been the use of ensemble forecasting. In ensemble forecasting, the probability information is generally derived by using several numerical model runs, with perturbation of the initial conditions, physical schemes or dynamic core of the numerical weather prediction (NWP) models. However, ensemble forecasting usually tends to be under‐dispersive. Statistical post‐processing has, therefore, become an essential component of any ensemble prediction system aiming to improve the quality of numerical weather forecasts as they seek to generate calibrated and sharp predictive distributions of future weather quantities. Different versions of the ensemble model output statistics (EMOS) and the Bayesian model averaging (BMA) post‐processing methods are used in the present paper to calibrate 24, 48 and 72 hr forecasts of 24 hr accumulative precipitation. The ensemble employs the weather and research forecasting (WRF) model with eight different configurations which were run over Iran for six months (September 2015–February 2016). The results reveal that the BMA and EMOS‐censored, shifted gamma (CSG) techniques are substantially successful at improving the raw WRF ensemble forecasts; however, each approach improves different aspects of the forecast quality. The BMA method is more accurate, skilful and reliable than the EMOS‐CSG method, but has poorer discrimination. Moreover, it has better resolution in predicting the probability of high‐precipitation events than the EMOS‐CSG method.

Flood risk analysis of reservoirs based on full-series ARIMA model under climate change

Tracking shallow and deep groundwater storage changes in North China Plain with improved fusion method and hybrid spectral analysis approach

The Yalong River basin (YRB), originating from the Tibetan Plateau, is critical for China's western development and electricity transmission. Climate change would affect the streamflow and potentially impact the hydropower in the YRB. In this study, the Automated Statistical Downscaling technique and the hydrological model SWAT were used to reproduce the observed precipitation, temperature and streamflow and predict them in the future. More importantly, the Bayesian Model Averaging (BMA) method was used to appraise the performance of four different Global Climate Models (GCMs) in simulating historical streamflow (1984–1997), and to quantify the uncertainties in future projections (2021–2100) under three Representative Concentration Pathways (RCPs) in the YRB. It was found that (i) the SWAT hydrological model has good applicability in the YRB; (ii) the BMA method can capture the historical streamflow reasonably better than individual GCM projections, providing a scientific reference for the future water resources planning and utilization in the YRB.

Probabilistic forecasts of wind vectors are becoming critical as interest grows in wind as a clean and renewable source of energy, in addition to a wide range of other uses, from aviation to recreational boating. Unlike other common forecasting problems, which deal with univariate quantities, statistical approaches to wind vector forecasting must be based on bivariate distributions. The prevailing paradigm in weather forecasting is to issue deterministic forecasts based on numerical weather prediction models. Uncertainty can then be assessed through ensemble forecasts, where multiple estimates of the current state of the atmosphere are used to generate a collection of deterministic predictions. Ensemble forecasts are often uncalibrated, however, and Bayesian model averaging (BMA) is a statistical way of postprocessing these forecast ensembles to create calibrated predictive probability density functions (PDFs). It represents the predictive PDF as a weighted average of PDFs centered on the individual bias-corrected forecasts, where the weights reflect the forecasts’ relative contributions to predictive skill over a training period. In this paper the authors extend the BMA methodology to use bivariate distributions, enabling them to provide probabilistic forecasts of wind vectors. The BMA method is applied to 48-h-ahead forecasts of wind vectors over the North American Pacific Northwest in 2003 using the University of Washington mesoscale ensemble and is shown to provide better-calibrated probabilistic forecasts than the raw ensemble, which are also sharper than probabilistic forecasts derived from climatology.

Precipitation is an important and difficult climate variable to predict. Skillful sub-seasonal precipitation forecast can provide useful information for agriculture and water resources management communities. Nevertheless, sub-seasonal forecasts have been given less attention compared with forecasts of shorter/longer time horizons. Recently, the S2S database has made sub-seasonal to seasonal forecasts/reforecasts from 11 operational centers available to researchers. In this work, reforecasts of the European Centre for Medium-Range Weather Forecasts (ECMWF) spanning over a 20-year period were evaluated. Raw and post-processed precipitation forecasts were put against observed precipitation series of a number of synoptic stations with different precipitation regimes for all months of the year. By comparison, precipitation forecasts were more skillful in wet months. Raw forecasts in December and February were better than that in other months, while the weakest results were detected in August. There was no significant relationship between precipitation regime and prediction skill. Furthermore, quantile mapping (QM), Bayesian model averaging (BMA), and QM_BMA combination were adopted for post-processing. BMA outperformed QM and QM_BMA through improving the correlation coefficient between observations and average of ensemble forecasts; however, BMA performed weaker in other evaluation criteria, in particular in humid regions. It is concluded that the application of post-processing techniques greatly improved the results of ensemble precipitation forecasts. However, in a number of stations/months, the forecast results were not acceptable even after post-processing.

Ensembles used for probabilistic weather forecasting often exhibit a spread-error correlation, but they tend to be underdispersive. This paper proposes a statistical method for postprocessing ensembles based on Bayesian model averaging (BMA), which is a standard method for combining predictive distributions from different sources. The BMA predictive probability density function (PDF) of any quantity of interest is a weighted average of PDFs centered on the individual bias-corrected forecasts, where the weights are equal to posterior probabilities of the models generating the forecasts and reflect the models' relative contributions to predictive skill over the training period. The BMA weights can be used to assess the usefulness of ensemble members, and this can be used as a basis for selecting ensemble members; this can be useful given the cost of running large ensembles. The BMA PDF can be represented as an unweighted ensemble of any desired size, by simulating from the BMA predictive distribution. The BMA predictive variance can be decomposed into two components, one corresponding to the between-forecast variability, and the second to the within-forecast variability. Predictive PDFs or intervals based solely on the ensemble spread incorporate the first component but not the second. Thus BMA provides a theoretical explanation of the tendency of ensembles to exhibit a spread-error correlation but yet be underdispersive. The method was applied to 48-h forecasts of surface temperature in the Pacific Northwest in January–June 2000 using the University of Washington fifth-generation Pennsylvania State University–NCAR Mesoscale Model (MM5) ensemble. The predictive PDFs were much better calibrated than the raw ensemble, and the BMA forecasts were sharp in that 90% BMA prediction intervals were 66% shorter on average than those produced by sample climatology. As a by-product, BMA yields a deterministic point forecast, and this had root-mean-square errors 7% lower than the best of the ensemble members and 8% lower than the ensemble mean. Similar results were obtained for forecasts of sea level pressure. Simulation experiments show that BMA performs reasonably well when the underlying ensemble is calibrated, or even overdispersed.