Sensitivity and Uncertainty

We dealt with a rather frequent and difficult situation while modelling extreme floods, namely, model output uncertainty in data sparse regions. A historical extreme flood event was chosen to illustrate the challenges involved. Our aim was to understand what the causes might have been and specifically to show how input and model parameter uncertainties affect the output. For this purpose, a conceptual model was calibrated and validated with recent data rich time period. Resulting model parameters were used to model the historical event which subsequently resulted in a rather poor hydrograph. Due to the bad model performance, a spatial simulation technique was used to invert the model for precipitation. Constraints, such as taking the precipitation values at historical observation locations in to account, with correct spatial structures and following the observed regional distributions were used to generate realistic precipitation fields. Results showed that the inverted precipitation improved the performance significantly even when using many different model parameters. We conclude that while modelling in data sparse conditions both model input and parameter uncertainties have to be dealt with simultaneously to obtain meaningful results.

Integrated Bayesian Multi-model approach to quantify input, parameter and conceptual model structure uncertainty in groundwater modeling

Joint analysis of input and parametric uncertainties in watershed water quality modeling: A formal Bayesian approach

Utilizing degraded quality waters such as saline water as irrigation water with proper management methods such as leaching application is a potential answer to water scarcity in agricultural systems. Leaching application requires understanding the relationship between the amount of irrigation water and its quality with the dynamic of salts in the soil. The HYDRUS-1D model can simulate the dynamic of soil salinity under saline water irrigation conditions. However, these simulations are subject to uncertainty. A study was conducted to assess the uncertainty of the HYDRUS-1D model parameters and outputs to simulate the dynamic of salts under saline water irrigation conditions using the Markov Chain Monte Carlo (MCMC) based Metropolis-Hastings algorithm in the R-Studio environment. Results indicated a low level of uncertainty in parameters related to the advection term (water movement simulation) and water stress reduction function for root water uptake in the solute transport process. However, a higher level of uncertainty was detected for dispersivity and diffusivity parameters, possibly because of the study’s scale or some error in initial or boundary conditions. The model output (predictive) uncertainty showed a high uncertainty in dry periods compared to wet periods (under irrigation or rainfall). The uncertainty in model parameters was the primary source of total uncertainty in model predictions. The implementation of the Metropolis-Hastings algorithm for the HYDRUS-1D was able to conveniently estimate the residual water content (θr) value for the water simulation processes. The model’s performance in simulating soil water content and soil water electrical conductivity (ECsw) was good when tested with the 50% quantile of the posterior distribution of the parameters. Uncertainty assessment in this study revealed the effectiveness of the Metropolis-Hastings algorithm in exploring uncertainty aspects of the HYDRUS-1D model for reproducing soil salinity dynamics under saline water irrigation at a field scale.

Propagation of uncertainty in outputs of a standard integrated model of transportation and land use has been examined. Austin-calibrated DRAM-EMPAL predictions of residence and work locations were used as inputs to a UTPP-type four-step travel demand model, and the resulting travel times were fed forward into the future period’s land use models. Covariance in inputs (including model parameters and demographic variables) was accommodated through multivariate Monte Carlo sampling of 200 scenarios. Variances in land use and travel predictions were analyzed over time and as a function of input values. Results indicate that output variations were most sensitive to the exponent of the link performance function, the split of trips between peak and off-peak, and several trip generation and attraction rates. Twenty years in the future, final uncertainty levels (as measured by coefficients of variation) solely due to input and parameter estimation errors are on the order of 38% for total regional peak-period vehicle miles traveled (VMT), 45% for peak-period flows, and 50% and 37% for residential and employment densities, respectively. This means central point estimates of key model outputs are very likely (>30%) to fall 38% to 50% below or above the mean value. In the Austin example, 15% of the region’s 200 simulated peak-period VMT estimates fell below 3.7 million mi per day and 15% exceeded 8.4 million mi. Such substantial variation is solely due to standard model parameter and input uncertainties. Other uncertainty about the future and about human behavior exists and will add more variation.

The remaining useful life (RUL) estimation is an important metric that helps in condition-based maintenance. Damage data obtained from the diagnostics techniques are often noisy, and the RUL estimated from the data is less reliable. Estimating the probabilistic RUL by quantifying the uncertainty in the predictive model parameters using the noisy data increases confidence in the predicted values. In this work, Bayesian inference is used to inversely quantify the uncertainty in the model parameters of the predictive damage model in adhesively bonded joints, and quantified parameter uncertainty is represented by a probability distribution function (PDF). It is difficult to obtain a closed-form solution for the RUL from the PDF of model parameters due to the nonlinearity of the damage models. Instead, statistical simulations are used to generate the model parameter samples from the PDF, and the RUL is estimated from the samples. In the statistical simulations, the predictive damage model is evaluated for each parameter sample. Using a physics-based model as the predictive model increases computational time, which makes the statistical techniques to be computationally expensive. It is essential to reduce the computational time to estimate the RUL in a feasible time. In this work, real-time probabilistic RUL estimation is demonstrated in adhesively bonded joints using the Sequential Monte Carlo (SMC) sampling method and cloud-based computations. The SMC sampling method is an alternative to traditional Markov Chain Monte Carlo (MCMC) methods, which enables generating statistical parameter samples in parallel. The parallel computational capabilities of the SMC methods are exploited by running the SMC simulation on multiple cloud calls. This approach is demonstrated by estimating fatigue RUL in the adhesively bonded joint. The accuracy of probabilistic RUL estimated by SMC is validated by comparing it with RUL estimated by the MCMC and the experimental values. The SMC simulation is run on the cloud and the computational speedup of the SMC is demonstrated.

Industrial ion-exchange chromatography development using discontinuous Galerkin methods coupled with forward sensitivity analysis

For decision-makers considering new medicines for reimbursement and public use, both value for money and affordability are important considerations. Whereas a cost-effectiveness model provides information about value for money, a budget impact assessment (BIA) is customized to a specific context and estimates the total investment needed; one part of affordability. Both analytic approaches have parameter uncertainty within them, yet comparatively little attention is given to parameter uncertainty in BIA. Currently, within BIA, uncertainty exploration is limited to point estimates for plausible scenarios, prompting the question: can a decision-maker be confident in point estimates? Within this paper, our intent is to revitalize the discussion of uncertainty in BIA. In the context of health technology assessments submitted to support reimbursement decision-making, we propose reliance on probabilistic sensitivity analysis conducted in the cost-effectiveness model. If assumptions made in a cost-effectiveness model are valid, probabilistic cost estimates from the model, with the same perspective adopted as the BIA, should also inform BIA. Mean and variance of population outcomes, given parameter uncertainty in model inputs, are estimable from model outputs. As sufficiently large random samples are drawn from a population, the distribution of sample means will follow an approximately normal distribution. Therefore, when drawing samples from the model to inform estimates of budget impact, the assumption of an approximately normal distribution for costs is reasonable. We propose that the variance in mean costs from the cost-effectiveness model also reflects the variance in budget impact estimates and should be used to estimate budget impact confidence intervals.

This paper aims to investigate the effect of uncertainty originating from model inputs, parameters and initial conditions on 10 day ensemble low flow forecasts. Two hydrological models, GR4J and HBV, are applied to the Moselle River and performance in the calibration, validation and forecast periods, and the effect of different uncertainty sources on the quality of low flow forecasts are compared. The forecasts are generated by using meteorological ensemble forecasts as input to GR4J and HBV. The ensembles provided the uncertainty range for the model inputs. The Generalized Likelihood Uncertainty Estimation (GLUE) approach is used to estimate parameter uncertainty. The quality of the probabilistic low flow forecasts has been assessed by the relative confidence interval, reliability and hit/false alarm rates. The daily observed low flows are mostly captured by the 90% confidence interval for both models. However, GR4J usually overestimates low flows whereas HBV is prone to underestimate them, particularly when the parameter uncertainty is included in the forecasts. The total uncertainty in GR4J outputs is higher than in HBV. The forecasts issued by HBV incorporating input uncertainty resulted in the most reliable forecast distribution. The parameter uncertainty was the main reason reducing the number of hits. The number of false alarms in GR4J is twice the number of false alarms in HBV when considering all uncertainty sources. The results of this study showed that the parameter uncertainty has the largest effect whereas the input uncertainty had the smallest effect on the medium range low flow forecasts.

Numerical simulations of metal alloy solidification are used to gain insight into physical phenomena that cannot be observed experimentally. These models produce results that are used to draw conclusions about a process or alloy and often compared to experimental results. However, uncertainty in model inputs cause uncertainty in model results, which have the potential to significantly affect conclusions drawn from their predictions. As a step toward understanding the effect of uncertain inputs on solidification modeling, uncertainty quantification (UQ) and sensitivity analysis are performed on a transient model of solidification of Al–4.5 wt % Cu in a rectangular cavity. The binary alloy considered has columnar solidification morphology, and this model solves equations for momentum, temperature, and species conservation. UQ and sensitivity analysis are performed for the degree of macrosegregation and solidification time. A Smolyak sparse grid algorithm is used to select input values to construct a polynomial response surface fit to model outputs. This polynomial is then used as a surrogate for the complete solidification model to determine the sensitivities and probability density functions (PDFs) of the model outputs. Uncertain model inputs of interest include the secondary dendrite arm spacing (SDAS), heat transfer coefficient, and material properties. The most influential input parameter for predicting the macrosegregation level is the dendrite arm spacing, which also strongly depends on the choice of permeability model. Additionally, the degree of uncertainty required to produce accurate predictions depends on the outputs of interest from the model.

The global animal food chain has a large contribution to the global anthropogenic greenhouse gas (GHG) emissions, but its share and sources vary highly across the world. However, the assessment of GHG emissions from livestock production is subject to various uncertainties, which have not yet been well quantified at large spatial scale. We assessed the uncertainties in the relations between animal production (milk, meat, egg) and the CO2, CH4, and N2O emissions in Africa, Latin America and the European Union, using the MITERRA-Global model. The uncertainties in model inputs were derived from time series of statistical data, literature review or expert knowledge. These model inputs and parameters were further divided into nine groups based on type of data and affected greenhouse gas. The final model output uncertainty and the uncertainty contribution of each group of model inputs to the uncertainty were quantified using a Monte Carlo approach, taking into account their spatial and cross-correlation. GHG emissions and their uncertainties were determined per livestock sector, per product and per emission source category. Results show large variation in the GHG emissions and their uncertainties for different continents, livestock sectors products or source categories. The uncertainty of total GHG emissions from livestock sectors is higher in Africa and Latin America than in the European Union. The uncertainty of CH4 emission is lower than that for N2O and CO2. Livestock parameters, CH4 emission factors and N emission factors contribute most to the uncertainty in the total model output. The reliability of GHG emissions from livestock sectors is relatively high (low uncertainty) at continental level, but could be lower at country level.

A Bayesian hierarchical framework for calibrating aquatic biogeochemical models

The traditional treatment of uncertainty in hydrological modelling primarily attributes it to model parameters, but rarely systematically considers meteorological input errors, especially in quantifying the impact of meteorological input errors on parameter uncertainty. This study developed a Bayesian‐based integrated approach to quantitatively investigate uncertainties in meteorological inputs (precipitation and temperature) and model parameters as well as the variation in parameter uncertainty due to meteorological input errors. Additionally, we analysed the propagation from these uncertainties to runoff response in snowmelt and non‐snowmelt periods. The applicability and advantages of this approach were presented by applying of the Soil and Water Assessment Tool to the Shitoukoumen Reservoir Catchment. Differential Evolution Adaptive Metropolis‐Markov Chain Monte Carlo was applied for the straightforward Bayesian inference the uncertainties of meteorological inputs and model parameters. On this basis, multilevel factorial analysis technology was used to quantitatively investigate the specific impact of the model parameters' individual and interactive effects due to meteorological input errors. Finally, the impact of meteorological input errors and model parameter uncertainty on the model performance were analysed and quantified systematically. The results showed that the meteorological input errors could affect the random characteristics of multiple model parameters. Moreover, meteorological input errors could further affect the model parameters' effects on annual average runoff. Overall, the above results have significant implications in enhancing hydrological model to simulate/predict runoff and understanding hydrological processes during different periods.

Recent efforts to ensure the reliability of computational model-based predictions in healthcare, such as the ASME V&V40 Standard, emphasize the importance of uncertainty quantification (UQ) and sensitivity analysis (SA) when evaluating computational models. UQ involves empirically determining the uncertainty in model inputs—typically resulting from natural variability or measurement error—and then calculating the resultant uncertainty in model outputs. SA involves calculating how uncertainty in model outputs can be apportioned to input uncertainty. Rigorous comprehensive UQ/SA provides confidence that model-based decisions are robust to underlying uncertainties. However, comprehensive UQ/SA is not currently feasible for whole heart models, due to numerous factors including model complexity and difficulty in measuring variability in the many parameters. Here, we present a significant step to developing a framework to overcome these limitations. We: (i) developed a novel action potential (AP) model of moderate complexity (six currents, seven variables, 36 parameters); (ii) prescribed input variability for all parameters (not empirically derived); (iii) used a single “hyper-parameter” to study increasing levels of parameter uncertainty; (iv) performed UQ and SA for a range of model-derived quantities with physiological relevance; and (v) present quantitative and qualitative ways to analyze different behaviors that occur under parameter uncertainty, including “model failure”. This is the first time uncertainty in every parameter (including conductances, steady-state parameters, and time constant parameters) of every ionic current in a cardiac model has been studied. This approach allowed us to demonstrate that, for this model, the simulated AP is fully robust to low levels of parameter uncertainty — to our knowledge the first time this has been shown of any cardiac model. A range of dynamics was observed at larger parameter uncertainty (e.g., oscillatory dynamics); analysis revealed that five parameters were highly influential in these dynamics. Overall, we demonstrate feasibility of performing comprehensive UQ/SA for cardiac cell models and demonstrate how to assess robustness and overcome model failure when performing cardiac UQ analyses. The approach presented here represents an important and significant step toward the development of model-based clinical tools which are demonstrably robust to all underlying uncertainties and therefore more reliable in safety-critical decision-making.

Parameter uncertainty in hydrologic modeling is commonly evaluated, but assessing the impact of spatial input data uncertainty in spatially descriptive ‘distributed’ models is not common. This study compares the significance of uncertainty in spatial input data and model parameters on the output uncertainty of a distributed hydrology and sediment transport model, HYdrology Simulation using Time-ARea method (HYSTAR). The Shuffled Complex Evolution Metropolis (SCEM-UA) algorithm was used to quantify parameter uncertainty of the model. Errors in elevation and land cover layers were simulated using the Sequential Gaussian/Indicator Simulation (SGS/SIS) techniques and then incorporated into the model to evaluate their impact on the outputs relative to those of the parameter uncertainty. This study demonstrated that parameter uncertainty had a greater impact on model output than did errors in the spatial input data. In addition, errors in elevation data had a greater impact on model output than did errors in land cover data. Thus, for the HYSTAR distributed hydrologic model, accuracy and reliability can be improved more effectively by refining parameters rather than further improving the accuracy of spatial input data and by emphasizing the topographic data over the land cover data.