Uncertainty and sensitivity analysis: tools for GIS-based model implementation

This paper reviews five related types of analysis, namely (i) sensitivity or what-if analysis, (ii) uncertainty or risk analysis, (iii) screening, (iv) validation, and (v) optimization. The main questions are: when should which type of analysis be applied; which statistical techniques may then be used? This paper claims that the proper sequence to follow in the evaluation of simulation models is as follows. 1) Validation, in which the availability of data on the real system determines which type of statistical technique to use for validation. 2) Screening: in the simulation‘s pilot phase the really important inputs can be identified through a novel technique, called sequential bifurcation, which uses aggregation and sequential experimentation. 3) Sensitivity analysis: the really important inputs should be subjected to a more detailed analysis, which includes interactions between these inputs; relevant statistical techniques are design of experiments (DOE) and regression analysis. 4) Uncertainty analysis: the important environmental inputs may have values that are not precisely known, so the uncertainties of the model outputs that result from the uncertainties in these model inputs should be quantified; relevant techniques are the Monte Carlo method and Latin hypercube sampling. 5) Optimization: the policy variables should be controlled; a relevant technique is Response Surface Methodology (RSM), which combines DOE, regression analysis, and steepest-ascent hill-climbing. The recommended sequence implies that sensitivity analysis procede uncertainty analysis. Several case studies for each phase are briefly discussed in this paper.

A sensitivity analysis of a personalized pulse wave propagation model for arteriovenous fistula surgery. Part A: Identification of most influential model parameters

A method for evaluating sensitivity and uncertainty analyses is presented here for use with chemical environmental exposure models. Sensitivity analysis allows one to assess the relationship between a set of potentially important model input variables and the model output. Uncertainty analysis allows one to assess the set of output values obtained and estimate various statistical properties such as the mean, median, standard deviation, and range. Absence of the ability to quantify parameter value uncertainty and model output uncertainty limits the validity of predictive model use. Use of the methods presented in this paper can increase the confidence in predictive environmental chemical fate model use. The present investigation employed the Latin hypercube sampling method developed by Iman and Shortencarier at Sandia National Laboratories, Albuquerque, NM. It requires that a set of input variables be identified, and the probability distribution of each be specified. The range of each input variable is divided into intervals having equal probabilities, and one value is randomly selected from each interval. Sets of model input variables are then generated which cover the range of each variable. Some model inputs may then be calculated as functions of these key input variables. The computer model is run for each set of input variables and the model output is obtained. Regression techniques can then be employed to examine the differential sensitivity of the model to the various input variables and functions thereof, for example, cross products of input variables. Since the model input values are chosen based on probabilities, the set of model outputs can also be interpreted on a probabilistic basis. The techniques discussed in this paper respond to the growing need by chemical exposure model users for better defined sensitivity analysis procedures which can aid in explaining and interpreting results. An example problem data set is presented which highlights the methodological techniques and their utility.

Sensitivity analysis in the context of uncertainty analysis for computationally intensive models

Recent efforts to ensure the reliability of computational model-based predictions in healthcare, such as the ASME V&V40 Standard, emphasize the importance of uncertainty quantification (UQ) and sensitivity analysis (SA) when evaluating computational models. UQ involves empirically determining the uncertainty in model inputs—typically resulting from natural variability or measurement error—and then calculating the resultant uncertainty in model outputs. SA involves calculating how uncertainty in model outputs can be apportioned to input uncertainty. Rigorous comprehensive UQ/SA provides confidence that model-based decisions are robust to underlying uncertainties. However, comprehensive UQ/SA is not currently feasible for whole heart models, due to numerous factors including model complexity and difficulty in measuring variability in the many parameters. Here, we present a significant step to developing a framework to overcome these limitations. We: (i) developed a novel action potential (AP) model of moderate complexity (six currents, seven variables, 36 parameters); (ii) prescribed input variability for all parameters (not empirically derived); (iii) used a single “hyper-parameter” to study increasing levels of parameter uncertainty; (iv) performed UQ and SA for a range of model-derived quantities with physiological relevance; and (v) present quantitative and qualitative ways to analyze different behaviors that occur under parameter uncertainty, including “model failure”. This is the first time uncertainty in every parameter (including conductances, steady-state parameters, and time constant parameters) of every ionic current in a cardiac model has been studied. This approach allowed us to demonstrate that, for this model, the simulated AP is fully robust to low levels of parameter uncertainty — to our knowledge the first time this has been shown of any cardiac model. A range of dynamics was observed at larger parameter uncertainty (e.g., oscillatory dynamics); analysis revealed that five parameters were highly influential in these dynamics. Overall, we demonstrate feasibility of performing comprehensive UQ/SA for cardiac cell models and demonstrate how to assess robustness and overcome model failure when performing cardiac UQ analyses. The approach presented here represents an important and significant step toward the development of model-based clinical tools which are demonstrably robust to all underlying uncertainties and therefore more reliable in safety-critical decision-making.

Predictive Modeling Methodology constitutes an innovative approach to perform uncertainty analysis (UA) that reduces the subjective and user-defined ways to manage experimental data and derive uncertainty of input parameters that characterize the Propagation of Input Uncertainties (PIU) and/or Propagation of Output Accuracies (POA) methods. The Code with the capability of Adjoint Sensitivity and Uncertainty AnaLysis by Internal Data ADjustment and assimilation (CASUALIDAD) method can be developed as a fully deterministic method based on advanced mathematical tools to internally perform in the thermal-hydraulic system code the sensitivity analysis (SA) and the UA. The method is based upon powerful mathematical tools to perform the SA and upon the Data Adjustment and Assimilation methodology by which experimental observations are combined with code predictions and their respective errors through the application of the Bayes theorem and of the Principle of Maximum Likelihood to provide an improved estimate of the system state and of the associated uncertainty considering all input parameters that affect any prediction. The methodology has been structured in two main steps. The first step generates the database of improved estimations (IEs) starting from the available set of experimental data and related qualified calculations. The second step deals with the use of the selected (from the obtained database) set of IEs for the uncertainty evaluation of the predicted nuclear power plant transient scenario. The proposed methodology clearly interrelates in a consistent and robust framework the code validation issue with the evaluation of the uncertainty of code responses passing through the quantification of input uncertainty parameters of code models, thus constituting a step forward with respect to the subjectivity of the current methods based on PIU and/or POA.

Propagation of uncertainty in outputs of a standard integrated model of transportation and land use has been examined. Austin-calibrated DRAM-EMPAL predictions of residence and work locations were used as inputs to a UTPP-type four-step travel demand model, and the resulting travel times were fed forward into the future period’s land use models. Covariance in inputs (including model parameters and demographic variables) was accommodated through multivariate Monte Carlo sampling of 200 scenarios. Variances in land use and travel predictions were analyzed over time and as a function of input values. Results indicate that output variations were most sensitive to the exponent of the link performance function, the split of trips between peak and off-peak, and several trip generation and attraction rates. Twenty years in the future, final uncertainty levels (as measured by coefficients of variation) solely due to input and parameter estimation errors are on the order of 38% for total regional peak-period vehicle miles traveled (VMT), 45% for peak-period flows, and 50% and 37% for residential and employment densities, respectively. This means central point estimates of key model outputs are very likely (>30%) to fall 38% to 50% below or above the mean value. In the Austin example, 15% of the region’s 200 simulated peak-period VMT estimates fell below 3.7 million mi per day and 15% exceeded 8.4 million mi. Such substantial variation is solely due to standard model parameter and input uncertainties. Other uncertainty about the future and about human behavior exists and will add more variation.

Simulation models are used to estimate, forecast, optimize and identify limiting factors and analyze changes in crop production. In order to obtain a functional and reliable mathematical model, it is necessary to know the source of uncertainty and identify the most influential parameters. This study aimed to carry out an uncertainty analysis (UA) and a global spatiotemporal sensitivity analysis (SA) for the parameters of the SIMPLE model, which uses 13 parameters, has two state variables and uses daily weather data to simulate crop growth and development. A Monte Carlo simulation was performed for the UA, and Sobol’s method was used for the SA. Four automatic weather stations representing the climatic conditions of the different bean-producing areas in Zacatecas, Mexico, and a four-year historical series of each station for irrigated and rainfed common bean crops were analyzed. From the UA the coefficients of variation (CV) for thermal time were 11.49% and 11.47%, for biomass the CV were 47.94% and 37.80% and for yield the CV were 49.52% and 39.70% for irrigated and rainfed beans, respectively. From the SA, the most influential parameters for irrigated beans were Tsum > Swater > Tbase > I50A > Topt and for rainfed beans, Tsum > Tbase > I50A > Topt > Swater, according to indices calculated on biomass and thermal time. In conclusion, UA was able to accurately quantify the uncertainty of the biomass, and SA allowed the identification of the most influential of the parameters of the SIMPLE model applied to a common bean crop.

Sensitivity analysis for transit equilibrium assignment and applications to uncertainty analysis

A probabilistic model (SHEDS-Wood) was developed to examine children's exposure and dose to chromated copper arsenate (CCA)-treated wood, as described in Part 1 of this two-part article. This Part 2 article discusses sensitivity and uncertainty analyses conducted to assess the key model inputs and areas of needed research for children's exposure to CCA-treated playsets and decks. The following types of analyses were conducted: (1) sensitivity analyses using a percentile scaling approach and multiple stepwise regression; and (2) uncertainty analyses using the bootstrap and two-stage Monte Carlo techniques. The five most important variables, based on both sensitivity and uncertainty analyses, were: wood surface residue-to-skin transfer efficiency; wood surface residue levels; fraction of hand surface area mouthed per mouthing event; average fraction of nonresidential outdoor time a child plays on/around CCA-treated public playsets; and frequency of hand washing. In general, there was a factor of 8 for the 5th and 95th percentiles and a factor of 4 for the 50th percentile in the uncertainty of predicted population dose estimates due to parameter uncertainty. Data were available for most of the key model inputs identified with sensitivity and uncertainty analyses; however, there were few or no data for some key inputs. To evaluate and improve the accuracy of model results, future measurement studies should obtain longitudinal time-activity diary information on children, spatial and temporal measurements of residue and soil concentrations on or near CCA-treated playsets and decks, and key exposure factors. Future studies should also address other sources of uncertainty in addition to parameter uncertainty, such as scenario and model uncertainty.

Uncertainty and sensitivity analyses as a validation tool for BWR bundle thermal–hydraulic predictions

Estimating the functional relation between the probabilistic response of a computational model and the distribution parameters of the model inputs is especially useful for 1) assessing the contribution of the distribution parameters of model inputs to the uncertainty of model output (parametric global sensitivity analysis), and 2) identifying the optimized distribution parameters of model inputs to efficiently and cheaply reduce the uncertainty of model output (parametric optimization). In this paper, the extended Monte Carlo simulation method is developed for this purpose, which provides four benefits to the parametric global sensitivity analysis and parametric optimization problems. First, the extended Monte Carlo simulation method is able to provide an unbiased or progressive unbiased estimate for the model whose behavior is even mainly governed by high nonlinearity or interaction terms. Second, only one set of model input–output samples is needed for implementing the method; thus, the computational burden is free of input dimensionality. Third, the extended Monte Carlo simulation is a derivative-free method. Fourth, the extended Monte Carlo simulation method enables us to solve problems with dependent and non-normally distributed model inputs. Additionally, the R-indices are introduced for conquering the overparameterized problem in the optimization process. An analytical example and two engineering examples are used to demonstrate the power of the proposed methods.

Computer models are widely used to help make decisions about real-world systems. As computer models of large and complex systems can have long run-times and high-dimensional input spaces, it is often necessary to use emulation to assess uncertainties in computer model output. This paper presents methodology for emulation of complex computer models motivated by a real-world example in energy policy. The computer model studied is an economic model of investment in electricity generation in Great Britain. The computer model was used to select parameters in a government policy designed to incentivize investment in renewable technologies to meet government targets. Limited computing time meant that few runs of the computer model were available to fit an emulator. The statistical methodology developed was therefore focused on accurately capturing the uncertainty in computer model output arising from the small number of available model runs. A varying coefficient emulator is proposed to model uncertainty in model output when extrapolating away from model runs. To maximize use of the small number of runs available, this varying coefficient emulator is paired with a criterion-based procedure for design selection.

Watershed-scale water quality (WWQ) models are now widely used to support management decision-making. However, significant uncertainty in the model outputs remains a largely unaddressed issue. In recent years, Markov Chain Monte Carlo (MCMC), a category of formal Bayesian approaches for uncertainty analysis (UA), has become popular in the field of hydrological modeling, but its applications to WWQ modeling have been rare. This study systematically evaluated the applicability of MCMC in assessing the uncertainty of WWQ modeling, using Differential Evolution Adaptive Metropolis (DREAM(ZS)) and SWAT as the representative MCMC algorithm and WWQ model, respectively. The nitrate pollution in Newport Bay watershed was the case study for numerical experiments. It has been concluded that the efficiency and effectiveness of a MCMC algorithm would depend on some critical designs of the UA, including: (i) how many and which model parameters to be considered as random in the MCMC analysis; (ii) where to fix the non-random model parameters; and (iii) which criteria to stop the Markov Chain. The study results also indicate that the MCMC UA has to be management-oriented, that is, management objectives should be factored into the designs of the UA, rather than be considered after the UA.

Recent literature on building energy performance simulation leans toward implementing uncertainty analysis (UA), instead of deterministic solutions, to handle ever-existing and pivotal uncertainties in building design decision-making process. Variations in weather temperature, degradation of building envelope material properties over time, and random behavior of occupants, among all, are the key sources of uncertainty in building energy consumption predictions. The UA couples to the sensitivity analysis (SA) to identify the most influential inputs on the uncertainties of the building energy consumption. This paper describes a newly-developed UA and SA predictive tool for building energy performance simulations. Energy performance simulations are based on a resistance-capacitance thermal model for the building. For a hypothetical residential building in College Station, Texas, USA, the present work describes and compares predicted probability distribution and sensitivity indexes produced by the UA-SA tool using a transient (dynamic) response analysis (TRA) and static response analysis (SRA). For brevity, the analysis considers uncertainty only for the exterior walls’ parameters including thickness, thermal conductivity, heat transfer coefficient, density, and heat capacity; i.e., a five-dimensional problem is solved. Compared to the TRA, predictions from the SRA underestimate the annual energy consumption up to 30%; however, SRA is significantly faster. Nonetheless, sensitivity indexes from the SRA and TRA closely match.