Input model uncertainty: why do we care and what should we do about it?

Sensitivity and Uncertainty

Integrated Bayesian Multi-model approach to quantify input, parameter and conceptual model structure uncertainty in groundwater modeling

10525 Background: To evaluate the incremental short- and long-term benefits and harms of primary prevention with risk reducing medication in high-risk women receiving screening mammography. Methods: We adapted an established, validated Cancer Intervention and Surveillance Modeling Network (CISNET) breast cancer discrete event microsimulation model developed to synthesize data the impact of using risk-reducing medication and annual mammography among women with a 3% or higher five-year risk of developing breast cancer. We also examined the effects of supplemental MRI. The model follows a simulated cohort of millions of US women from birth to death. We used large observational and clinical trial data to derive input parameters for cohort-specific birth rates, breast cancer risk, incidence and stage, screening performance, survival by age, stage, and subtype, treatment efficacy, and other cause mortality. Breast cancer risk was modeled based on family history, breast density, age and history of past breast biopsy. We compared two strategies, annual 3D mammography alone vs. annual 3D mammography and a 5-year course of risk reducing medication at various starting ages, and adding MRI to each approach. Outcomes included the benefits of risk-reducing drugs (avoiding breast cancer) and screening (stage, breast cancer death). Harms included drug side effects and screening false positives and overdiagnosis. Sensitivity analysis tested the impact of uncertainty in model inputs and assumptions on results. Results: Compared to mammography alone, adding risk reducing medication could decrease invasive breast cancer incidence by 30%, and breast cancer deaths by 30% (Table). However, due to reduction in breast cancer incidence, risk reducing medication could result in a 3% increase in false positive results; adding MRI increases benefits but also increases false-positive results. Benefits and harms of risk reducing medication and breast cancer screening strategies for women at high-risk of developing breast cancer. Conclusions: Risk-reducing mediation reduces the risk of hormone-receptor positive breast cancer, and combining this with mammography (and/or MRI) improves earlier detection. Additional work is ongoing to incorporate adverse effects of therapy. Simulation modeling can be used to provide individualized data to facilitate discussions about breast cancer prevention and early detection among high-risk women seen in clinical practice.[Table: see text]

Uncertainty in hydrological modelling is often associated with model structure or calibration coefficients, and little consideration is given to the visualisation of uncertainty in either input or output data. As hydrological modelling is often a component of landscape design, there is continuing interest in the visualisation of uncertainty in model inputs and outputs, which could include catchment size or landcover (spatial inputs), and the subsequent impact on uncertainty in model predictions (such as streamflow). In this paper, inputs and outputs associated with hydrological models are discussed in the context of spatial uncertainty and how this information may be presented to users by means of statistical and spatial visualisation techniques. A regional study in hydrological modelling is used to demonstrate the sensitivity to spatial uncertainty and its consequences. Tools and techniques are discussed for the visual presentation of uncertainty, with implications for land management and policy decisions. A description is given of an experimental software tool developed for visualisation of spatial uncertainty in hydrological landscape cross-sections, where depth-to-water-table is used as a predictor for dryland salinity. Uncertainty in the estimate of this depth is important to analysts and policy makers.

Joint analysis of input and parametric uncertainties in watershed water quality modeling: A formal Bayesian approach

A review of uncertainty quantification techniques is provided for a variety of situations involving uncertainties in model inputs (independent variables). The situations of interest are divided into three categories: (i) when model prediction uncertainties are quantified based on uncertainties in uncertain inputs, (ii) when parameter estimate uncertainties are calculated by propagation of uncertainties from measured inputs and outputs, and (iii) when model prediction uncertainties are quantified based on corresponding uncertainties in measured inputs and uncertain parameter estimates. For all three situations, linearization‐based and Monte Carlo‐based techniques are reviewed and details for their corresponding algorithms are presented. Recommendations are provided on which uncertainty quantification techniques are best for different types of chemical engineering models based on the amount of input uncertainty and nonlinearity over the range of plausible input and parameter values.

Effects of error in model input: experiments with a forest patch model

The global animal food chain has a large contribution to the global anthropogenic greenhouse gas (GHG) emissions, but its share and sources vary highly across the world. However, the assessment of GHG emissions from livestock production is subject to various uncertainties, which have not yet been well quantified at large spatial scale. We assessed the uncertainties in the relations between animal production (milk, meat, egg) and the CO2, CH4, and N2O emissions in Africa, Latin America and the European Union, using the MITERRA-Global model. The uncertainties in model inputs were derived from time series of statistical data, literature review or expert knowledge. These model inputs and parameters were further divided into nine groups based on type of data and affected greenhouse gas. The final model output uncertainty and the uncertainty contribution of each group of model inputs to the uncertainty were quantified using a Monte Carlo approach, taking into account their spatial and cross-correlation. GHG emissions and their uncertainties were determined per livestock sector, per product and per emission source category. Results show large variation in the GHG emissions and their uncertainties for different continents, livestock sectors products or source categories. The uncertainty of total GHG emissions from livestock sectors is higher in Africa and Latin America than in the European Union. The uncertainty of CH4 emission is lower than that for N2O and CO2. Livestock parameters, CH4 emission factors and N emission factors contribute most to the uncertainty in the total model output. The reliability of GHG emissions from livestock sectors is relatively high (low uncertainty) at continental level, but could be lower at country level.

Technoeconomic feasibility of a sustainable charcoal industry to reduce deforestation in Haiti

Uncertainty propagation analysis of an N 2O emission model at the plot and landscape scale

The effects of parametric uncertainties in simulations of a reactive plume using a Lagrangian stochastic model

Numerical simulations of metal alloy solidification are used to gain insight into physical phenomena that cannot be observed experimentally. These models produce results that are used to draw conclusions about a process or alloy and often compared to experimental results. However, uncertainty in model inputs cause uncertainty in model results, which have the potential to significantly affect conclusions drawn from their predictions. As a step toward understanding the effect of uncertain inputs on solidification modeling, uncertainty quantification (UQ) and sensitivity analysis are performed on a transient model of solidification of Al–4.5 wt % Cu in a rectangular cavity. The binary alloy considered has columnar solidification morphology, and this model solves equations for momentum, temperature, and species conservation. UQ and sensitivity analysis are performed for the degree of macrosegregation and solidification time. A Smolyak sparse grid algorithm is used to select input values to construct a polynomial response surface fit to model outputs. This polynomial is then used as a surrogate for the complete solidification model to determine the sensitivities and probability density functions (PDFs) of the model outputs. Uncertain model inputs of interest include the secondary dendrite arm spacing (SDAS), heat transfer coefficient, and material properties. The most influential input parameter for predicting the macrosegregation level is the dendrite arm spacing, which also strongly depends on the choice of permeability model. Additionally, the degree of uncertainty required to produce accurate predictions depends on the outputs of interest from the model.

This paper gives a broad overview of a complete framework for assessing the predictive uncertainty of scientific computing applications. The framework is complete in the sense that it treats both types of uncertainty (aleatory and epistemic) and inco rporates uncertainty due to the form of the model and any numerical approximations used. Aleatory (or random) uncertainties in model inputs are treated using cumulative distribution functions, while epistemic (lack of knowledge) uncertainties are treated a s intervals. Approaches for propagating both types of uncertainties through the model to the system response quantities of interest are discussed. Numerical approximation errors (due to discretization, iteration, and round off) are estimated using verifica tion techniques, and the conversion of these errors into epistemic uncertainties is discussed. Model form uncertainties are quantified using model validation procedures, which include a comparison of model predictions to experimental data and then extrapolation of this uncertainty structure to points in the application domain where experimental data do not exist. Finally, methods for conveying the total predictive uncertainty to decision makers are presented.

BackgroundWe evaluated the cost-effectiveness of nivolumab versus everolimus in patients with advanced renal cell carcinoma (RCC) from a US payer perspective.MethodsA partitioned survival model consisting of three health states, progression-free survival (PFS), progressive disease, and death, was developed to evaluate the cost-effectiveness of intravenous nivolumab versus oral everolimus over a lifetime. The proportion of patients in each state was calculated based on parametric distributions fitted to PFS and overall survival (OS) data from CheckMate 025 (N = 821), a large randomized phase 3 trial of nivolumab versus everolimus for advanced RCC. Health state utility data were derived from CheckMate 025 EQ-5D data. Scenario analyses and deterministic and probabilistic sensitivity analyses assessed the impact of uncertainty in model inputs on outcomes.ResultsOver a 25-year lifetime horizon, treatment with nivolumab resulted in a gain of 0.64 quality-adjusted life-years (QALYs) versus everolimus. Nivolumab had greater total costs versus everolimus ($US197,089 vs. $US163,902), mainly due to higher acquisition costs. The incremental cost-utility ratio (ICUR), a measure of incremental costs divided by incremental QALYs, was $US51,714 per QALY gained for nivolumab versus everolimus, and an incremental cost-effectiveness ratio was $US44,576 per life-year gained for nivolumab versus everolimus. In sensitivity analyses, average body weight had the greatest impact on the ICUR for nivolumab versus everolimus (base case $US51,714; range $US8863–$US94,566). At a $US150,000 willingness-to-pay (WTP) threshold, nivolumab had a 91.7% probability of being cost-effective versus everolimus.ConclusionsIn the United States, at a WTP threshold of $US150,000 per QALY, nivolumab was found to be cost-effective. Key drivers of cost-effectiveness were survival inputs for OS and the average weight of patients; the latter directly affects nivolumab drug acquisition cost.

Restoration of shoulder function is an important treatment goal in upper brachial plexus injury (UBPI). Combined dual motor nerve transfer (CDNT) of spinal accessory to suprascapular and radial to axillary nerves demonstrates good functional recovery with minimal risk of perioperative complications. To evaluate the cost-effectiveness of CDNT vs alternative operative and nonoperative treatments for UBPI. A decision model was constructed to evaluate costs ($, third-party payer) and effectiveness (quality-adjusted life years [QALYs]) of CDNT compared to glenohumeral arthrodesis (GA), conservative management, and nontreatment strategies. Estimates for branch probabilities, costs, and QALYs were derived from published studies. Incremental cost-effectiveness ratios (ICER, $/QALY) were calculated to compare the competing strategies. One-way, 2-way, and probabilistic sensitivity analyses with 100 000 iterations were performed to account for effects of uncertainty in model inputs. Base case model demonstrated CDNT effectiveness, yielding an expected 21.04 lifetime QALYs, compared to 20.89 QALYs with GA, 19.68 QALYs with conservative management, and 19.15 QALYs with no treatment. The ICERs for CDNT, GA, and conservative management vs nontreatment were $5776.73/QALY, $10 483.52/QALY, and $882.47/QALY, respectively. Adjusting for potential income associated with increased likelihood of returning to work after clinical recovery demonstrated CDNT as the dominant strategy, with ICER=-$56 459.54/QALY relative to nontreatment. Probabilistic sensitivity analysis showed CDNT cost-effectiveness at a willingness-to-pay threshold of $50 000/QALY in 78.47% and 81.97% of trials with and without income adjustment, respectively. Conservative management dominated in <1% of iterations. CDNT and GA are cost-effective interventions to restore shoulder function in patients with UBPI.