Guidance on evaluating parametric model uncertainty at decision-relevant scales

An interactive graphical interface tool for parameter calibration, sensitivity analysis, uncertainty analysis, and visualization for the Soil and Water Assessment Tool

Ecosystem models are useful tools for understanding ecological processes and for sustainable management of resources. In biogeochemical field, numerical models have been widely used for investigating carbon dynamics under global changes from site to regional and global scales. However, it is still challenging to optimize parameters and estimate parameterization uncertainty for complex process‐based models such as the Erosion Deposition Carbon Model (EDCM), a modified version of CENTURY, that consider carbon, water, and nutrient cycles of ecosystems. This study was designed to conduct the parameter identifiability, optimization, sensitivity, and uncertainty analysis of EDCM using our developed EDCM‐Auto, which incorporated a comprehensive R package—Flexible Modeling Framework (FME) and the Shuffled Complex Evolution (SCE) algorithm. Using a forest flux tower site as a case study, we implemented a comprehensive modeling analysis involving nine parameters and four target variables (carbon and water fluxes) with their corresponding measurements based on the eddy covariance technique. The local sensitivity analysis shows that the plant production‐related parameters (e.g., PPDF1 and PRDX) are most sensitive to the model cost function. Both SCE and FME are comparable and performed well in deriving the optimal parameter set with satisfactory simulations of target variables. Global sensitivity and uncertainty analysis indicate that the parameter uncertainty and the resulting output uncertainty can be quantified, and that the magnitude of parameter‐uncertainty effects depends on variables and seasons. This study also demonstrates that using the cutting‐edge R functions such as FME can be feasible and attractive for conducting comprehensive parameter analysis for ecosystem modeling.

Preface. Securitisation deals have come into focus during the recent years due to the challenges in their assessments and their role in the recent credit crises. These deals are created by the pooling of assets and the tranching of liabilities. The later are backed by the collateral pools. Tranching makes it possible to create liabilities of a variety of seniorities and risk-return profiles. The assessment of a securitisation deal is based on qualitative and quantitative assessments of the risks inherent in the transaction and how well the structure manages to mitigate these risks. Example of risks related to the performance of a transaction are credit risk, prepayment risk, market risk, liquidity risk, counterparty risk, operational risk and legal risk. In the light of the recent credit crisis, model risk and parameter uncertainty have come in focus. Model risk refers to the fact that the outcome of the assessment of a securitisation transaction can be influenced by the choice of the model used to derive defaults and prepayments. The uncertainties in the parameter values used as input to these models add to the uncertainty of the output of the assessment. The aim of this report is to give an overview of recent performed research on model risk and parameter sensitivity of asset backed securities ratings. The outline of the text is as follows.3 In Chapter 1, an introduction to asset-backed securities (ABSs) is given. We describe, for example, key securitisation parties, structural characteristics and credit enhancements. The cashflow modelling of ABS deals can be divided into two parts: (1) the modelling of the cash collections from the asset pool and the distribution of these collections to the note holders, discussed in Chapter 2, and (2) the modelling of defaults and prepayments. Deterministic models to generate default and prepayment scenarios are presented in Chapter 3; a collection of stochastic models is presented in Chapter 4. In Chapter 5, two of the major rating agencies quantitative methodologies for ABS rating are discussed. Next, the model risk in rating ABSs is discussed and we elaborate on the parameter sensitivity of ABS ratings. More precisely, in Chapter 6 we look at how the choice of default model influences the ratings of an ABS structure. We illustrate this using a two tranche ABS. Furthermore, we also investigate the influence of changing some of the input parameters one at a time. A more systematic parameter sensitivity analysis is presented in Chapter 7. In this chapter we introduce global sensitivity analysis techniques, which allow us to systematically analyse how the uncertainty in each input parameter’s value contributes to the uncertainty of the expected loss and the expected average life of the notes and hence the rating. The report concludes with an summary of the findings in Chapter 8.

A parametric uncertainty and sensitivity analysis of hydrodynamic processes was conducted for a large shallow freshwater lake, Lake Taihu, China. Ten commonly used parameters in five groups were considered including: air–water interface factor, water–sediment interface factor, surrounding terrain factor, turbulent diffusion parameters and turbulent intensity parameters. Latin hypercube sampling (LHS) was used for sampling the parametric combinations, which gave predictive uncertainty results directly without using surrogate models, and the impacts of different parametric distribution functions on the results were investigated. The results showed that the different parametric distribution functions (e.g. uniform, normal, lognormal and triangular) for sampling had very little impact on the uncertainty and sensitivity analysis of the lake hydrodynamic model. The air–water interface factor (wind drag coefficient) and surrounding terrain factor (wind shelter coefficient) had the greatest influence on the spatial distribution of lake hydrodynamic processes, especially in semi-closed bays and lake regions with complex topography, accounting for about 60–70% and 20%, respectively, of the uncertainty on the results. Vertically, velocity in the surface layer was also largely influenced by the two factors, followed by velocity in the bottom layer; the middle velocity had minimal impact. Likewise, the water–sediment interface factor (i.e. bottom roughness height) ranked third, contributing about 10% to the uncertainty of the hydrodynamic processes of the lake. In contrast, turbulent diffusion parameters and turbulent intensity parameters in the lake hydrodynamic model had little effect on the uncertainty of simulated results (less than 1% contribution). The findings were sufficiently significant to reduce the parameter uncertainties and calibration workload of the hydrodynamic model in large shallow lakes.Editor Z. W. Kundzewicz; Associate editor S. Grimaldi

Physically-based urban rainfall-runoff models are mostly applied without parameter calibration. Given some preliminary estimates of the uncertainty of the model parameters the associated model output uncertainty can be calculated. Monte-Carlo simulation followed by multi-linear regression is used for this analysis. The calculated model output uncertainty can be compared to the uncertainty estimated by comparing model output and observed data. Based on this comparison systematic or spurious errors can be detected in the observation data, the validity of the model structure can be confirmed, and the most sensitive parameters can be identified. If the calculated model output uncertainty is unacceptably large the most sensitive parameters should be calibrated to reduce the uncertainty. Observation data for which systematic and/or spurious errors have been detected should be discarded from the calibration data. This procedure is referred to as preliminary uncertainty analysis; it is illustrated with an example. The HYSTEM program is applied to predict the runoff volume from an experimental catchment with a total area of 68 ha and an impervious area of 20 ha. Based on the preliminary uncertainty analysis, for 7 of 10 events the measured runoff volume is within the calculated uncertainty range, i.e. less than or equal to the calculated model predictive uncertainty. The remaining 3 events include most likely systematic or spurious errors in the observation data (either in the rainfall or the runoff measurements). These events are then discarded from further analysis. After calibrating the model the predictive uncertainty of the model is estimated.

A calibration procedure is essential step to achieve a realistic model simulation particularly in hydrological model which simulates water cycle in the basin. This process is always faced with challenges due to selection of objective function and highly time-consuming. This study aimed to take advantage of parallel processing to accelerate the computations involved with simulation process of hydrologic model linked with the multi-objective optimization algorithm of AMALGAM for multi-site calibration of SWAT hydrologic model parameters. In order to illustrate how meaningful SWAT model calibration trade-off between the four objective functions involved in AMALGAM optimization program, the Pareto solution sets were provided. Furthermore, it is implemented a group of model runs with a number of cores involved (from one to eight) to demonstrate and evaluate the running of parallelized AMALGAM with taking advantages of “spmd” method to decrease the running time of the SWAT model. The results revealed the robustness of the method in reducing computational time of the parameter calibration significantly. This strategy with 4-objective functions focuses on high streamflow (Nash–Sutcliffe coefficient), low streamflow (Box–Cox transformed root–mean–square error), water balance (runoff coefficient error), and flashiness (slope of the flow duration curve error) provided an efficient tool to decide about the best simulation based on the investigated objective functions. This study also provides a strong basis for multi-objective optimization of hydrological and water quality models and its general analytical framework could be applied to other parts of the world.

In this paper, for the first time, a unique pTFET-based biosensor device having channel epilayer has been introduced and its performance in the sensing domain has been studied in terms of five different sensitivity parameters. A rigorous analysis of the effect of the existent corner point (corner-effect) in the proposed device architecture on the sensitivity metrics has been performed for five different channel epilayer thicknesses, covering five different types of protein-molecules, viz., Apomyoglobin, Myoglobin, Protein-G, Ferricytochrome-C, and Ferrocytochrome-C. This is followed by determining detectability and optimum length-window of nanogap cavity of the proposed sensor device for the successful detection. Interestingly, it has been found that the undesired corner-effect, generally known for its unfavorable impact on the electrical behaviors of the MOS-based structures, actually comes to the aid by enhancing the detectability of the proposed device compared to its equivalent conventional SOI pTFET sensor device by almost 67% (for Protein-G, Ferricytochrome-C, and Ferrocytochrome-C) and 50% (for Apomyoglobin and Myoglobin) yet maintaining significantly good values for the sensitivity metrics throughout. Furthermore, smaller epilayer thicknesses ensure smaller optimum length-window of the nanogap cavity, resulting in more scaled-down device, compared to larger epilayer thickness values. A position-dependent variability study for the detectability also follows, and it has been found that for small epilayer thicknesses this dependency becomes large leading to less stable device, although, at the same time producing higher sensitivity metrics compared to the devices having large epilayer thicknesses. This makes the choice of channel epilayer thickness as application-specific.

The mapped rivers and streams of the contiguous United States are available in a geographic information system (GIS) dataset called National Hydrography Dataset Plus (NHDPlus). This hydrographic dataset has about 3 million river and water body reaches along with information on how they are connected into networks. The U.S. Geological Survey (USGS) National Water Information System (NWIS) provides streamflow observations at about 20 thousand gauges located on the NHDPlus river network. A river network model called Routing Application for Parallel Computation of Discharge (RAPID) is developed for the NHDPlus river network whose lateral inflow to the river network is calculated by a land surface model. A matrix-based version of the Muskingum method is developed herein, which RAPID uses to calculate flow and volume of water in all reaches of a river network with many thousands of reaches, including at ungauged locations. Gauges situated across river basins (not only at basin outlets) are used to automatically optimize the Muskingum parameters and to assess river flow computations, hence allowing the diagnosis of runoff computations provided by land surface models. RAPID is applied to the Guadalupe and San Antonio River basins in Texas, where flow wave celerities are estimated at multiple locations using 15-min data and can be reproduced reasonably with RAPID. This river model can be adapted for parallel computing and although the matrix method initially adds a large overhead, river flow results can be obtained faster than with the traditional Muskingum method when using a few processing cores, as demonstrated in a synthetic study using the upper Mississippi River basin.

Parameter uncertainty is a primary source of uncertainty in ocean ecosystem simulations. The deep chlorophyll maximum (DCM) is a ubiquitous ecological phenomenon in the ocean. Using a theoretical nutrients-phytoplankton model and the conditional nonlinear optimal perturbation approach related to parameters, we investigated the effects of parameter uncertainties on DCM simulations. First, the sensitivity of single parameter was analyzed. The sensitivity ranking of 10 parameters was obtained by analyzing the top four specifically. The most sensitive parameter (background turbidity) affects the light supply for DCM formation, whereas the other three parameters (nutrient content of phytoplankton, nutrient recycling coefficient, and vertical turbulent diffusivity) control nutrient supply. To explore the interactions among different parameters, the sensitivity of multiple parameters was further studied by examining combinations of four parameters. The results show that background turbidity is replaced by the phytoplankton loss rate in the optimal parameter combination. In addition, we found that interactions among these parameters are responsible for such differences. Finally, we found that reducing the uncertainties of sensitive parameters could improve DCM simulations remarkably. Compared with the sensitive parameters identified in the single parameter analysis, reducing parameter uncertainties in the optimal combination produced better model performance. This study shows the importance of nonlinear interactions among various parameters in identifying sensitive parameters. In the future, the conditional nonlinear optimal perturbation approach related to parameters, especially optimal parameter combinations, is expected to greatly improve DCM simulations in complex ecosystem models.

Abstract. Soil organic carbon (SOC) has a significant effect on carbon emissions and climate change. However, the current SOC prediction accuracy of most models is very low. Most evaluation studies indicate that the prediction error mainly comes from parameter uncertainties, which can be improved by parameter calibration. Data assimilation techniques have been successfully employed for the parameter calibration of SOC models. However, data assimilation algorithms, such as the sampling-based Bayesian Markov chain Monte Carlo (MCMC), generally have high computation costs and are not appropriate for complex global land models. This study proposes a new parameter calibration method based on surrogate optimization techniques to improve the prediction accuracy of SOC. Experiments on three types of soil carbon cycle models, including the Community Land Model with the Carnegie–Ames–Stanford Approach biogeochemistry submodel (CLM-CASA') and two microbial models show that the surrogate-based optimization method is effective and efficient in terms of both accuracy and cost. Compared to predictions using the tuned parameter values through Bayesian MCMC, the root mean squared errors (RMSEs) between the predictions using the calibrated parameter values with surrogate-base optimization and the observations could be reduced by up to 12 % for different SOC models. Meanwhile, the corresponding computational cost is lower than other global optimization algorithms.

Physics-based models, particularly urban canopy models (UCM), have been widely used to investigate urban microclimate conditions of various urban areas. UCMs have been developed to simulate the interaction between buildings, roads, and vegetation, including the surface energy balance, to predict urban climate. Despite its developments, deployment of UCM for city-scale urban simulations presents major challenges as follows: (1) UCM requires many uncertain input parameters, referred to as parameter uncertainty, due to the lack of data availability; (2) UCM is based on the concept of urban canyon that assumes homogeneous urban settings with one representative building geometry, and this assumption pertains to model-form uncertainty. Few studies attempted to address the parameter uncertainty via calibration of uncertain input parameters. Other some studies have compared the prediction accuracy of multiple UCM types or integration of UCMs with different models. Yet, the effect of homogeneous urban canyon assumption on the model discrepancy have not been sufficiently explored and there is no knowledge base to identify key urban factors, unrepresented in UCMs, relating to model discrepancy.To address the research gap, this work aims to improve the prediction accuracy of an existing UCM model for wind speed prediction by handling both parameter uncertainty and model-form uncertainty under various urban contexts. First, an existing UCM, specifically the vertical city weather generator (VCWG) model, was used to predict wind speed at 248 weather stations in Seoul, Republic of Korea. Our previous study demonstrated that the root-mean-square error (RMSE) of VCWG wind speed model at 248 weather stations ranged between 0.14 and 20.14 m/s [1]. Second, we calibrated three important uncertain parameters selected on the basis of sensitivity analysis results, namely turbulence coefficient during unstable conditions, pressure gradient coefficient, and aerodynamic roughness length. Third, to address model form uncertainty, we identified urban factors that are not represented in the VCWG model, such as urban heterogeneity, topography, and blue infrastructure. Fourth, we develop statistical model(s) associated with unrepresented urban factors to quantify the model-form uncertainty and, ultimately, reduce the gap between the model and reality. It is noteworthy that the 248 weather stations used were deployed in different vertical layers: urban canopy-level, roughness sublayer-level, and inertial sublayer-level, which allows to improve UCM prediction in different vertical heights. This study will deliver a calibrated VCWG model that is able to accurately predict actual wind speed and provide a knowledge base regarding the possible causes of model discrepancies.

In the present study empirical modeling was used to estimate extreme monthly streamflow in 161 locations of hydroelectric plants in the main Brazilians basins through a climate indices set. Principal components analysis was applied to capture the combined influences of climate indices variability. The main predictors of the models were climate patterns of tropical Atlantic and Pacific (Tropical Southern Atlantic and El Niño Southern Oscillation (ENSO) Modoki), complemented by high latitudes patterns (Antarctic Oscillation and Pacific North American Pattern). The contribution of the ENSO/Pacific Decadal Oscillation mode occurs in preferential months, especially in the transition seasons. There is also a contribution of the lagged streamflow itself, mainly in basins located in the Southeast of Brazil. The accuracy of the model for most of the Brazilian basins is higher than 70%, with higher values than 90% for the estimate of very low streamflow in northeast and north-central Brazil, as well as for very high streamflow in south-central region, which performance decreases with the increase in the lag. The results showed that the climate indices have a highly predictive potential for extreme streamflow, with a higher predictability for more long-term forecasts in case of very low streamflow than very high streamflow.

<div> <p>The latest generation of integrated hydrologic models provides new opportunities to better understand and hypothesize about the connections between hydrological, ecological and energy transfer processes across a range of scales. Parallel to this has been unprecedented growth in new technologies to observe components of Earth’s biophysical system through satellite remote sensing or on-the-ground instruments. However, along with growth in available data and advanced modelling platforms comes a challenge to ensure models are representative of catchment systems and are not unrealistically confident in their predictions. Many hydrologic and ecosystem variables are measured infrequently, measured with significant error, or are measured at a scale different to their representation in a model. In fact, the modelled variable of interest is frequently not directly observed but inferred from surrogate measurements. This introduces errors in model calibration that will affect whether our models are representative of the systems we seek to understand.</p> </div><div> <p>In recent years, Bayesian inference has emerged as a powerful tool in the environmental modeler’s toolbox, providing a convenient framework in which to model parameter and observational uncertainties. The Bayesian approach is ideal for multivariate model calibration, by defining proper prior distributions that can be considered analogous to the weighting often prescribed in traditional multi-objective calibration. </p> </div><div> <p>In this study, we develop a multi-objective Bayesian approach to hydrologic model inference that explicitly capitalises on a priori knowledge of observational errors to improve parameter estimation and uncertainty estimation. We introduce a novel error model, which partitions observation and model residual error according to prior knowledge of the estimated uncertainty in the calibration data. We demonstrate our approach in two case studies: an ecohydrologic model where we make use of the known uncertainty in satellite retrievals of Leaf Area Index (LAI), and a water quality model using turbidity as a proxy for Total Suspended Solids (TSS). Overall, we aim to demonstrate the need to properly account for known observational errors in proper hydrologic model calibration.</p> </div>

Assessment of statistical parameter uncertainty in the reliability analysis of jacket platforms

Development and calibration of digital twins for human skin growth in tissue expansion.