Framework for Understanding Structural Errors (FUSE): A modular framework to diagnose differences between hydrological models

Simulations of human behavior in water resources systems are challenged by uncertainty in model structure and parameters. The increasing availability of observations describing these systems provides the opportunity to infer a set of plausible model structures using data‐driven approaches. This study develops a three‐phase approach to the inference of model structures and parameterizations from data: problem definition, model generation, and model evaluation, illustrated on a case study of land use decisions in the Tulare Basin, California. We encode the generalized decision problem as an arbitrary mapping from a high‐dimensional data space to the action of interest and use multiobjective genetic programming to search over a family of functions that perform this mapping for both regression and classification tasks. To facilitate the discovery of models that are both realistic and interpretable, the algorithm selects model structures based on multiobjective optimization of (1) their performance on a training set and (2) complexity, measured by the number of variables, constants, and operations composing the model. After training, optimal model structures are further evaluated according to their ability to generalize to held‐out test data and clustered based on their performance, complexity, and generalization properties. Finally, we diagnose the causes of good and bad generalization by performing sensitivity analysis across model inputs and within model clusters. This study serves as a template to inform and automate the problem‐dependent task of constructing robust data‐driven model structures to describe human behavior in water resources systems.

A new method is developed for calculating Bayes posterior distributions of future catches that conform to a specified harvest control law while incorporating uncertainty in biological reference points, natural mortality, and some aspects of model structure in addition to the usual stochastic noise. A Markov chain Monte Carlo approach is used to calculate Bayesian posterior distributions for critical parameters of a Norwegian spring-spawning herring (Clupea harengus) stock assessment using an assessment model that incorporates catch-at-age, survey, and tag release and recapture observations. Exceptionally, the approach allows prior uncertainty in model structure (e.g., whether survey observation errors should be treated as normal, lognormal, or gamma variates; whether Ricker or Beverton-Holt forms are used to model recruitment). This modelling approach is a useful tool that allows management advice to be provided that takes into account uncertainty in model structures and in some parameters that, by conventional methods, need to be specified as arbitrary "best" choices. The method is also used to quantify uncertainty in some biological reference points and to calculate a probability distribution for a future catch with respect to a specified harvest control law. This has the advantage of a consistent treatment of uncertainty throughout the process of stock modelling, reference point estimation, and concomitant catch forecasting.

Abstract. Benchmarking model performance across large samples of catchments is useful to guide model selection and future model development. Given uncertainties in the observational data we use to drive and evaluate hydrological models, and uncertainties in the structure and parameterisation of models we use to produce hydrological simulations and predictions, it is essential that model evaluation is undertaken within an uncertainty analysis framework. Here, we benchmark the capability of several lumped hydrological models across Great Britain by focusing on daily flow and peak flow simulation. Four hydrological model structures from the Framework for Understanding Structural Errors (FUSE) were applied to over 1000 catchments in England, Wales and Scotland. Model performance was then evaluated using standard performance metrics for daily flows and novel performance metrics for peak flows considering parameter uncertainty. Our results show that lumped hydrological models were able to produce adequate simulations across most of Great Britain, with each model producing simulations exceeding a 0.5 Nash–Sutcliffe efficiency for at least 80 % of catchments. All four models showed a similar spatial pattern of performance, producing better simulations in the wetter catchments to the west and poor model performance in central Scotland and south-eastern England. Poor model performance was often linked to the catchment water balance, with models unable to capture the catchment hydrology where the water balance did not close. Overall, performance was similar between model structures, but different models performed better for different catchment characteristics and metrics, as well as for assessing daily or peak flows, leading to the ensemble of model structures outperforming any single structure, thus demonstrating the value of using multi-model structures across a large sample of different catchment behaviours. This research evaluates what conceptual lumped models can achieve as a performance benchmark and provides interesting insights into where and why these simple models may fail. The large number of river catchments included in this study makes it an appropriate benchmark for any future developments of a national model of Great Britain.

<p>Recent investigations have shown it is possible to simultaneously calibrate model structures and model parameters to identify appropriate models for a given task (Spieler et al., 2020). However, this is computationally challenging, as different model structures may use a different number of parameters. While some parameters may be shared between model structures, others might be relevant for only a few structures, which theoretically requires the calibration of conditionally active parameters. Additionally, shared model parameters might cause different effects in different model structures, causing their optimal values to differ across structures. In this study, we tested how two current “of the shelf” mixed-integer optimization algorithms perform when having to handle these peculiarities during the automatic model structure identification (AMSI) process recently introduced by Spieler et al. (2020).</p><p>To validate the current performance of the AMSI approach, we conduct a benchmark experiment with a model space consisting of 6912 different model structures.  First, all model structures are independently calibrated and validated for three hydro-climatically differing catchments using the CMA-ES algorithm and KGE as the objective function. This is referred to as standard calibration procedure. We identify the best performing model structure(s) based on validation performance and analyze the range of performance as well as the number of structures performing in a similar range. Secondly, we run AMSI on all three catchments to automatically identify the most feasible model structure based on the KGE performance. Two different mixed-integer optimization algorithms are used – namely DDS and CMA-ES. Afterwards, we compare the results to the best performing models of the standard calibration of all 6912 model structures.</p><p>Within this experimental setup, we analyze if the best performing model structure(s) AMSI identifies are identical to the best performing structures of the standard calibration and if there are differences in performance when using different optimization algorithms for AMSI. We also validate if AMSI can identify the best performing model structures for a catchment at a fraction of the computational cost than the standard calibration procedure requires by using “off the shelf” mixed-integer optimization algorithms.</p><p> </p><p> </p><p> </p><p>Spieler, D., Mai, J., Craig, J. R., Tolson, B. A., & Schütze, N. (2020). Automatic Model Structure Identification for Conceptual Hydrologic Models. Water Resources Research, 56(9). https://doi.org/10.1029/2019WR027009</p>

BackgroundIdentifying a proper model structure, using methods that address both structural and parameter uncertainty, is a crucial problem within the systems approach to biology. And yet, it has a marginal presence in the recent literature. While many existing approaches integrate methods for simulation and parameter estimation of a single model to address parameter uncertainty, only few of them address structural uncertainty at the same time. The methods for handling structure uncertainty often oversimplify the problem by allowing the human modeler to explicitly enumerate a relatively small number of alternative model structures. On the other hand, process-based modeling methods provide flexible modular formalisms for specifying large classes of plausible model structures, but their scope is limited to deterministic models. Here, we aim at extending the scope of process-based modeling methods to inductively learn stochastic models from knowledge and data.ResultsWe combine the flexibility of process-based modeling in terms of addressing structural uncertainty with the benefits of stochastic modeling. The proposed method combines search trough the space of plausible model structures, the parsimony principle and parameter estimation to identify a model with optimal structure and parameters. We illustrate the utility of the proposed method on four stochastic modeling tasks in two domains: gene regulatory networks and epidemiology. Within the first domain, using synthetically generated data, the method successfully recovers the structure and parameters of known regulatory networks from simulations. In the epidemiology domain, the method successfully reconstructs previously established models of epidemic outbreaks from real, sparse and noisy measurement data.ConclusionsThe method represents a unified approach to modeling dynamical systems that allows for flexible formalization of the space of candidate model structures, deterministic and stochastic interpretation of model dynamics, and automated induction of model structure and parameters from data. The method is able to reconstruct models of dynamical systems from synthetic and real data.

<p>Recent studies have introduced methods to simultaneously calibrate model structure choices and parameter values to identify an appropriate (conceptual) model structure for a given catchment. This can be done through mixed-integer optimization to identify the graph structure that links dominant flow processes (Spieler et al., 2020) or, likewise, by continuous optimization of weights when blending multiple flux equations to describe flow processes within a model (Chlumsky et al., 2021). Here, we use the combination of the mixed-integer optimization algorithm DDS and the modular modelling framework RAVEN and refer to it as Automatic Model Structure Identification (AMSI) framework.</p><p>This study validates the AMSI framework by comparing the performance of the identified AMSI model structures to two different benchmark ensembles. The first ensemble consists of the best model structures from the brute force calibration of all possible structures included in the AMSI model space (7488+). The second ensemble consists of 35+ MARRMoT structures representing a structurally more divers set of models than currently implemented in the AMSI framework. These structures stem from the MARRMoT Toolbox introduced by Knoben et al. (2019) providing established conceptual model structures based on hydrologic literature.</p><p>We analyze if the model structure(s) AMSI identifies are identical to the best performing structures of the brute force calibration and comparable in their performance to the MARRMoT ensemble. We can conclude that model structures identified with the AMSI framework can compete with the structurally more divers MARRMoT ensemble. In fact, we were surprised to see how well we do with a simple two storage structure over the 12 tested MOPEX catchments (Duan et al.,2006). We aim to discuss several emerging questions, such as the selection of a robust model structure, Equifinality in model structures, and the role of structural complexity.</p><p> </p><p>Spieler et al. (2020). https://doi.org/10.1029/2019WR027009</p><p>Chlumsky et al. (2021). https://doi.org/10.1029/2020WR029229</p><p>Knoben et al. (2019). https://doi.org/10.5194/gmd-12-2463-2019</p><p>Duan et al. (2006). https://doi.org/10.1016/j.jhydrol.2005.07.031</p>

Structural uncertainty relates to differences in model structure and parameterization. For many published health economic analyses in oncology, substantial differences in model structure exist, leading to differences in analysis outcomes and potentially impacting decision-making processes. The objectives of this analysis were (1) to identify differences in model structure and parameterization for cost-effectiveness analyses (CEAs) comparing tamoxifen and anastrazole for adjuvant breast cancer (ABC) treatment; and (2) to quantify the impact of these differences on analysis outcome metrics. The analysis consisted of four steps: (1) review of the literature for identification of eligible CEAs; (2) definition and implementation of a base model structure, which included the core structural components for all identified CEAs; (3) definition and implementation of changes or additions in the base model structure or parameterization; and (4) quantification of the impact of changes in model structure or parameterizations on the analysis outcome metrics life-years gained (LYG), incremental costs (IC) and the incremental cost-effectiveness ratio (ICER). Eleven CEA analyses comparing anastrazole and tamoxifen as ABC treatment were identified. The base model consisted of the following health states: (1) on treatment; (2) off treatment; (3) local recurrence; (4) metastatic disease; (5) death due to breast cancer; and (6) death due to other causes. The base model estimates of anastrazole versus tamoxifen for the LYG, IC and ICER were 0.263 years, €3,647 and €13,868/LYG, respectively. In the published models that were evaluated, differences in model structure included the addition of different recurrence health states, and associated transition rates were identified. Differences in parameterization were related to the incidences of recurrence, local recurrence to metastatic disease, and metastatic disease to death. The separate impact of these model components on the LYG ranged from 0.207 to 0.356 years, while incremental costs ranged from €3,490 to €3,714 and ICERs ranged from €9,804/LYG to €17,966/LYG. When we re-analyzed the published CEAs in our framework by including their respective model properties, the LYG ranged from 0.207 to 0.383 years, IC ranged from €3,556 to €3,731 and ICERs ranged from €9,683/LYG to €17,570/LYG. Differences in model structure and parameterization lead to substantial differences in analysis outcome metrics. This analysis supports the need for more guidance regarding structural uncertainty and the use of standardized disease-specific models for health economic analyses of adjuvant endocrine breast cancer therapies. The developed approach in the current analysis could potentially serve as a template for further evaluations of structural uncertainty and development of disease-specific models.

Abstract. Low flows are often poorly reproduced by commonly used hydrological models, which are traditionally designed to meet peak flow situations. Hence, there is a need to improve hydrological models for low flow prediction. This study assessed the impact of model structure on low flow simulations and recession behaviour using the Framework for Understanding Structural Errors (FUSE). FUSE identifies the set of subjective decisions made when building a hydrological model and provides multiple options for each modeling decision. Altogether 79 models were created and applied to simulate stream flows in the snow dominated headwater catchment Narsjø in Norway (119 km2). All models were calibrated using an automatic optimisation method. The results showed that simulations of summer low flows were poorer than simulations of winter low flows, reflecting the importance of different hydrological processes. The model structure influencing winter low flow simulations is the lower layer architecture, whereas various model structures were identified to influence model performance during summer.

Climate warming is rapidly modifying glacier and snowmelt processes, with impacts that propagate across spatial and temporal scales, from high-elevation cryospheric dynamics to basin-scale streamflow. Capturing these cross-scale interactions requires hydrological models that explicitly represent cryospheric processes, such as glacier dynamics and snow redistribution, while also accounting for uncertainty arising from data scarcity and model structure. To address these challenges, we present a one-way coupling that integrates the Global Glacier Evolution Model (GloGEM) with Raven, a flexible hydrological modeling framework designed to emulate multiple model structures for systematic uncertainty assessment. Glacier simulations are first improved by incorporating snowline observations as an additional calibration constraint to the geodetic mass balance, reducing parameter equifinality and enhancing glacier runoff estimates. The resulting glacier runoff is then provided to the hydrological model, leading to a more robust representation of snowmelt processes and their contribution to streamflow. By evaluating multiple hydrological model structures within Raven, we further quantify uncertainties in simulated melt contributions arising from structural model choices. The framework is applied to multiple data-scarce Indus headwaters as representative high-mountain catchments, simulating melt contributions and streamflow across scales. By explicitly coupling cryosphere and hydrological models, this framework aims to improve projections of future water availability in data-scarce regions and provides a transferable approach for integrated cryosphere-hydrology modeling in complex mountain regions.

Added value of distribution in rainfall-runoff models for the Meuse basin

To determine a suitable hydrological model structure for a specific application context using integrated modelling frameworks, modellers usually need to manually select the required hydrological processes, identify the appropriate algorithm for each process, and couple the algorithms' software components. However, these modelling steps are difficult and require corresponding knowledge. It is not easy for modellers to master all of the required knowledge. To alleviate this problem, a knowledge-based method is proposed to automatically determine hydrological model structures. First, modelling knowledge for process selection, algorithm identification, and component coupling is formalized in the formats of the Rule Markup Language (RuleML) and Resource Description Framework (RDF). Second, the formalized knowledge is applied to an inference engine to determine model structures. The method is applied to three hypothetical experiments and a real experiment. These experiments show how the knowledge-based method could support modellers in determining suitable model structures. The proposed method has the potential to reduce the knowledge burden on modellers and would be conducive to the promotion of integrated modelling frameworks.

Artificial neural networks (ANNs) have been extensively used to forecast monthly precipitation for water resources management over the past few decades. Efforts to produce more accurate and stable forecasts face ongoing challenges as the so‐called single‐ANN (S‐ANN) approach has several limitations, particularly regarding uncertainty. Many attempts have been made to deal with different types of uncertainties by applying ensemble approaches. Here, we propose a new ANN ensemble model (ANN‐ENS) dealing with uncertainty in model structure and input variable selection to provide a more accurate and stable forecasting performance. This model is structured by generating various input layers, considering all the candidate input variables (i.e.,large‐scale climate indices and lagged precipitation). We developed a modified backward elimination method to select the preliminary input variables from all the candidate input variables. Then, we tested and validated the proposed ANN‐ENS using observed monthly precipitation from 10 meteorological stations in the Han River basin, South Korea. Our results demonstrated that the ANN‐ENS enhanced the forecasting performance in terms of both accuracy and stability. Although a significant uncertainty was introduced by using all the candidate input variables, the forecasting result outperformed S‐ANNs for all employed stations. Additionally, the ANN‐ENS provided a more stable forecasting performance in comparison with S‐ANNs, which are highly sensitive. Moreover, the generated ensemble members were slightly biased at some stations but were generally reliable.

Integrated Bayesian Multi-model approach to quantify input, parameter and conceptual model structure uncertainty in groundwater modeling

Various hydrological models have been developed for estimating root zone soil moisture dynamics. These models, however, incorporated their own parameterization approaches indicating that the output from the different model inherent structures might include uncertainties because we do not know which model structure is correct for describing the real system. More recently, multimodel approaches using a Bayesian Model Averaging (BMA) scheme can improve the overall predictive skill while individual models retain their own uncertainties for simulated soil moisture based on a single set of weights in modeling under different land surface wetness conditions (e.g., wet, moderately wet, and dry conditions). In order to overcome their limitations, we developed a BMA‐based multimodel simulation approach based on various soil wetness conditions for estimating effective surface soil moisture dynamics (0–5 cm) and quantifying uncertainties efficiently based on the land surface wetness conditions. The newly developed approach adapts three different hydrological models (i.e., Noah Land Surface Model, Noah LSM; Soil‐Water‐Atmosphere‐Plant, SWAP; and Community Land Model, CLM) for simulating soil moisture. These models were integrated with a modified‐microGA (advanced version of original Genetic Algorithm (GA)) to search for optimized soil parameters for each model. Soil moisture was simulated from the estimated soil parameters using the hydrological models in a forward mode. It was found that SWAP performed better than others during wet condition, while Noah LSM and CLM showed a good agreement with measurements during dry condition. Thus, we inferred that performance of individual models with different model structures can be different with land surface wetness. Taking into account the effects of soil wetness on different model performances, we categorized soil moisture measurements and estimated different weights for each category using the BMA scheme. Effective surface soil moisture dynamics were obtained by aggregating multiple weighted soil moisture. Our findings demonstrated that the effective soil moisture estimates derived by this approach showed a better match with the measurements compared to the original models and single‐weighted outputs. Multimodel simulation approach based on land surface wetness enhances the ability to predict reliable soil moisture dynamics and reflects the strengths of different hydrological models under various soil wetness conditions.

Understanding and modelling low-flow regimes in ungauged basins remains a key challenge in hydrology and water resources management under a changing climate and anthropogenic pressures. Low-flow rivers, characterized by reduced streamflow over extended periods, play an essential role in ecosystem functioning and water availability. Often overlooked, they represent a substantial portion of the global river network, spanning a wide range of climatic regions, including arid, semi-arid, temperate, humid tropical, boreal, and alpine environments. The variability observed in these watercourses is shaped by regional physiographic and climatic factors, making their behaviour complex to model, especially in ungauged contexts.To address this limitation, we use SMASH (Spatially-distributed Modelling and ASsimilation for Hydrology), a modular and open-source platform developed at INRAE for distributed hydrological modelling. SMASH supports the combination of process-based conceptual structures and data-driven components such as neural networks, and allows spatially distributed calibration using multi-source hydrometeorological data. It is designed to simulate discharge hydrographs and hydrological states across all grid cells in a catchment, making it particularly suited for representing diverse hydrological responses, including low-flow regimes under varying physiographic and climatic conditions.This work aims to assess the performance of different hydrological model structures and regionalization strategies through large-scale calibration and validation experiments for simulating low-flow regimes. The assessment relies on a 40-year daily dataset covering 248 French catchments representative of diverse hydrometeorological conditions. Each model–regionalization configuration is assessed based on its ability to simulate low-flow dynamics, capture seasonal variability, and preserve overall water balance, with particular attention to its generalization capacity in ungauged contexts. In this framework, artificial neural networks (ANN) are applied to perform parameter regionalization based on physiographic attributes, as part of a data-driven strategy aimed at improving transferability to ungauged basins.Results will highlight the most effective combinations of model structure and regionalization method. This work aims to improve hydrological modelling in ungauged basins through hybrid approaches combining conceptual models with data-driven tools.