How good does Automatic Model Structure Identification work? A Benchmark Study with 6915 Model Structures.

<p>Recent studies have introduced methods to simultaneously calibrate model structure choices and parameter values to identify an appropriate (conceptual) model structure for a given catchment. This can be done through mixed-integer optimization to identify the graph structure that links dominant flow processes (Spieler et al., 2020) or, likewise, by continuous optimization of weights when blending multiple flux equations to describe flow processes within a model (Chlumsky et al., 2021). Here, we use the combination of the mixed-integer optimization algorithm DDS and the modular modelling framework RAVEN and refer to it as Automatic Model Structure Identification (AMSI) framework.</p><p>This study validates the AMSI framework by comparing the performance of the identified AMSI model structures to two different benchmark ensembles. The first ensemble consists of the best model structures from the brute force calibration of all possible structures included in the AMSI model space (7488+). The second ensemble consists of 35+ MARRMoT structures representing a structurally more divers set of models than currently implemented in the AMSI framework. These structures stem from the MARRMoT Toolbox introduced by Knoben et al. (2019) providing established conceptual model structures based on hydrologic literature.</p><p>We analyze if the model structure(s) AMSI identifies are identical to the best performing structures of the brute force calibration and comparable in their performance to the MARRMoT ensemble. We can conclude that model structures identified with the AMSI framework can compete with the structurally more divers MARRMoT ensemble. In fact, we were surprised to see how well we do with a simple two storage structure over the 12 tested MOPEX catchments (Duan et al.,2006). We aim to discuss several emerging questions, such as the selection of a robust model structure, Equifinality in model structures, and the role of structural complexity.</p><p> </p><p>Spieler et al. (2020). https://doi.org/10.1029/2019WR027009</p><p>Chlumsky et al. (2021). https://doi.org/10.1029/2020WR029229</p><p>Knoben et al. (2019). https://doi.org/10.5194/gmd-12-2463-2019</p><p>Duan et al. (2006). https://doi.org/10.1016/j.jhydrol.2005.07.031</p>

The problems of identifying the most appropriate model structure for a given problem and quantifying the uncertainty in model structure remain outstanding research challenges for the discipline of hydrology. Progress on these problems requires understanding of the nature of differences between models. This paper presents a methodology to diagnose differences in hydrological model structures: the Framework for Understanding Structural Errors (FUSE). FUSE was used to construct 79 unique model structures by combining components of 4 existing hydrological models. These new models were used to simulate streamflow in two of the basins used in the Model Parameter Estimation Experiment (MOPEX): the Guadalupe River (Texas) and the French Broad River (North Carolina). Results show that the new models produced simulations of streamflow that were at least as good as the simulations produced by the models that participated in the MOPEX experiment. Our initial application of the FUSE method for the Guadalupe River exposed relationships between model structure and model performance, suggesting that the choice of model structure is just as important as the choice of model parameters. However, further work is needed to evaluate model simulations using multiple criteria to diagnose the relative importance of model structural differences in various climate regimes and to assess the amount of independent information in each of the models. This work will be crucial to both identifying the most appropriate model structure for a given problem and quantifying the uncertainty in model structure. To facilitate research on these problems, the FORTRAN‐90 source code for FUSE is available upon request from the lead author.

<p>A recently introduced framework for Automatic Model Structure Identification (AMSI) allows to simultaneously optimize model structure choices (integer decision variables) and parameter values (continuous decision variables) in hydrologic modelling. By combining the mixed-integer optimization algorithm DDS and the flexible hydrologic modelling framework RAVEN, AMSI is able to test a vast number of model structure and parameter combinations in order to identify the most suitable model structure for representing the rainfall runoff behavior of a catchment. The model structure and all potentially active model parameters are calibrated simultaneously. This causes a certain degree of inefficiency during the calibration process, as variables might be perturbed that are not currently relevant for the tested model structure. In order to avoid this, we propose an adaption of the current DDS algorithm allowing for conditional parameter estimation. Parameters will only be perturbed during the calibration process if they are relevant for the model structure that is currently tested. The conditional parameter estimation setup will be compared to the standard DDS algorithm for multiple AMSI test cases. We will show if and how conditional parameter estimation increases the efficiency of AMSI.</p>

This study presents a goal programming approach to determine the best self-drive tourism route in Terengganu. The goal programming was formulated based on the model of Traveling Salesman Problem (TSP). This study involved seven cities in Terengganu which can be connected by road network. The data of the tourist attractions for each city was used in this study. The model of self-drive tourism route was constructed so as to connect all these seven cities by minimizing the travel distance, minimizing accommodation costs and maximizing the total number of tourist attractions. To achieve the objectives of this study, we develop the goal programming model that considers three model structures with each of them consists three goal constraints which are subjected to four hard constraints. This model has been solved using LINGO12.0 software. The results obtained from each model structures will be compared in order to get the most optimum route for self-drive tourism. The result showed that the first model structure was suitable to be used because this model structure gave the optimum result compared to other model structures. It produced the minimum total traveling cost of RM967.70 compared to the model structure two of RM977.10 and model structure three of RM1233.10.

Although the nonlinear power form model structure is widely accepted by practitioners in the flood regionalization modelling, there is a lack of studies on whether there is a room for further improvement, and if the answer is yes, what should be done to explore alternative model structures. A framework is proposed in this study towards investigating this issue by the following steps: (i) a universal data‐driven model is utilized to see if there is a room for improvement compared with the conventional model, and (ii) if improvement is achieved, this means that there should exist more effective model structures than the current form. However, because the universal data‐driven models are usually opaque, more explicit model structures should be explored, which are convenient for practical usage. In this study, the proposed framework is applied in a case study using the catchment characteristics from the Flood Estimation Handbook in conjunction with the gamma test, support vector machine (SVM) and genetic programming (GP). First, the gamma test is used for the purpose of input variables selection where no model structure needs to be defined as a priori, and therefore, the result can be applied to any model structures for model building. Second, an SVM, which is a powerful data‐driven nonlinear model capable of modelling a variety of nonlinear systems, is applied to the index flood model for the first time. Once the best model is determined using those two data‐driven tools, GP is employed to find an alternative model structure. As the SVM is not formulated for producing explicit model functional form, the GP offers an advantage at this point where it can infer an explicit mathematical model functional form. Copyright © 2012 John Wiley & Sons, Ltd.

Multi-model studies are widespread in large-sample hydrology. However, significant challenges remain in identifying interpretable connections between high-performing model structures and catchment characteristics, and thus in developing a coherent strategy for developing tailored multi-model ensembles. Here, we assess the importance of selecting model structures that are consistent with the expected hydrological variability across the study domain. We compare results of two modular modelling frameworks across 998 catchments in Great Britain. The RRMT framework includes model structures historically evolved in the UK, while the FUSE framework employs model structures from diverse global origins. While both groups of model structures contain high-performing members, the historically evolved group members separate between catchments in line with our expectation of hydrologic differences. We find that four hydrologic signatures organize these distinctions. Our results emphasize (1) the importance of model structure selection based on explicit perceptual models, and (2) the need to look beyond statistical performance alone.

Simulations of human behavior in water resources systems are challenged by uncertainty in model structure and parameters. The increasing availability of observations describing these systems provides the opportunity to infer a set of plausible model structures using data‐driven approaches. This study develops a three‐phase approach to the inference of model structures and parameterizations from data: problem definition, model generation, and model evaluation, illustrated on a case study of land use decisions in the Tulare Basin, California. We encode the generalized decision problem as an arbitrary mapping from a high‐dimensional data space to the action of interest and use multiobjective genetic programming to search over a family of functions that perform this mapping for both regression and classification tasks. To facilitate the discovery of models that are both realistic and interpretable, the algorithm selects model structures based on multiobjective optimization of (1) their performance on a training set and (2) complexity, measured by the number of variables, constants, and operations composing the model. After training, optimal model structures are further evaluated according to their ability to generalize to held‐out test data and clustered based on their performance, complexity, and generalization properties. Finally, we diagnose the causes of good and bad generalization by performing sensitivity analysis across model inputs and within model clusters. This study serves as a template to inform and automate the problem‐dependent task of constructing robust data‐driven model structures to describe human behavior in water resources systems.

The biological sequences that occur in DNA, RNA and proteins can be considered as strings formed over the well defined chemical alphabets. Such gene sequences form structure based on the complementary pair and the structures can be interpreted as languages. Matrix insertion-deletion system has been introduced a few years back that modelled several bio-molecular structures occur at intramolecular and intermolecular level. In this paper, we identify some structures that are frequently noticed during RNA folding process such as double bulge loop, extended internal loop, triple stem and loop and we give the corresponding formal language representation to such structures. Further, we model the structures using Matrix insertion-deletion systems. This work is pioneering to give the language representation and modelling the structures of RNA folding process using formal grammar.

Hydrological simulations in the Indian subcontinent are impacted by substantial uncertainty contributed by the selection of model structure and parameterization. Nevertheless, most studies in the Indian subcontinent have relied on a single model structure to simulate streamflow, without evaluating how such a choice might impact simulations in ungauged basins. In this study, we comprehensively evaluate the 277 gauged basins spanning across the diverse hydro-climatic regions of the Indian subcontinent using the Framework for Understanding Structural Errors (FUSE). FUSE allows testing different model structures within a controlled experimental framework, enabling the systematic evaluation of the impact of different model structures on streamflow simulations. For each gauged basin, we calibrate 78 FUSE structures and evaluate their performance with respect to the basic benchmarking models using the HydroBM python package and simulations from Noah-Multiparameterization Land Surface Model (Noah-MP LSM).For regionalization, we generate ensembles of 500 parameter sets for each selected decision structure using Latin Hypercube Sampling and propagated these through FUSE to characterize predictive uncertainty in ungauged basin simulations. These simulations are subsequently used to train surrogate emulators of the performance surface response, enabling efficient transfer of parameters from gauge to ungauged basins.Results indicate strong variability in predictive skill, uncertainty, and regionalization performance across model structures, highlighting that structures identified as optimal in calibrated basins do not necessarily generalize under parameter transfer. These findings underscore the need to consider structural benchmarking, uncertainty reliability, and regionalization performance when developing hydrological modelling frameworks for data-scarce regions such as the Indian subcontinent.

Adaptive control in an artificial pancreas for people with type 1 diabetes

Probabilistic logic programming offers a powerful way to describe and evaluate structured statistical models. To investigate the practicality of probabilistic logic programming for structure learning in bioinformatics, we undertook a simplified bacterial gene-finding benchmark in PRISM, a probabilistic dialect of Prolog. We evaluate Hidden Markov Model structures for bacterial protein-coding gene potential, including a simple null model structure, three structures based on existing bacterial gene finders and two novel model structures. We test standard versions as well as ADPH length modeling and three-state versions of the five model structures. The models are all represented as probabilistic logic programs and evaluated using the PRISM machine learning system in terms of statistical information criteria and gene-finding prediction accuracy, in two bacterial genomes. Neither of our implementations of the two currently most used model structures are best performing in terms of statistical information criteria or prediction performances, suggesting that better-fitting models might be achievable. The source code of all PRISM models, data and additional scripts are freely available for download at: http://github.com/somork/codonhmm. Supplementary data are available at Bioinformatics online.

In relevance logic it has become commonplace to associate with each logic both an algebraic counterpart and a relational counterpart. The former comes from the Lindenbaum construction; the latter, called a model structure, is designed for semantical purposes. Knowing that they are related through the logic, we may enquire after the algebraic relationship between the algebra and the model structure. This paper offers a complete solution for the relevance logic R⌝. Namely, R⌝-algebras and R⌝-model structures can be obtained from each other, and represented in terms of each other, by application of power constructions.

BackgroundIdentifying a proper model structure, using methods that address both structural and parameter uncertainty, is a crucial problem within the systems approach to biology. And yet, it has a marginal presence in the recent literature. While many existing approaches integrate methods for simulation and parameter estimation of a single model to address parameter uncertainty, only few of them address structural uncertainty at the same time. The methods for handling structure uncertainty often oversimplify the problem by allowing the human modeler to explicitly enumerate a relatively small number of alternative model structures. On the other hand, process-based modeling methods provide flexible modular formalisms for specifying large classes of plausible model structures, but their scope is limited to deterministic models. Here, we aim at extending the scope of process-based modeling methods to inductively learn stochastic models from knowledge and data.ResultsWe combine the flexibility of process-based modeling in terms of addressing structural uncertainty with the benefits of stochastic modeling. The proposed method combines search trough the space of plausible model structures, the parsimony principle and parameter estimation to identify a model with optimal structure and parameters. We illustrate the utility of the proposed method on four stochastic modeling tasks in two domains: gene regulatory networks and epidemiology. Within the first domain, using synthetically generated data, the method successfully recovers the structure and parameters of known regulatory networks from simulations. In the epidemiology domain, the method successfully reconstructs previously established models of epidemic outbreaks from real, sparse and noisy measurement data.ConclusionsThe method represents a unified approach to modeling dynamical systems that allows for flexible formalization of the space of candidate model structures, deterministic and stochastic interpretation of model dynamics, and automated induction of model structure and parameters from data. The method is able to reconstruct models of dynamical systems from synthetic and real data.

A progressive perceptual understanding approach was used to identify a model structure able to represent the non‐linear behaviour of the hydrological cycle in a small intermittent Mediterranean stream. The initial lumped model structure consisting of a series of four connected water tanks (LU3) progressed to a model with five tanks (LU4), and finally to a semidistributed model structure (SD4) in which spatial variability of the evapotranspiration according to the vegetation cover and to the local aspect was considered. In the final model structure, which gave the best fit (Nash–Sutcliffe efficiency index = 0·78), an additional tank representing the riparian zone was included (SD4‐R). Results showed that the abrupt changes of the riparian water table during summer and the formation of a perched water table during the transition from dry to wet conditions were the main mechanisms leading to the non‐linear hydrological behaviour. The transpiration process from the saturated zone and the spatial variability of evapotranspiration resulted in key factors successfully representing the annual water balance. The spatial and temporal validations carried out for each of the four model structures considered in this study supported the hypothesis adopted during the calibration process. Copyright © 2008 John Wiley & Sons, Ltd.

A metric for attributing variability in modelled streamflows