Laplace based Bayesian inference for ordinary differential equation models using regularized artificial neural networks

Mathematical models are routinely calibrated to experimental data, with goals ranging from building predictive models to quantifying parameters that cannot be measured. Whether or not reliable parameter estimates are obtainable from the available data can easily be overlooked. Such issues of parameter identifiability have important ramifications for both the predictive power of a model, and the mechanistic insight that can be obtained. Identifiability analysis is well-established for deterministic, ordinary differential equation (ODE) models, but there are no commonly adopted methods for analysing identifiability in stochastic models. We provide an accessible introduction to identifiability analysis and demonstrate how existing ideas for analysis of ODE models can be applied to stochastic differential equation (SDE) models through four practical case studies. To assess structural identifiability, we study ODEs that describe the statistical moments of the stochastic process using open-source software tools. Using practically motivated synthetic data and Markov chain Monte Carlo methods, we assess parameter identifiability in the context of available data. Our analysis shows that SDE models can often extract more information about parameters than deterministic descriptions. All code used to perform the analysis is available on Github.

Experimental results have shown that anti-cancer therapies, such as radiotherapy and chemotherapy, can modulate the cell cycle and generate cell cycle phase-dependent responses. As a result, obtaining a detailed understanding of the cell cycle is one possible path towards improving the efficacy of many of these therapies. Here, we consider a basic structured partial differential equation (PDE) model for cell progression through the cell cycle, and derive expressions for key quantities, such as the population growth rate and cell phase proportions. These quantities are shown to be periodic and, as such, we compare the PDE model to a corresponding ordinary differential equation (ODE) model in which the parameters are linked by ensuring that the long-term ODE behaviour agrees with the average PDE behaviour. By design, we find that the ODE model does an excellent job of representing the mean dynamics of the PDE model within just a few cell cycles. However, by probing the parameter space we find cases in which this mean behaviour is not a good measure of the PDE population growth. Our analytical comparison of two caricature models (one PDE and one ODE system) provides insight into cases in which the simple ODE model is an appropriate approximation to the PDE model.

Bayesian inference for higher-order ordinary differential equation models

Identification of contaminant sources in rivers is crucial for river protection and emergency response. This study presents an innovative approach for identifying river pollution sources by using Bayesian inference and cellular automata (CA) modeling. A general Bayesian framework is proposed that combines the CA model with observed data to identify unknown sources of river pollution. To reduce the computational burden of the Bayesian inference, a CA contaminant transport model is developed to efficiently simulate pollutant concentration values in the river. These simulated concentration values are then used to calculate the likelihood function of available measurements. The Markov chain Monte Carlo (MCMC) method is used to produce the posterior distribution of contaminant source parameters, which is a sampling-based method that enables the estimation of complex posterior distributions. The suggested methodology is applied to a real case study of the Fen River in Yuncheng City, Shanxi Province, Northern China, and it estimates the release time, release mass, and source location with relative errors below 19%. The research indicates that the proposed methodology is an effective and flexible way to identify the location and concentrations of river contaminant sources.

Modeling and analysis of complex systems are important aspects of understanding systemic behavior. In the lack of detailed knowledge about a system, we often choose modeling equations out of convenience and search the (high-dimensional) parameter space randomly to learn about model properties. Qualitative modeling sidesteps the issue of choosing specific modeling equations and frees the inference from specific properties of the equations. We consider classes of ordinary differential equation (ODE) models arising from interactions of species/entities, such as (bio)chemical reaction networks or ecosystems. A class is defined by imposing mild assumptions on the interaction rates. In this framework, we investigate whether there can be multiple positive steady states in some ODE models in a given class. We have developed and implemented a method to decide whether any ODE model in a given class cannot have multiple steady states. The method runs efficiently on models of moderate size. We tested the method on a large set of models for gene silencing by sRNA interference and on two publicly available databases of biological models, KEGG and Biomodels. We recommend that this method is used as (i) a pre-screening step for selecting an appropriate model and (ii) for investigating the robustness of non-existence of multiple steady state for a given ODE model with respect to variation in interaction rates. Scripts and examples in Maple are available in the Supplementary Information. wiuf@math.ku.dk Supplementary data are available at Bioinformatics online.

For system identification, the ordinary differential equations (ODEs) model is popular for its accuracy and effectiveness. Consequently, the ODEs model is extended to the stochastic differential equations (SDEs) model to tackle the stochastic case intuitively. But the existence of stochastic integral is a rigid barrier. We simply transform the SDEs to their corresponding stochastic difference equations (SDCEs) to eliminate stochastic integrals and propose an easy but effective solution to stochastic system identification. In this solution, the maximum likelihood estimation can be applied and the evolutionary algorithms are used to determine structures and parameters of the unknown system.

Ordinary differential equation (ODE) models are quite popular for modelling complex dynamic processes in many scientific fields, and the parameters in these models are usually unknown, and we need to estimate them using statistical methods. When some observations are contaminated, regular estimation methods, such as nonlinear least-square estimation, will bring large bias. In this paper, robust estimations of both constant and time-varying parameters in ODE models using M-estimators are proposed, and their asymptotic properties are obtained under some mild conditions. We focus on Huber M-estimator, and also provide a method to adjust the Huber parameter automatically to the observations. The proposed method is compared to existing methods in numerical simulations and CD8+ T cell data analysis. It is demonstrated that our method gain substantial efficiency as well as robust properties.

US high school (HS) students are lagging behind their foreign peers in math performance. It is thus necessary to design teaching modules that motivate US HS students to learn more math and to improve their math skills. Our study set out to test two different software programs designed for the development of such skills. MATLAB Simulink provides a friendly interface with which HS students can build ordinary differential equation (ODE) models. An alternative programming language, the R language is free and extensively used in the field of statistical learning and engineering. In this study, a group of HS students first got trained in both Simulink and R and then solved an ODE model of a microbial fuel cell (MFC), which can convert the organic compounds found in waste water into electricity, using both Simulink and R. Three anonymous surveys were conducted for: 1) students’ feedback on the module-based teaching approaches with R and Simulink training, 2) students’ self-evaluation in the improvement of their skills in solving ODE models following their training in R and Simulink, and 3) students’ comparison of R and Simulink in their learning and modeling experience. The instructor also evaluated students’ modeling skills with R and Simulink based on their performance in developing the MFC ODE model. Both students’ self-evaluation and the instructor’s direct evaluation indicate that students’ skills in solving ODE models with both R and Simulink were significantly improved after this R-Simulink modeling project. The surveys show that students generally enjoyed the module-based teaching format and that they would recommended this project to their peers. It was shown in the surveys that students preferred Simulink over R in solving ODE models and that they preferred learning Simulink first and then following up by learning R. Students’ comments for opening questions indicate that the major hurdle in this project was their limited background knowledge in biology and electrochemistry for microbial fuel cells.

Vector-borne diseases provide unique challenges to public health because the epidemiology is so closely tied to external environmental factors such as climate, landscape, and population migration, as well as the complicated biology of vector-transmitted pathogens. In particular, this close link between the epidemiology, the environment, and pathogen biology means that the traditional view that many vector-borne diseases are relatively stable in numerous regions does not provide a complete picture of their complexity. In fact, several regions exist with low levels of endemicity most of the time, punctuated by severe, often explosive, epidemics. These regions are considered unstable transmission settings. Ordinary differential equation (ODE) models have thus far dominated the study of vector-borne disease and have provided considerable insight into our understanding of transmission and effective control in stable transmission settings. To address the short-­comings of autonomous ODE models, we present a class of models, differential-delay equation (DDE) models, that have the potential to better describe unstable endemic settings for vector-borne disease. These models develop naturally out of the biology of diseases transmitted by vectors because of the extrinsic and intrinsic incubation periods and vector maturation process necessary for successful transmission of vector-transmitted pathogens. In this chapter, we introduce five examples of vector-borne diseases that span the globe, and discuss the clinical implications of unstable transmission of these diseases. Next, we present the original ODE version of the Ross-Macdonald model for vector-borne diseases, modify this model by introducing different types of naturally occurring delays, then illustrate how these models can exhibit more complex behavior such as oscillations via Hopf bifurcation and chaos via period-doubling, that the ODE model cannot produce. Finally, we explore the possibility for delay-models to contribute to our understanding of unstable transmission settings, which in turn will inform the development of effective control strategies for these epidemic-prone regions.

A common approach to understand and analyze complex biological systems is to describe the dynamics in terms of a system of ordinary differential equations (ODE) depending on numerous biologically meaningful and descriptive parameters that are estimated using observed data. The ODE models are often based on the implicit assumption of well-mixed dynamics, i.e., the delay of interaction due to spatial distribution is not included in the model. In this article, we address the question how the heterogeneity of the underlying system affects the estimated parameter values of the ODE model, and on the other hand, what information about the microscopic system can be drawn from these values. The system we are considering is a pairwise growth competition assay used to quantify ex vivo replicative fitness of different HIV-1 isolates. To overcome the lack of ground truth, we generate data using a detailed microscopic spatially distributed hybrid stochastic-deterministic (HSD) infection model in which the dynamics is controlled by parameters directly related to cell level infection, virus production processes, and diffusion of virus particles. The synthetic data sets are then analyzed using an ODE based well-mixed model, in which the corresponding macroscopic parameter distributions are estimated using Markov chain Monte Carlo (MCMC) methods. This approach provides a comprehensive picture of the statistical dependencies of the model parameter across different scales.

Traditional full-waveform inversion (FWI) methods only render a “best-fit” model that cannot account for uncertainties of the ill-posed inverse problem. Additionally, local optimization-based FWI methods cannot always converge to a geologically meaningful solution unless the inversion starts with an accurate background model. We seek the solution for FWI in the Bayesian inference framework to address those two issues. In Bayesian inference, the model space is directly probed by sampling methods such that we obtain a reliable uncertainty appraisal, determine optimal models, and avoid entrapment in a small local region of the model space. The solution of such a statistical inverse method is completely described by the posterior distribution, which quantifies the distributions for parameters and inversion uncertainties. To efficiently sample the posterior distribution, we introduce a sampling algorithm in which the proposal distribution is constructed by the local gradient and the diagonal approximate Hessian of the local log posterior. Our algorithm is called the gradient-based Markov chain Monte Carlo (GMCMC) method. The GMCMC FWI method can quantify inversion uncertainties with estimated posterior distribution given sufficiently long Markov chains. By directly sampling the posterior distribution, we obtain a global view of the model space. Theoretically speaking, statistical assessments do not depend on starting models. Our method is applied to the 2D Marmousi model with the frequency-domain FWI setting. Numerical results suggest that our method can be readily applied to 2D cases with affordable computational efforts.

We find concrete evidence for a recently discovered form of intermittency, referred to as in–out intermittency, in both partial differential equation (PDE) and ordinary differential equation (ODE) models of mean field dynamos. This type of intermittency [introduced in P. Ashwin, E. Covas, and R. Tavakol, Nonlinearity 9, 563 (1999)] occurs in systems with invariant submanifolds and, as opposed to on–off intermittency which can also occur in skew product systems, it requires an absence of skew product structure. By this we mean that the dynamics on the attractor intermittent to the invariant manifold cannot be expressed simply as the dynamics on the invariant subspace forcing the transverse dynamics; the transverse dynamics will alter that tangential to the invariant subspace when one is far enough away from the invariant manifold. Since general systems with invariant submanifolds are not likely to have skew product structure, this type of behavior may be of physical relevance in a variety of dynamical settings. The models employed here to demonstrate in–out intermittency are axisymmetric mean-field dynamo models which are often used to study the observed large-scale magnetic variability in the Sun and solar-type stars. The occurrence of this type of intermittency in such models may be of interest in understanding some aspects of such variabilities.

The mixture cure model for analyzing survival data is characterized by the assumption that the population under study is divided into a group of subjects who will experience the event of interest over some finite time horizon and another group of cured subjects who will never experience the event irrespective of the duration of follow‐up. When using the Bayesian paradigm for inference in survival models with a cure fraction, it is common practice to rely on Markov chain Monte Carlo (MCMC) methods to sample from posterior distributions. Although computationally feasible, the iterative nature of MCMC often implies long sampling times to explore the target space with chains that may suffer from slow convergence and poor mixing. Furthermore, extra efforts have to be invested in diagnostic checks to monitor the reliability of the generated posterior samples. A sampling‐free strategy for fast and flexible Bayesian inference in the mixture cure model is suggested in this article by combining Laplace approximations and penalized B‐splines. A logistic regression model is assumed for the cure proportion and a Cox proportional hazards model with a P‐spline approximated baseline hazard is used to specify the conditional survival function of susceptible subjects. Laplace approximations to the posterior conditional latent vector are based on analytical formulas for the gradient and Hessian of the log‐likelihood, resulting in a substantial speed‐up in approximating posterior distributions. The spline specification yields smooth estimates of survival curves and functions of latent variables together with their associated credible interval are estimated in seconds. A fully stochastic algorithm based on a Metropolis‐Langevin‐within‐Gibbs sampler is also suggested as an alternative to the proposed Laplacian‐P‐splines mixture cure (LPSMC) methodology. The statistical performance and computational efficiency of LPSMC is assessed in a simulation study. Results show that LPSMC is an appealing alternative to MCMC for approximate Bayesian inference in standard mixture cure models. Finally, the novel LPSMC approach is illustrated on three applications involving real survival data.

Combined network and mathematical modeling suggests novel inflammation positive feedback circuit and role for IL-1α in experimental trauma/hemorrhage and hepatocyte hypoxia

We study the presence of social communities in mobility traces from vehicular fleets. By analyzing publicly available sets of fleet vehicle mobility traces obtained from two real-world deployments — consisting of more than 2000 taxis in Shanghai and Beijing respectively, we confirm the existence of small numbers of distinct social communities in vehicular networks, which is in direct contrast to the general belief that vehicular networks are best modeled as a relatively homogeneous system. We examine the spatio-temporal characteristics of social communities, gaining the insight that they are driven primarily by social proximity induced by geographic locality. We then develop a parsimonious multi-community ordinary differential equation (ODE) model, which uses the heterogeneous structure introduced by social communities to model information dissemination. We show through simulations that this approach dramatically outperforms the conventional homogeneous ODE model in capturing the dynamics of the dissemination process. We further demonstrate that the use of the ODE model to optimize seeding of an initial set of vehicles results in improved utility for information dissemination compared to seed-optimization using a homogeneous model.