Improved Reachability Analysis in DTMC via Divide and Conquer

The prediction of numbers of newborns is an important issue in hospital management. Relying on the inherent non-aftereffect property, discrete-time Markov chain (DTMC) is a candidate for solving the problem. But the classical DTMC is unable to handle the uncertainty of states, especially when the state space is not discrete, which would lead to instable predicted results. In order to overcome the limitation of the existing DTMC model, a belief Markov chain (BMC) model is proposed by synthesizing the classical DTMC and Dempster-Shafer theory effectively. Depending on the advantages of Dempster-Shafer theory in expressing uncertainty, the proposed BMC model is capable of dealing with various uncertainties, which improves and perfects the classical DTMC model. An illustrative example demonstrates the effectiveness of the proposed model. Moreover, a comparison between the proposed BMC model and the classical and fuzzy states modified DTMC models is given to show the superiority of the proposed model against the other two. Finally, the stability of the proposed model has been proven.

ABSTRACTIn epidemic modelling, the emergence of a disease is characterized by the low numbers of infectious individuals. Environmental randomness impacts outcomes such as the outbreak or extinction of the disease in this case. This randomness can be accounted for by modelling the system as a continuous time Markov chain, . The probability of extinction given some initial state is the probability of hitting a subset of the state space associated with extinction for the initial state. This hitting probability can be studied by passing to the discrete time Markov chain (DTMC), . An approach is presented to approximate a DTMC on a countably infinite state space by a DTMC on a finite state space for the purpose of solving general hitting problems. This approach is applied to approximate the probability of disease extinction in an epidemic model. It is also applied to evaluate a heterogeneous disease control strategy in a metapopulation.

Markov decision processes (MDPs) are extensively used to model systems with both probabilistic and nondeterministic behavior. The problem of calculating the probability of reaching certain system states (hereafter reachability analysis) is central to the MDP-based system analysis. It is known that existing approaches on reachability analysis for MDPs are often inefficient when a given MDP contains a large number of states and loops, especially with the existence of multiple probability distributions. In this work, we propose a method to eliminate strongly connected components (SCCs) in an MDP using a divide-and-conquer algorithm, and actively remove redundant probability distributions in the MDP based on the convex property. With the removal of loops and parts of probability distributions, the probabilistic reachability analysis can be accelerated, as evidenced by our experiment results.KeywordsConvex HullModel CheckMarkov Decision ProcessDiscrete Time Markov ChainReachability AnalysisThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

dtsdPBC extends the well-known algebra of parallel processes, Petri box calculus (PBC), by incorporating discrete time stochastic and deterministic delays. To analyze performance in this extended calculus, the underlying semi-Markov chains, and the related (complete) and reduced discrete time Markov chains of the process expressions are built. The semi-Markov chains are extracted using the embedding method, which constructs the embedded discrete time Markov chains and calculates the sojourn time distributions in the states. The reductions of the discrete time Markov chains are obtained through the elimination method, which removes the vanishing states (those with zero sojourn times) and recalculates the transition probabilities among the tangible states (those with positive sojourn times). We prove that the reduced semi-Markov chain coincides with the reduced discrete time Markov chain, by demonstrating that an additional embedding into the reduced semi-Markov chain is needed for the reduced embedded discrete time Markov chain to match the embedded reduced discrete time Markov chain, and by comparing the respective sojourn times.

Goods Allocation by Queuing and the Occurrence of Violence: A Probabilistic Analysis

Model checking epistemic–probabilistic logic using probabilistic interpreted systems

An Introduction to Markov Chain Monte Carlo

Asymptotics of a matrix valued Markov chain arising in sociology

We study the nearest neighbors one dimensional uniform q-model of force fluctuations in bead packs [Coppersmith et al (1996)], a stochastic model to simulate the stress of granular media in two dimensional silos. The vertical coordinate plays the role of time, and the horizontal coordinate the role of space. The process is a discrete time Markov process with state space $\R^{\{1,...,N\}}$. At each layer (time), the weight supported by each grain is a random variable of mean one (its own weight) plus the sum of random fractions of the weights supported by the nearest neighboring grains at the previous layer. The fraction of the weight given to the right neighbor of the successive layer is a uniform random variable in $[0,1]$ independent of everything. The remaining weight is given to the left neighbor. In the boundaries, a uniform fraction of the weight leans on the wall of the silo. This corresponds to \emph{absorbing boundary conditions}. For this model we show that there exists a unique invariant measure. The mean weight at site $i$ under the invariant measure is $i(N+1-i)$; we prove that its variance is $\frac12(i(N+1-i))^2 + O(N^3)$ and the covariances between grains $i\neq j$ are of order $O(N^3)$. Moreover, as $N\to\infty$, the law under the invariant measure of the weights divided by $N^2$ around site (integer part of) $rN$, $r\in (0,1)$, converges to a product of gamma distributions with parameters 2 and $2(r(1-r))^{-1}$ (sum of two exponentials of mean $r(1-r)/2$). Liu {\it et al} (1995) proved that for a silo with infinitely many weightless grains, any product of gamma distributions with parameters 2 and $2/\rho$ with $\rho\in [0,\infty)$ are invariant. Our result shows that as the silo grows, the model selects exactly one of these Gamma's at each macroscopic place.

Evolution algebras are non-associative algebras. In this work we provide an extension of this class of algebras, in a framework of Hilbert spaces, and illustrate the applicability of our approach by discussing a connection with discrete-time Markov chains with infinite countable state space. Specifically, if we associate to each possible state of such a Markov process a generator of the Hilbert evolution algebra structure, then the whole dynamics of the process can be described through consecutive applications of the evolution operator, provided certain boundedness conditions on the transition probabilities hold.

A new theoretical analysis approach for a multi-agent spatial Parrondo’s game

Modeling of rate-based congestion control schemes in cognitive radio sensor networks

We study time-bounded probabilistic reachability for Chemical Reaction Networks (CRNs) using the Linear Noise Approximation (LNA). The LNA approximates the discrete stochastic semantics of a CRN in terms of a continuous space Gaussian process. We consider reachability regions expressed as intersections of finitely many linear inequalities over the species of a CRN. This restriction allows us to derive an abstraction of the original Gaussian process as a time-inhomogeneous discrete-time Markov chain (DTMC), such that the dimensionality of its state space is independent of the number of species of the CRN, ameliorating the state space explosion problem. We formulate an algorithm for approximate computation of time-bounded reachability probabilities on the resulting DTMC and show how to extend it to more complex temporal properties. We implement the algorithm and demonstrate on two case studies that it permits fast and scalable computation of reachability properties with controlled accuracy.

Modeling and analysis for multi-state systems with discrete-time Markov regime-switching

We propose a distance between continuous-time Markov chains (CTMCs) and study the problem of computing it by comparing three different algorithmic methodologies: iterative, linear program, and on-the-fly. In a work presented at FoSSaCS'12, Chen et al. characterized the bisimilarity distance of Desharnais et al. between discrete-time Markov chains as an optimal solution of a linear program that can be solved by using the ellipsoid method. Inspired by their result, we propose a novel linear program characterization to compute the distance in the continuous-time setting. Differently from previous proposals, ours has a number of constraints that is bounded by a polynomial in the size of the CTMC. This, in particular, proves that the distance we propose can be computed in polynomial time. Despite its theoretical importance, the proposed linear program characterization turns out to be inefficient in practice. Nevertheless, driven by the encouraging results of our previous work presented at TACAS'13, we propose an efficient on-the-fly algorithm, which, unlike the other mentioned solutions, computes the distances between two given states avoiding an exhaustive exploration of the state space. This technique works by successively refining over-approximations of the target distances using a greedy strategy, which ensures that the state space is further explored only when the current approximations are improved. Tests performed on a consistent set of (pseudo)randomly generated CTMCs show that our algorithm improves, on average, the efficiency of the corresponding iterative and linear program methods with orders of magnitude.