SIMULATION FROM ENDPOINT-CONDITIONED, CONTINUOUS-TIME MARKOV CHAINS ON A FINITE STATE SPACE, WITH APPLICATIONS TO MOLECULAR EVOLUTION.

In the previous chapter, we studied a discrete-time stochastic process {X n, n ≥ 0} on finite state space with Markov property at times n = 0, 1, 2 ···. Now we would like to study a continuous-time stochastic process {X(t), t ≥ 0} on a finite state space with Markov property at each time t ≥ 0. We shall call such a process continuous-time Markov Chain (CTMC). We shall see in the next chapter that a finite-state CTMC spends an exponentially distributed amount of time in a given state before jumping out of it. Thus exponential distributions play an important role in CTMCs. In addition, the Poisson distribution and Poisson processes also form the foundation of many CTMC models. Hence we study these topics in this chapter.KeywordsPoisson ProcessComputational ProblemCompound Poisson ProcessExponential Random VariablePoisson Random VariableThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

In this paper, we present several novel results on controllability and stabilizability properties of the Kolmogorov forward equation of a continuous time Markov chain (CTMC) evolving on a finite state space, using the transition rates as the control parameters. First, we characterize all the stationary distributions that are stabilizable using time-independent control parameters. We then present a result on small-time local and global controllability of the system from and to strictly positive equilibrium distributions when the underlying graph is strongly connected. Additionally, we show that any target distribution can be reached asymptotically using time-varying control parameters. For bidirected graphs, we construct rational and polynomial density feedback laws that stabilize strictly positive stationary distributions while satisfying the additional constraint that that the feedback law takes zero value at equilibrium. This last result enables the construction of decentralized density feedback controllers, using tools from linear systems theory and sum-of-squares based polynomial optimization, that stabilize a swarm of agents modeled as a CTMC to a target state distribution with no state-switching at equilibrium. We validate the effectiveness of the constructed feedback laws with stochastic simulations of the CTMC for finite numbers of agents and numerical solutions of the corresponding mean-field models.

This review article is concerned to provide a global context to several works on the fitting of continuous time nonhomogeneous Markov chains with finite state space and also to point out some selected aspects of two techniques previously introduced—estimation and calibration—relevant for applications and used to fit a continuous time Markov chain model to data by the adequate selection of parameters. The denomination estimation suits the procedure better when statistical techniques—e.g., maximum likelihood estimators—are employed, while calibration covers the case where, for instance, some optimisation technique finds a best approximation parameter to ensure good model fitting. For completeness, we provide a short summary of well-known important notions and results formulated for nonhomogeneous Markov chains that, in general, can be transferred to the homogeneous case. Then, as an illustration for the homogeneous case, we present a selected Billingsley’s result on parameter estimation for irreducible chains with finite state space. In the nonhomogeneous case, we quote two recent results, one of the calibration type and the other with more of a statistical flavour. We provide an ample set of bibliographic references so that the reader wanting to pursue her/his studies will be able to do so more easily and productively.

We study the Classical Probability analogue of the unitary dilations of a quantum dynamical semigroup defined in Quantum Probability via quantum stochastic differential equations. Given a homogeneous Markov chain in continuous time in a finite state space E, we introduce a second system, an environment, and a deterministic invertible time-homogeneous global evolution of the system E with this environment such that the original Markov evolution of E can be realized by a proper choice of the initial random state of the environment. We also compare this dilations with the unitary dilations of a quantum dynamical semigroup in Quantum Probability.

Mathematically, stochastic hybrid systems can be described by Markov processes with discrete and continuous behaviours. This chapter provides in an incremental way the necessary mathematical background for understanding such complex systems. The chapter starts with discrete and continuous-time Markov chains. In this book, a Markov chain is understood as a stochastic process with the Markov property (or memoryless property, i.e. its future evolution depends only on the current state) defined on a discrete (finite or countable) state space. The Markov property is illustrated in the Chapman-Kolmogorov equation satisfied by the transition probabilities. Usually, in the continuous-time Markov case, the evolution of these transition probabilities is described by forward/backward Kolmogorov equations. These equations are expressed in terms of the stochastic matrix (called also infinitesimal generator) associated to the Markov chain. This is the matrix of transition rates and the practical use of Markov chains resides in our ability of handling it.The second part of this chapter is dedicated to Markov processes defined on continuous state spaces that can be equipped with additional structures as sigma-algebras, metrics, topologies, norms or others like specific algebraic structures (lattice, vector space). Characterisations of such processes are usually given using functional analysis operators like: the operator semigroup/resolvent, the infinitesimal generator or other operators that can be associated to a Markov process. At an abstract level, these operators describe the evolution the evolution of transition probabilities for a Markov process. Chapman-Kolmogorov equation, Kolmogorov equations and other properties of these operators (e.g. Dynkin formula, martingale problem) represent the natural tools for studying such Markov processes.KeywordsMarkov ChainTopological SpaceMarkov ProcessMarkov PropertyPolish SpaceThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Robust μ-stability analysis of Markovian switching uncertain stochastic genetic regulatory networks with unbounded time-varying delays

Tandem queues are good mathematical models for description of information transmission in various communication systems and networks. These queues play also an important role for the validation of different decomposition algorithms designed for investigating more general queueing networks. So, their investigation is interesting for theory and applications. In this paper, we consider tandem queue suitable for modeling the systems and networks where information flows are correlated and bursty what is typical for many modern telecommunication networks. Possible correlation of customers inter-arrival times and batch arrivals are taken into account via of consideration of the Batch Markovian Arrival Process (BMAP) as input stream to the system. The system consists of two stations. The service time at the station 1 is assumed to be generally distributed. There is no buffer at this station, and customers who meet the busy server repeat attempts to enter the system in random time intervals. The service process at the station 2 is assumed to be described by the continuous time Markov chain with a finite state space. This assumption holds good, e.g., if the station 2 has a finite buffer, consists of a finite number of identical or heterogeneous servers where the service time distribution is assumed to be of PH (PHase) type. Markov chain embedded at service completion epochs at the station 1 and the process of system states at arbitrary time are under study. Ergodicity condition and algorithms for computing the steady state probabilities are presented.

Convex Stochastic Fluid Programs with Average Cost

In this work, we establish a framework for balanced realization of linear systems subject to regime switching modulated by a continuous-time Markov chain with a finite state space. First, a definition of balanced realization is given. Then a ρ-balanced realization is developed to approximate the system of balancing equations, which is a system of time-varying algebraic equations. When the state space of the Markov chain is large, the computational effort becomes a real concern. To resolve this problem, we introduce a two-time-scale formulation and use decomposition/aggregation and averaging techniques to reduce the computational complexity. Based on the two-time-scale formulation, further approximation procedures are developed. Numerical examples are also presented for demonstration.

The gating mechanism of a single ion channel is usually modelled by a continuous-time Markov chain with a finite state space, partitioned into two classes termed ‘open’ and ‘closed’. It is possible to observe only which class the process is in. A burst of channel openings is defined to be a succession of open sojourns separated by closed sojourns all having duration less than t 0 . Let N(t) be the number of bursts commencing in (0, t]. Then are measures of the degree of temporal clustering of bursts. We develop two methods for determining the above measures. The first method uses an embedded Markov renewal process and remains valid when the underlying channel process is semi-Markov and/or brief sojourns in either the open or closed classes of state are undetected. The second method uses a ‘backward’ differential-difference equation. The observed channel process when brief sojourns are undetected can be modelled by an embedded Markov renewal process, whose kernel is shown, by exploiting connections with bursts when all sojourns are detected, to satisfy a differential-difference equation. This permits a unified derivation of both exact and approximate expressions for the kernel, and leads to a thorough asymptotic analysis of the kernel as the length of undetected sojourns tends to zero.

This paper is concerned with a class of hybrid stock market models, in which both the return rate and the volatility depend on a hidden, continuous-time Markov chain with a finite state space. One of the crucial issues is to estimate the generator of the underlying Markov chain. We develop a stochastic optimization procedure for this task, prove its convergence, and establish the rate of convergence. Numerical tests are carried out via simulation as well as using real market data. In addition, we demonstrate how to use the estimated generator in making stock liquidation decisions.

Fluid queues to solve jump processes

For discrete-time stochastic processes, there is a close connection between return (resp. waiting) times and entropy (resp. relative entropy). Such a connection cannot be straightforwardly extended to the continuous-time setting. Contrarily to the discrete-time case one needs a reference measure on path space and so the natural object is relative entropy rather than entropy. In this paper we elaborate on this in the case of continuous-time Markov processes with finite state space. A reference measure of special interest is the one associated to the time-reversed process. In that case relative entropy is interpreted as the entropy production rate. The main results of this paper are: almost-sure convergence to relative entropy of the logarithm of waiting-times ratios suitably normalized, and their fluctuation properties (central limit theorem and large deviation principle).

Taeniasis and cysticercosis pose a health concern on both humans and animals, as well as the economy of livestock farmers in rural areas. This study examines cysticercosis and taeniasis transmission dynamics in human, pig and cattle populations. Both deterministic and continuous time Markov chain (CTMC) stochastic approaches are used. For deterministic and CTMC stochastic models, we used the next generation approach and the multitype branching process respectively to calculate the basic reproduction number and the stochastic threshold. The potential probability of cysticercosis and taeniasis extinction is computed through numerical simulations for the CTMC model using 10,000 sample paths and altering the initial values for classes that are infected. The findings demonstrate that when diseases’ outbreak occur, the CTMC stochastic model’s solutions resemble those of deterministic model quite closely. The findings also suggest that the likelihood of diseases’ extinction is high if they develop from a small number of taenia eggs. If the infections, however, emerge from humans with cysticercosis, they will perish. If the infections arise from either infected beef and pork or humans with taeniasis, there is a significant diseases’ outbreak in the human, pig and cattle populations. Therefore, at the beginning of a diseases’ outbreak, management strategies that concentrate on reducing taeniasis-infected individuals and consumption of infectious beef and pork can help in regulating the transmission of the diseases in humans, pigs, and cattle.

We consider continuous and impulse control of a Markov chain (MC) with a finite set of states in continuous time. Continuous control determines the intensity of transitions between MC states, while transition times and their directions are random. Nevertheless, sometimes it is necessary to ensure a transition that leads to an instantaneous change in the state of the MC. Since such transitions require different influences and can produce different effects on the state of the MC, such controls can be interpreted as impulse controls. In this work, we use the martingale representation of a controllable MC and give an optimality condition, which, using the principle of dynamic programming, is reduced to a form of quasi-variational inequality. The solution to this inequality can be obtained in the form of a dynamic programming equation, which for an MC with a finite set of states reduces to a system of ordinary differential equations with one switching line. We prove a sufficient optimality condition and give examples of problems with deterministic and random impulse action.