Mean-field controllability and decentralized stabilization of Markov chains

We present in this paper a mathematical method based on stochastic comparisons of Markov chains in order to compute in a mobile network performance measures that are important Q parameters for users: the dropping handover of voice and the blocking probability of a new voice call. The key idea of this methodology is that given a complex system represented by a multi-dimensional Markov chain which is too large to be solved, we propose to reduce the state space, and so to define a new Markov chain which is a simplified version of the original one. This reduced Markov chain is defined as an aggregated one, which represents a stochastic bound for performance measures written as increasing reward function on the stationary distribution. The main steps of the construction of the aggregated Markov chain applied into mobile networks are presented in this paper. As the number of mobile users with different kinds of applications increases, the associated model is more complex and it can be represented by multi-dimensional continuous time Markov chain with a very large size. Thus, we define an aggregated Markov chain represented by a multi-dimensional birth and death process which is very easy to solve. We have proved that there is a weak ordering between the original Markov chain and the aggregated one using increasing set formalism. We have computed upper bounds of dropping handover and blocking probability for different values of input parameters. Numerical results prove that upper bounds give good results and so the stochastic methodology is an interesting mathematical tool for the performance evaluation of complex systems. Keywords: Mobile Networks, Quality of Service, Stochastic ordering, stochastic comparisons, Continuous Time Markov Chains.

Analyses of serially-sampled data often begin with the assumption that the observations represent discrete samples from a latent continuous-time stochastic process. The continuous-time Markov chain (CTMC) is one such generative model whose popularity extends to a variety of disciplines ranging from computational finance to human genetics and genomics. A common theme among these diverse applications is the need to simulate sample paths of a CTMC conditional on realized data that is discretely observed. Here we present a general solution to this sampling problem when the CTMC is defined on a discrete and finite state space. Specifically, we consider the generation of sample paths, including intermediate states and times of transition, from a CTMC whose beginning and ending states are known across a time interval of length T. We first unify the literature through a discussion of the three predominant approaches: (1) modified rejection sampling, (2) direct sampling, and (3) uniformization. We then give analytical results for the complexity and efficiency of each method in terms of the instantaneous transition rate matrix Q of the CTMC, its beginning and ending states, and the length of sampling time T. In doing so, we show that no method dominates the others across all model specifications, and we give explicit proof of which method prevails for any given Q, T, and endpoints. Finally, we introduce and compare three applications of CTMCs to demonstrate the pitfalls of choosing an inefficient sampler.

This review article is concerned to provide a global context to several works on the fitting of continuous time nonhomogeneous Markov chains with finite state space and also to point out some selected aspects of two techniques previously introduced—estimation and calibration—relevant for applications and used to fit a continuous time Markov chain model to data by the adequate selection of parameters. The denomination estimation suits the procedure better when statistical techniques—e.g., maximum likelihood estimators—are employed, while calibration covers the case where, for instance, some optimisation technique finds a best approximation parameter to ensure good model fitting. For completeness, we provide a short summary of well-known important notions and results formulated for nonhomogeneous Markov chains that, in general, can be transferred to the homogeneous case. Then, as an illustration for the homogeneous case, we present a selected Billingsley’s result on parameter estimation for irreducible chains with finite state space. In the nonhomogeneous case, we quote two recent results, one of the calibration type and the other with more of a statistical flavour. We provide an ample set of bibliographic references so that the reader wanting to pursue her/his studies will be able to do so more easily and productively.

In the previous chapter, we studied a discrete-time stochastic process {X n, n ≥ 0} on finite state space with Markov property at times n = 0, 1, 2 ···. Now we would like to study a continuous-time stochastic process {X(t), t ≥ 0} on a finite state space with Markov property at each time t ≥ 0. We shall call such a process continuous-time Markov Chain (CTMC). We shall see in the next chapter that a finite-state CTMC spends an exponentially distributed amount of time in a given state before jumping out of it. Thus exponential distributions play an important role in CTMCs. In addition, the Poisson distribution and Poisson processes also form the foundation of many CTMC models. Hence we study these topics in this chapter.KeywordsPoisson ProcessComputational ProblemCompound Poisson ProcessExponential Random VariablePoisson Random VariableThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

The use of probabilistic ideas by applied mathematicians has seen a continued increase in recent decades. Probability now appears frequently at the modeling stage. There is widespread interest in investigating the effects of noise and uncertainty. Probabilistic algorithms are routinely applied with much success to the solution of deterministic problems (think of Monte Carlo quadrature or stochastic gradient descent). Markov chains give a simple, widely used tool to describe systems that evolve randomly. They are also a very popular topic in applied mathematics courses. At times \(0\), \(1\), \(2\), łdots, a Markov chain jumps from its current state \(x\) to a randomly chosen new state \(y\); the set of possible states is discrete. The requirement that the jumping times be uniformly spaced is a clear limitation in many situations, and this shortcoming is avoided by considering continuous-time Markov chains. In the continuous-time setting, if the chain has reached state \(x\) at time \(t_i\), it will jump to the randomly chosen \(y\) at time \(t_i+1=t_i+\tau_i\), where the waiting time \(\tau_i>0\) is itself random. Markov chains in continuous time are featured in many applications, including chemistry, ecology, and epidemiology. In a chemical system containing \(n\) species \(S_1\), łdots, \(S_n\), each state corresponds to vector \((c_1\), łdots, \( c_n)\) where \(c_j\) is the number of molecules of species \(S_j\). The species may take part in a number of chemical reactions, let us say \(S_1+2S_2 \rightarrow S_3\), \(2S_1+3S_3 \rightarrow S_4+2S_6\), and so on. At random times, one or another of the \(m\) possible reactions will take place and the state will change. Such a stochastic, molecular approach typically provides a more accurate description of the system than deterministic treatments where the concentrations of the different species are regarded as continuous variables that evolve according to a set of differential equations. This is particularly true in systems where, for some species, the number \(c_j\) is low, as is the case in many biological reactions. The following Survey and Review paper, “Stationary Distributions of Continuous-Time Markov Chains: A Review of Theory and Truncation-Based Approximations,” has been written by Juan Kuntz, Philipp Thomas, Guy-Bart Stan, and Mauricio Barahona. Section 2 provides a reader-friendly introduction to continuous-time Markov chains and their stationary distributions; these are important because they determine the long-time behavior of the chain. A salient future is that the authors present, in an accessible way, results that do not assume the chain to be irreducible (roughly speaking, irreducibility means that the chain is not the juxtaposition of two or more smaller chains that do not talk to each other; irreducibility simplifies the math, but is not a reasonable hypothesis in some applications). After that, the authors concentrate on how to compute invariant distributions. The paper contains numerical results, an extensive bibliography, and a detailed discussion of open problems.

Tandem queues are good mathematical models for description of information transmission in various communication systems and networks. These queues play also an important role for the validation of different decomposition algorithms designed for investigating more general queueing networks. So, their investigation is interesting for theory and applications. In this paper, we consider tandem queue suitable for modeling the systems and networks where information flows are correlated and bursty what is typical for many modern telecommunication networks. Possible correlation of customers inter-arrival times and batch arrivals are taken into account via of consideration of the Batch Markovian Arrival Process (BMAP) as input stream to the system. The system consists of two stations. The service time at the station 1 is assumed to be generally distributed. There is no buffer at this station, and customers who meet the busy server repeat attempts to enter the system in random time intervals. The service process at the station 2 is assumed to be described by the continuous time Markov chain with a finite state space. This assumption holds good, e.g., if the station 2 has a finite buffer, consists of a finite number of identical or heterogeneous servers where the service time distribution is assumed to be of PH (PHase) type. Markov chain embedded at service completion epochs at the station 1 and the process of system states at arbitrary time are under study. Ergodicity condition and algorithms for computing the steady state probabilities are presented.

Fluid queues to solve jump processes

Mathematically, stochastic hybrid systems can be described by Markov processes with discrete and continuous behaviours. This chapter provides in an incremental way the necessary mathematical background for understanding such complex systems. The chapter starts with discrete and continuous-time Markov chains. In this book, a Markov chain is understood as a stochastic process with the Markov property (or memoryless property, i.e. its future evolution depends only on the current state) defined on a discrete (finite or countable) state space. The Markov property is illustrated in the Chapman-Kolmogorov equation satisfied by the transition probabilities. Usually, in the continuous-time Markov case, the evolution of these transition probabilities is described by forward/backward Kolmogorov equations. These equations are expressed in terms of the stochastic matrix (called also infinitesimal generator) associated to the Markov chain. This is the matrix of transition rates and the practical use of Markov chains resides in our ability of handling it.The second part of this chapter is dedicated to Markov processes defined on continuous state spaces that can be equipped with additional structures as sigma-algebras, metrics, topologies, norms or others like specific algebraic structures (lattice, vector space). Characterisations of such processes are usually given using functional analysis operators like: the operator semigroup/resolvent, the infinitesimal generator or other operators that can be associated to a Markov process. At an abstract level, these operators describe the evolution the evolution of transition probabilities for a Markov process. Chapman-Kolmogorov equation, Kolmogorov equations and other properties of these operators (e.g. Dynkin formula, martingale problem) represent the natural tools for studying such Markov processes.KeywordsMarkov ChainTopological SpaceMarkov ProcessMarkov PropertyPolish SpaceThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

This work develops asymptotic expansions for solutions of systems of backward equations of time-inhomogeneous Markov chains in continuous time. Owing to the rapid progress in technology and the increasing complexity in modeling, the underlying Markov chains often have large state spaces, which make the computational tasks infeasible. To reduce the complexity, two-time-scale formulations are used. By introducing a small parameter ɛ > 0 and using suitable decomposition and aggregation procedures, it is formulated as a singular perturbation problem. Both Markov chains having recurrent states only and Markov chains including also transient states are treated. Under certain weak irreducibility and smoothness conditions of the generators, the desired asymptotic expansions are constructed. Then error bounds are obtained.

We study a quantum entanglement switch that serves $k$ users in a star topology. We model variants of the system using Markov chains and standard queueing theory and obtain expressions for switch capacity and the expected number of qubits stored in memory at the switch. While it is more accurate to use a discrete-time Markov chain (DTMC) to model such systems, we quickly encounter practical constraints of using this technique and switch to using continuous-time Markov chains (CTMCs). Using CTMCs allows us to obtain a number of analytic results for systems in which the links are homogeneous or heterogeneous and for switches that have infinite or finite buffer sizes. In addition, we can model the effects of decoherence of quantum states fairly easily using CTMCs. We also compare the results we obtain from the DTMC against the CTMC in the case of homogeneous links and infinite buffer, and learn that the CTMC is a reasonable approximation of the DTMC. From numerical observations, we discover that decoherence has little effect on capacity and expected number of stored qubits for homogeneous systems. For heterogeneous systems, especially those operating close to stability constraints, buffer size and decoherence can have significant effects on performance metrics. We also learn that in general, increasing the buffer size from one to two qubits per link is advantageous to most systems, while increasing the buffer size further yields diminishing returns.

In this work, we establish a framework for balanced realization of linear systems subject to regime switching modulated by a continuous-time Markov chain with a finite state space. First, a definition of balanced realization is given. Then a ρ-balanced realization is developed to approximate the system of balancing equations, which is a system of time-varying algebraic equations. When the state space of the Markov chain is large, the computational effort becomes a real concern. To resolve this problem, we introduce a two-time-scale formulation and use decomposition/aggregation and averaging techniques to reduce the computational complexity. Based on the two-time-scale formulation, further approximation procedures are developed. Numerical examples are also presented for demonstration.

This paper is concerned with a class of hybrid stock market models, in which both the return rate and the volatility depend on a hidden, continuous-time Markov chain with a finite state space. One of the crucial issues is to estimate the generator of the underlying Markov chain. We develop a stochastic optimization procedure for this task, prove its convergence, and establish the rate of convergence. Numerical tests are carried out via simulation as well as using real market data. In addition, we demonstrate how to use the estimated generator in making stock liquidation decisions.

Reproduction numbers, like the basic reproduction number R0, play an important role in the analysis and application of dynamic models, including contagion models and ecological population models. One difficulty in deriving these quantities is that they must be computed on a model-by-model basis, since it is typically impractical to obtain general reproduction number expressions applicable to a family of related models, especially if these are of different dimensions (i.e., differing numbers of state variables). For example, this is typically the case for SIR-type infectious disease models derived using the linear chain trick. Here we show how to find general reproduction number expressions for such model families (which vary in their number of state variables) using the next generation operator approach in conjunction with the generalized linear chain trick (GLCT). We further show how the GLCT enables modelers to draw insights from these results by leveraging theory and intuition from continuous time Markov chains (CTMCs) and their absorption time distributions (i.e., phase-type probability distributions). To do this, we first review the GLCT and other connections between mean-field ODE model assumptions, CTMCs, and phase-type distributions. We then apply this technique to find reproduction numbers for two sets of models: a family of generalized SEIRS models of arbitrary finite dimension, and a generalized family of finite dimensional predator-prey (Rosenzweig-MacArthur type) models. These results highlight the utility of the GLCT for the derivation and analysis of mean field ODE models, especially when used in conjunction with theory from CTMCs and their associated phase-type distributions.

We study the Classical Probability analogue of the unitary dilations of a quantum dynamical semigroup defined in Quantum Probability via quantum stochastic differential equations. Given a homogeneous Markov chain in continuous time in a finite state space E, we introduce a second system, an environment, and a deterministic invertible time-homogeneous global evolution of the system E with this environment such that the original Markov evolution of E can be realized by a proper choice of the initial random state of the environment. We also compare this dilations with the unitary dilations of a quantum dynamical semigroup in Quantum Probability.

Life history data are generally incomplete. Usually, they do not cover for each individual in the study the entire life span or the life segment of interest. If data are collected retrospectively, observation ends at interview date, and no information is available on events and experiences after the date. Data collected prospectively are incomplete because events and other experiences are recorded during a limited period of time only. To deal with data limitations, models are introduced. The model that is considered in this chapter describes life histories. The model is based on the premise that life histories are realisations of a continuous-time Markov process. A Markov process is a stochastic process that describes a system with multiple states and transitions between the states. The time at which a transition occurs is random but the distribution of the time to transition is known. In the continuous-time Markov process, the transition time has an exponential distribution. The rate of transition out of the current state (exit rate) is the parameter of the exponential distribution. It depends on the current state only and is independent of the history of the stochastic process. In a system with multiple states, an individual who leaves the current state may enter one of several states. In competing risks models, states in the state space are viewed as competing destinations and transition rates are destination-specific. The Markov process is a first-order process: the destination state depends on the current state only and is independent of states occupied previously.