Some Invariance Principles for Functionals of a Markov Chain

Convergence theory of some classes of iterative aggregation/disaggregation methods for computing stationary probability vectors of stochastic matrices

Markov chains on a countable state space are studied under the assumption that the transition probabilities $(P_n(x,y))$ constitute a stationary stochastic process. An introductory section exposing some basic results of Nawrotzki and Cogburn is followed by four sections of new results.

This paper presents an efficient direct method for the solution of a queueing network consisting of two Coxian servers. The structure of the transition rate matrix associated with this network is examined in some detail, and it is shown how advantage may be taken of this structure to derive the required stationary probability vector in a number of operations which is proportional to N, the number of customers in the network. The direct method which is employed, is based on a single step of inverse iteration. The method is shown to be stable.

The general problem of convergence in distribution of Markov chains with a countable state space is investigated in this chapter. For the GI/GI/1 queue in Chapter 2, the convergence in distribution of the Markov chain (W n ) has been obtained by using an explicit representation of the random variable W n as a functional of the random walk associated with the interarrival times and the service times. Markov chains describing the behavior of most queueing systems cannot, in general, be represented in such a simple way. In this chapter, simple criteria are given to determine whether a given Markov chain is ergodic or transient. The main results are Theorem 8.6 for ergodicity and Theorem 8.10 for transience. In practice, these results can be used in many applications. These criteria can be seen as an extension, in a probabilistic setting, of a classical stability result of ordinary differential equations due to Lyapunov[Lia07] in 1892. In a stochastic context, the first results of this type are apparently due to Khasminskii for diffusions (see Khasminskii [26]). The stability criterion by Lyapunov is recalled at the end of the chapter (see Hirsch and Smale [27] for a detailed presentation of these questions). In Chapter 9, an important scaling method is introduced that will give a more precise picture of the relation between the stability of ordinary differential equations and the stability properties of Markov

Markov chains with periodic graphs arise frequently in a wide range of modelling experiments. Application areas range from flexible manufacturing systems in which pallets are treated in a cyclic manner to computer communication networks. The primary goal of this paper is to show how advantage may be taken of this property to reduce the amount of computer memory and computation time needed to compute stationary probability vectors of periodic Markov chains. After reviewing some basic properties of Markov chains whose associated graph is periodic, we introduce a ‘reduced scheme’ in which only a subset of the probability vector need be computed. We consider the effect of the application of both direct and iterative methods to the original Markov chain (permuted to normal periodic form) as well as to the reduced system. We show how the periodicity of the Markov chain may be efficiently computed as the chain itself is being generated. Finally, some numerical experiments to illustrate the theory, are presented.

Mathematically, stochastic hybrid systems can be described by Markov processes with discrete and continuous behaviours. This chapter provides in an incremental way the necessary mathematical background for understanding such complex systems. The chapter starts with discrete and continuous-time Markov chains. In this book, a Markov chain is understood as a stochastic process with the Markov property (or memoryless property, i.e. its future evolution depends only on the current state) defined on a discrete (finite or countable) state space. The Markov property is illustrated in the Chapman-Kolmogorov equation satisfied by the transition probabilities. Usually, in the continuous-time Markov case, the evolution of these transition probabilities is described by forward/backward Kolmogorov equations. These equations are expressed in terms of the stochastic matrix (called also infinitesimal generator) associated to the Markov chain. This is the matrix of transition rates and the practical use of Markov chains resides in our ability of handling it.The second part of this chapter is dedicated to Markov processes defined on continuous state spaces that can be equipped with additional structures as sigma-algebras, metrics, topologies, norms or others like specific algebraic structures (lattice, vector space). Characterisations of such processes are usually given using functional analysis operators like: the operator semigroup/resolvent, the infinitesimal generator or other operators that can be associated to a Markov process. At an abstract level, these operators describe the evolution the evolution of transition probabilities for a Markov process. Chapman-Kolmogorov equation, Kolmogorov equations and other properties of these operators (e.g. Dynkin formula, martingale problem) represent the natural tools for studying such Markov processes.KeywordsMarkov ChainTopological SpaceMarkov ProcessMarkov PropertyPolish SpaceThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

A note on the existence of optimal stationary policies for average Markov decision processes with countable states

We consider single-server queueing systems that are modulated by a discrete-time Markov chain on a countable state space. The underlying stochastic process is a Markov random walk (MRW) whose increments can be expressed as differences between service times and interarrival times. We derive the joint distributions of the waiting and idle times in the presence of the modulating Markov chain. Our approach is based on properties of the ladder sets associated with this MRW and its time-reversed counterpart. The special case of a Markov-modulated M/M/1 queueing system is then analysed and results analogous to the classical case are obtained.

This note presents a method of evaluating the distribution of a path integral for Markov chains on a countable state space.

Adaptive control of finite-state Markov chains is discussed, The optimal performance is characterized through the minimization of a long-run average cost functional, subject to constraints on several other such functionals. Under mild structural and feasibility conditions, two explicit adaptive control policies are exhibited for the case where the transition probabilities are unknown. The policies are optimal under the constrained optimization criterion. They rely on a powerful estimation scheme which provides consistent estimators for the transition probabilities. This scheme is of independent interest, as it provides strong consistency under a large number of adaptive schemes and is independent of any identifiability conditions. As an application, an optimal adaptive policy is derived for a system of K competing queues with countable state space, for which the constrained criteria arise naturally in the context of communication networks. >

We consider single-server queueing systems that are modulated by a discrete-time Markov chain on a countable state space. The underlying stochastic process is a Markov random walk (MRW) whose increments can be expressed as differences between service times and interarrival times. We derive the joint distributions of the waiting and idle times in the presence of the modulating Markov chain. Our approach is based on properties of the ladder sets associated with this MRW and its time-reversed counterpart. The special case of a Markov-modulated M/M/1 queueing system is then analysed and results analogous to the classical case are obtained.

Local area computer network simulations are inherently non-Markovian in that the underlying stochastic process cannot be modeled as a Markov chain with countable state space. We restrict attention to local network simulations whose underlying stochastic process can be represented as a generalized semi-Markov process (GSMP). Using new better than used distributional assumptions and sample path properties of the GSMP, we provide a geometric trials criterion for recurrence in this setting. We also provide conditions which ensure that a GSMP is a regenerative process and that the expected time between regeneration points is finite. Steady-state estimation procedures for ring and bus network simulations follow from these results.

One obtains the analogues of the “Ray's description” of the local time for one-dimensional Brownian motion, valid for arbitrary homogeneous Markov chains with discrete time and countable state space. Contrary to the case of the Brownian motion, one establishes the absence of the Markov property for the process of the occupation time in the case of the simplest one-dimensional symmetric random walk.

In this paper, we use the Markov chain censoring technique to study infinite state Markov chains whose transition matrices possess block-repeating entries. We demonstrate that a number of important probabilistic measures are invariant under censoring. Informally speaking, these measures involve first passage times or expected numbers of visits to certain levels where other levels are taboo;they are closely related to the so-called fundamental matrix of the Markov chain which is also studied here. Factorization theorems for the characteristic equation of the blocks of the transition matrix are obtained. Necessary and sufficient conditions are derived for such a Markov chain to be positive recurrent, null recurrent, or transient based either on spectral analysis, or on a property of the fundamental matrix. Explicit expressions are obtained for key probabilistic measures, including the stationary probability vector and the fundamental matrix, which could be potentially used to develop various recursivealgorithms for computing these measures.

The classical model of probability theory, due principally to Kolmogorov, defines probability as a totally-one measure on a sigma-algebra of subsets (events) of a given set (the sample space), and random variables as real-valued functions on the sample space, such that the inverse images of all Borel sets are events. From this model, all the results of probability theory are derived. However, the assertion that any given concrete situation is subject to probability theory is a scientific hypothesis verifiable only experimentally, by appropriate sampling, and never totally certain. Furthermore, classical probability theory allows for the possibility of "outliers"—sampled values which are misleading. In particular, Kolmogorov's Strong Law of Large Numbers asserts that, if, as is usually the case, a random variable has a finite expectation (its integral over the sample space), then the average value of N independently sampled values of this function converges to the expectation with probability 1, as N tends to infinity. This implies that there may be sample sequences (belonging to a set of probability 0) for which this convergence does not occur; these are the "outliers".