Markov Chains with Random Transition Probabilities

Markovian jumping systems have arisen naturally in the mathematical modeling of phenomena spanning disciplines in the social sciences, natural sciences, and engineering. This kind of stochastic dynamical systems can be employed to model the dynamics when parameters are subject to random abrupt changes due to sudden environment changes, subsystem switching, system noises, executor faults, and so forth. Much attention has been given to modeling, optimization, and real applications of such stochastic dynamical systems in the literature in recent years. As the advanced control and optimization will provide a basis for the design and application of such stochastic systems, these advanced techniques would result in substantial and sustainable benefits. The accepted papers in this special issue include stochastic stability, stabilization, stochastic control optimization, system modeling and identification methods, predictive control, signal processing, robust filtering, multiagent systems, networked control systems, time-delayed systems, neural networks, the Takagi-Sugeno fuzzy systems, simulated annealing, and fault detection methods.

We consider Markov chains with random transition probabilities which, moreover, fluctuate randomly with time. We describe such a system by a product of stochastic matrices, , with the factors Mi drawn independently from an ensemble of random Markov matrices, whose columns are independent Dirichlet random variables. The statistical properties of the columns of , its largest eigenvalue and its spectrum are obtained exactly for N = 2 and numerically investigated for general N. For large t, the columns are Dirichlet-distributed, however the distribution is different from the initial one. As for the spectrum, we find that the eigenvalues converge to zero exponentially fast and investigate the statistics of the largest stability and Lyapunov exponents, which are approximated by Gamma distributions. We also observe a concentration of the spectrum on the real line for large t.

Mathematically, stochastic hybrid systems can be described by Markov processes with discrete and continuous behaviours. This chapter provides in an incremental way the necessary mathematical background for understanding such complex systems. The chapter starts with discrete and continuous-time Markov chains. In this book, a Markov chain is understood as a stochastic process with the Markov property (or memoryless property, i.e. its future evolution depends only on the current state) defined on a discrete (finite or countable) state space. The Markov property is illustrated in the Chapman-Kolmogorov equation satisfied by the transition probabilities. Usually, in the continuous-time Markov case, the evolution of these transition probabilities is described by forward/backward Kolmogorov equations. These equations are expressed in terms of the stochastic matrix (called also infinitesimal generator) associated to the Markov chain. This is the matrix of transition rates and the practical use of Markov chains resides in our ability of handling it.The second part of this chapter is dedicated to Markov processes defined on continuous state spaces that can be equipped with additional structures as sigma-algebras, metrics, topologies, norms or others like specific algebraic structures (lattice, vector space). Characterisations of such processes are usually given using functional analysis operators like: the operator semigroup/resolvent, the infinitesimal generator or other operators that can be associated to a Markov process. At an abstract level, these operators describe the evolution the evolution of transition probabilities for a Markov process. Chapman-Kolmogorov equation, Kolmogorov equations and other properties of these operators (e.g. Dynkin formula, martingale problem) represent the natural tools for studying such Markov processes.KeywordsMarkov ChainTopological SpaceMarkov ProcessMarkov PropertyPolish SpaceThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Existing stock price models are based on time series methodologies which are hard to estimate and involves lots of assumptions. This study, in contrast, assumes that the stock prices follow stochastic process that possesses Markov dependency with finite state transition probabilities. For this purpose, daily stock index data from Pakistan Stock Exchange (PSE) is collected from 2010-2015 and categorized in to 10 state spaces. Based on the results of state transition model, the study highlights the most probable state of return and also its transition into another state. Further, the study used Monte Carlo method of stock index simulations both Markov chain and original stock index. The analysis shows that it is possible to model and forecast stock index by capturing Markov process. The results of the study are helpful for investors in selecting right time of making investment and for academician to think about more sophisticated methods of state identification.

ABSTRACTFinite memory sources and variable‐length Markov chains have recently gained popularity in data compression and mining, in particular, for applications in bioinformatics and language modelling. Here, we consider denser data compression and prediction with a family of sparse Bayesian predictive models for Markov chains in finite state spaces. Our approach lumps transition probabilities into classes composed of invariant probabilities, such that the resulting models need not have a hierarchical structure as in context tree‐based approaches. This can lead to a substantially higher rate of data compression, and such non‐hierarchical sparse models can be motivated for instance by data dependence structures existing in the bioinformatics context. We describe a Bayesian inference algorithm for learning sparse Markov models through clustering of transition probabilities. Experiments with DNA sequence and protein data show that our approach is competitive in both prediction and classification when compared with several alternative methods on the basis of variable memory length.

A batch Markov arrival process (BMAP) X* = (N, J) is a 2-dimensional Markov process with two components, one is the counting process N and the other one is the phase process J. It is proved that the phase process is a time-homogeneous Markov chain with a finite state-space, or for short, Markov chain. In this paper, a new and inverse problem is proposed firstly: given a Markov chain J, can we deploy a process N such that the 2-dimensional process X* = (N, J) is a BMAP? The process X* = (N, J) is said to be an adjoining BMAP for the Markov chain J. For a given Markov chain the adjoining processes exist and they are not unique. Two kinds of adjoining BMAPs have been constructed. One is the BMAPs with fixed constant batches, the other one is the BMAPs with independent and identically distributed (i.i.d) random batches. The method we used in this paper is not the usual matrix-analytic method of studying BMAP, it is a path-analytic method. We constructed directly sample paths of adjoining BMAPs. The expressions of characteristic (D k , k = 0, 1, 2 · · ·) and transition probabilities of the adjoining BMAP are obtained by the density matrix Q of the given Markov chain J. Moreover, we obtained two frontal Theorems. We present these expressions in the first time.

Equilibria in stochastic economic models are often time series which fluctuate in complex ways. But it is sometimes possible to summarize the long run, characteristics of these fluctuations. For example, if the law of motion determined by economic interactions is Markovian and if the equilibrium time series converges in a specific probabilistic sense then the long run behavior is completely determined by an invariant probability distribution. This paper develops and unifies a number of results found in the probability literature which enable one to prove, under very general conditions, the existence of an invariant distribution and the convergence of the corresponding Markov process. VIRTUALLY ALL OF ECONOMIC THEORY focuses upon the study of economic equilibrium. This concept has recently undergone several subtle elaborations. No longer must a system of markets in equilibrium be thought of as one at rest in a static steady state. Instead there is a growing body of literature (e.g., [4, 5, 12, 16, 20, 21]) which defines equilibrium as a stochastic process of market clearing prices and quantities which is consistent with the self-interested behavior of economic agents. Needless to say equilibrium stochastic processes can be very complex time series which fluctuate in irregular ways. For theoretical and econometric purposes it is useful to have a convenient way of summarizing the average behavior of such processes over time. This paper draws together and unifies a number of fundamental results from the probability literature which enable one to do this for discrete time, Markov processes on general state spaces. The starting point of the analysis is a set S of economic states (e.g., prices and/or quantities). The only technical restriction placed upon S is that it be a Borel subset of a complete, separable metric space. The second datum is a transition probability P(s, ) on S. The number P(s,A) records the probability that the economic system moves from the state s to some state in the Borel subset A of S during one unit of elapsed time. In economic applications the transition probability is usually derived from hypotheses about market clearing and the maximizing behavior of economic agents. The transition probability (together with an initial probability measure on S) defines a discrete time Markov process. One way of summarizing the dynamic behavior implied by P is to look for an invariant probability. A probability measure X on S is invariant for P if for all Borel subsets A of S one has the equality f P(s, A )X(ds) = X(A). An invariant probability is a kind of probabilistic steady state for the dynamics defined by P. Of course there may be no invariant probability for P at all; and even if one exists it may convey no information about the behavior of the process over time except under very special initial conditions. There is a second way of summarizing the behavior of Markov processes defined by the transition probability P. Let P (s,A) denote the n step transition

In this paper, we study some strong limit theorems relative to the geometric average of random transition probabilities of nonhomogeneous Markov chains indexed by a tree. This result is an extension of some strong limit theorems relative to the geometric average of random transition probabilities of arbitrary finite nonhomogeneous Markov chains.

This paper describes a Bayesian approach to determining the order of a finite state Markov chain whose transition probabilities are themselves governed by a homogeneous finite state Markov chain. It extends previous work on homogeneous Markov chains to more general and applicable hidden Markov models. The method we describe uses a Markov chain Monte Carlo algorithm to obtain samples from the (posterior) distribution for both the order of Markov dependence in the observed sequence and the other governing model parameters. These samples allow coherent inferences to be made straightforwardly in contrast to those which use information criteria. The methods are illustrated by their application to both simulated and real data sets.

Let C be a collection of particles, each of which is independently undergoing the same Markov chain, and let d be a metric on the state space. Then, using transition probabilities, for distinct p, q in C, any time t and real x, we can calculate F (t)pq (x) = Pr [d (p,q)<x at time t]. For each time t ≧0, the collection C is shown to be a probabilistic metric space under the triangle function \(\tau _{T_m } \). In this paper we study the structure and limiting behavior of PM spaces so constructed. We show that whenever the transition probabilities have non-degenerate limits then the limit of the family of PM spaces exists and is a PM space under the same triangle function. For an irreducible, aperiodic, positive recurrent Markov chain, the limiting PM space is equilateral. For an irreducible, positive recurrent Markov chain with period p, the limiting PM space has at most only [p/2]+2 distinct distance distribution functions. Finally, we exhibit a class of Markov chains in which all of the states are transient, so that P ij(t)→0 for all states i, j, but for which the {F ttpq } all have non-trivial limits and hence a non-trivial limiting PM space does exist.

This review article is concerned to provide a global context to several works on the fitting of continuous time nonhomogeneous Markov chains with finite state space and also to point out some selected aspects of two techniques previously introduced—estimation and calibration—relevant for applications and used to fit a continuous time Markov chain model to data by the adequate selection of parameters. The denomination estimation suits the procedure better when statistical techniques—e.g., maximum likelihood estimators—are employed, while calibration covers the case where, for instance, some optimisation technique finds a best approximation parameter to ensure good model fitting. For completeness, we provide a short summary of well-known important notions and results formulated for nonhomogeneous Markov chains that, in general, can be transferred to the homogeneous case. Then, as an illustration for the homogeneous case, we present a selected Billingsley’s result on parameter estimation for irreducible chains with finite state space. In the nonhomogeneous case, we quote two recent results, one of the calibration type and the other with more of a statistical flavour. We provide an ample set of bibliographic references so that the reader wanting to pursue her/his studies will be able to do so more easily and productively.

In this paper, we consider continuous‐time Markov chains with a finite state space under nonlinear expectations. We define so‐called Q‐operators as an extension of Q‐matrices or rate matrices to a nonlinear setup, where the nonlinearity is due to model uncertainty. The main result gives a full characterization of convex Q‐operators in terms of a positive maximum principle, a dual representation by means of Q‐matrices, time‐homogeneous Markov chains under convex expectations, and a class of nonlinear ordinary differential equations. This extends a classical characterization of generators of Markov chains to the case of model uncertainty in the generator. We further derive an explicit primal and dual representation of convex semigroups arising from Markov chains under convex expectations via the Fenchel–Legendre transformation of the generator. We illustrate the results with several numerical examples, where we compute price bounds for European contingent claims under model uncertainty in terms of the rate matrix.

An analytical approach to the study of countable non-homogeneous Markov chains based on the z -transformation is proposed. Using two variants of the initial data for the transition probabilities of a Markov chain as an example, it is shown that estimates of the steady-state mode can significantly distort the understanding of the system behavior. Analytical procedures for obtaining probability functions for real and complex-conjugate eigenvalues of the transition probability matrix are described for the case when its elements change abruptly. Estimates are given for the boundaries of the onset of a steady-state mode in the clock time. The main calculations are illustrated by an assessment of the characteristics of Markov chains taking into account the influence of transition dynamics under variance of the state probabilities and the piecewise constant change in transition probabilities over the operating interval in clock time.

In this paper the using of Nelson Aalen estimators are presented to estimate transition probabilities of multistate model. Based on discrete time Markov, we will get transition matrices?e?a which the elements are transition probabilities from Nelson Aalen estimator. Because of the data that used in the construction of transition matrices are person?oOe¼Oaos health histories, then it can be seen as a morbidity value, which can be used to premium pricing. ?e?a Key words : Markov process, multistate model, Nelson Aalen, transition probability.

Collapsing of non-homogeneous Markov chains