The stability of conditional Markov processes and Markov chains in random environments

We consider random iterated function systems giving rise to Markov chains in random (stationary) environments. Conditions ensuring unique ergodicity and a “pure type” characterization of the limiting “randomly invariant” probability measure are provided. We also give a dimension formula and an algorithm for simulating exact samples from the limiting probability measure.

A general model for the description of a discrete-time Markov chain in random environment is introduced. We study the stationary distribution of this model and its relationship with the Palm (embedded) distributions at certain environmental transition epochs. We also study the Events See Time Averages (ESTA) property for this model and we show that it is related to certain partial balance equations and a product-form stationary distribution. Next, we study certain forms of interaction between the environmental process and the process of interest that lead to tractable stationary distributions. More specifically, the stationary distributions of a large class of Markov chains in random environment can be computed efficiently using α-potentials of smaller Markov chains. This decomposability result is also the basis for developing approximations for the stationary distributions of intractable models that evolve in a slowly changing random environment. Moreover, explicit error bounds for these approximations are given. The model can be used for the performance evaluation of discrete time queueing systems in random environment.

A general formulation of the stochastic model for a Markov chain in a random environment is given, including an analysis of the dependence relations between the environmental process and the controlled Markov chain, in particular the problem of feedback. Assuming stationary environments, the ergodic theory of Markov processes is applied to give conditions for the existence of finite invariant measure (equilibrium distributions) and to obtain ergodic theorems, which provide results on convergence of products of random stochastic matrices. Coupling theory is used to obtain results on direct convergence of these products and the structure of the tail σ-field. State properties including classification and communication properties are discussed.

Tree-Indexed Markov Chains in Random Environment and Some of Their Strong Limit Properties

Optimal Hedging in Discrete and Continuous Time

Simple SummaryUnderstanding and predicting vector population and related disease dynamics, is crucial for gaining insight into the abundance and dynamics of arthropod disease vectors, and for the design of effective vector control strategies. Several mathematical and standard epidemiological models have been proposed in studying vector transmitted infectious disease dynamics. However, most models are of deterministic nature and are not able to estimate other relevant metrics such as the probability of vector population emergence as well as the probability and expected time to reach certain population and/or infection state. Here we are focusing on stochastic modeling of mosquito abundance data using weather driven Markov chains (MCs) and are particularly interested in estimating transition probabilities (TPs) between different population levels. A MC model is based on the assumption that the future state of the variable is only dependent on the present state and is suitable in cases of short and noisy data characterized by a complex and random behavior. The aim is to introduce and generalize a formulation of conditional Markov chain models (CMSs) for predicting probability transition estimates of arthropod vector populations. In this context, first we present the basic principles and assumptions behind Markov chain modeling approach, with an intuitive interpretation of the integration of conditional Markov chains (CMCs) and then demonstrate the usefulness of the approach in predicting the abundance of Culex sp. We conclude that the conditional Markov chain technique is recommended as viable for modeling populations that explicit random dynamics and predict their future evolution. Although, the Markov models generated in this work provide an accurate abstraction of the vector disease progress observed within the dataset used for their generation, we envision the current approach as an entry point into the medical entomology literature and methods for predicting arthropod vector diseases dynamics.Understanding and predicting mosquito population dynamics is crucial for gaining insight into the abundance of arthropod disease vectors and for the design of effective vector control strategies. In this work, a climate-conditioned Markov chain (CMC) model was developed and applied for the first time to predict the dynamics of vectors of important medical diseases. Temporal changes in mosquito population profiles were generated to simulate the probabilities of a high population impact. The simulated transition probabilities of the mosquito populations achieved from the trained model are very near to the observed data transitions that have been used to parameterize and validate the model. Thus, the CMC model satisfactorily describes the temporal evolution of the mosquito population process. In general, our numerical results, when temperature is considered as the driver of change, indicate that it is more likely for the population system to move into a state of high population level when the former is a state of a lower population level than the opposite. Field data on frequencies of successive mosquito population levels, which were not used for the data inferred MC modeling, were assembled to obtain an empirical intensity transition matrix and the frequencies observed. Our findings match to a certain degree the empirical results in which the probabilities follow analogous patterns while no significant differences were observed between the transition matrices of the CMC model and the validation data (ChiSq = 14.58013, df = 24, p = 0.9324451). The proposed modeling approach is a valuable eco-epidemiological study. Moreover, compared to traditional Markov chains, the benefit of the current CMC model is that it takes into account the stochastic conditional properties of ecological-related climate variables. The current modeling approach could save costs and time in establishing vector eradication programs and mosquito surveillance programs.

We derive some key extremal features for stationary kth-order Markov chains that can be used to understand how the process moves between an extreme state and the body of the process. The chains are studied given that there is an exceedance of a threshold, as the threshold tends to the upper endpoint of the distribution. Unlike previous studies with $k>1$ , we consider processes where standard limit theory describes each extreme event as a single observation without any information about the transition to and from the body of the distribution. Our work uses different asymptotic theory which results in non-degenerate limit laws for such processes. We study the extremal properties of the initial distribution and the transition probability kernel of the Markov chain under weak assumptions for broad classes of extremal dependence structures that cover both asymptotically dependent and asymptotically independent Markov chains. For chains with $k>1$ , the transition of the chain away from the exceedance involves novel functions of the k previous states, in comparison to just the single value, when $k=1$ . This leads to an increase in the complexity of determining the form of this class of functions, their properties, and the method of their derivation in applications. We find that it is possible to derive an affine normalization, dependent on the threshold excess, such that non-degenerate limiting behaviour of the process, in the neighbourhood of the threshold excess, is assured for all lags. We find that these normalization functions have an attractive structure that has parallels to the Yule–Walker equations. Furthermore, the limiting process is always linear in the innovations. We illustrate the results with the study of kth-order stationary Markov chains with exponential margins based on widely studied families of copula dependence structures.

Nonlinear PDEs and measure-valued branching type processes

A generalization to continuous time is given for a discrete-time model of a birth and death process in a random environment. Some important properties of this process in the continuous-time setting are stated and proved including instability and extinction conditions, and when suitable absorbing barriers have been defined, methods are given for the calculation of extinction probabilities and the expected duration of the process.

We consider continuous and impulse control of a Markov chain (MC) with a finite set of states in continuous time. Continuous control determines the intensity of transitions between MC states, while transition times and their directions are random. Nevertheless, sometimes it is necessary to ensure a transition that leads to an instantaneous change in the state of the MC. Since such transitions require different influences and can produce different effects on the state of the MC, such controls can be interpreted as impulse controls. In this work, we use the martingale representation of a controllable MC and give an optimality condition, which, using the principle of dynamic programming, is reduced to a form of quasi-variational inequality. The solution to this inequality can be obtained in the form of a dynamic programming equation, which for an MC with a finite set of states reduces to a system of ordinary differential equations with one switching line. We prove a sufficient optimality condition and give examples of problems with deterministic and random impulse action.

The fluctuation of stock prices is modelled as a sequence of temporary equilibria on a financial market with different types of agents. I summarize joint work with M. Schweizer on the class of Ornstein-Uhlenbeck processes in a random environment which appears in the diffusion limit. Moreover, it is shown how the random environment may be generated by the interaction of a large set of agents modelled by Markov chains as they appear in the theory of probabilistic cellular automata.

We consider Markov processes, which describe, e.g. queueing network processes, in a random environment which influences the network by determining random breakdown of nodes, and the necessity of repair thereafter. Starting from an explicit steady-state distribution of product form available in the literature, we note that this steady-state distribution does not provide information about the correlation structure in time and space (over nodes). We study this correlation structure via one-step correlations for the queueing-environment process. Although formulas for absolute values of these correlations are complicated, the differences of correlations of related networks are simple and have a nice structure. We therefore compare two networks in a random environment having the same invariant distribution, and focus on the time behaviour of the processes when in such a network the environment changes or the rules for travelling are perturbed. Evaluating the comparison formulas we compare spectral gaps and asymptotic variances of related processes.

This paper studies the reversibility conditions of stationary Markov chains (discrete-time Markov processes) with general state space. In particular, we investigate the Markov chains having atomic points in the state space. Such processes are often seen in storage models, for example waiting time in a queue, insurance risk reserve, dam content and so on. The necessary and sufficient conditions for reversibility of these processes are obtained. Further, we apply these conditions to some storage models and present some interesting results for single-server queues and a finite insurance risk model.

This paper is concerned with the problem of nonlinear filter stability of ergodic Markov processes. The main contribution is the conditional Poincaré inequality (PI), which is shown to yield filter stability. The proof is based upon a recently discovered duality which is used to transform the nonlinear filtering problem into a stochastic optimal control problem for a backward stochastic differential equation (BSDE). Based on these dual formalisms, a comparison is drawn between the stochastic stability of a Markov process and the filter stability. The latter relies on the conditional PI described in this paper, whereas the former relies on the standard form of PI.

Equilibria in stochastic economic models are often time series which fluctuate in complex ways. But it is sometimes possible to summarize the long run, characteristics of these fluctuations. For example, if the law of motion determined by economic interactions is Markovian and if the equilibrium time series converges in a specific probabilistic sense then the long run behavior is completely determined by an invariant probability distribution. This paper develops and unifies a number of results found in the probability literature which enable one to prove, under very general conditions, the existence of an invariant distribution and the convergence of the corresponding Markov process. VIRTUALLY ALL OF ECONOMIC THEORY focuses upon the study of economic equilibrium. This concept has recently undergone several subtle elaborations. No longer must a system of markets in equilibrium be thought of as one at rest in a static steady state. Instead there is a growing body of literature (e.g., [4, 5, 12, 16, 20, 21]) which defines equilibrium as a stochastic process of market clearing prices and quantities which is consistent with the self-interested behavior of economic agents. Needless to say equilibrium stochastic processes can be very complex time series which fluctuate in irregular ways. For theoretical and econometric purposes it is useful to have a convenient way of summarizing the average behavior of such processes over time. This paper draws together and unifies a number of fundamental results from the probability literature which enable one to do this for discrete time, Markov processes on general state spaces. The starting point of the analysis is a set S of economic states (e.g., prices and/or quantities). The only technical restriction placed upon S is that it be a Borel subset of a complete, separable metric space. The second datum is a transition probability P(s, ) on S. The number P(s,A) records the probability that the economic system moves from the state s to some state in the Borel subset A of S during one unit of elapsed time. In economic applications the transition probability is usually derived from hypotheses about market clearing and the maximizing behavior of economic agents. The transition probability (together with an initial probability measure on S) defines a discrete time Markov process. One way of summarizing the dynamic behavior implied by P is to look for an invariant probability. A probability measure X on S is invariant for P if for all Borel subsets A of S one has the equality f P(s, A )X(ds) = X(A). An invariant probability is a kind of probabilistic steady state for the dynamics defined by P. Of course there may be no invariant probability for P at all; and even if one exists it may convey no information about the behavior of the process over time except under very special initial conditions. There is a second way of summarizing the behavior of Markov processes defined by the transition probability P. Let P (s,A) denote the n step transition