ON DETERMINISTIC FINITE STATE MACHINES IN RANDOM ENVIRONMENTS

This note is concerned with Bayesian estimation of the transition probabilities of a binary Markov chain observed from heterogeneous individuals. The model is founded on the Jeffreys′ prior which allows for transition probabilities to be correlated. The Bayesian estimator is approximated by means of Monte Carlo Markov chain (MCMC) techniques. The performance of the Bayesian estimates is illustrated by analyzing a small simulated data set.

A general model for the description of a discrete-time Markov chain in random environment is introduced. We study the stationary distribution of this model and its relationship with the Palm (embedded) distributions at certain environmental transition epochs. We also study the Events See Time Averages (ESTA) property for this model and we show that it is related to certain partial balance equations and a product-form stationary distribution. Next, we study certain forms of interaction between the environmental process and the process of interest that lead to tractable stationary distributions. More specifically, the stationary distributions of a large class of Markov chains in random environment can be computed efficiently using α-potentials of smaller Markov chains. This decomposability result is also the basis for developing approximations for the stationary distributions of intractable models that evolve in a slowly changing random environment. Moreover, explicit error bounds for these approximations are given. The model can be used for the performance evaluation of discrete time queueing systems in random environment.

We consider a discrete time hidden Markov model where the signal is a stationary Markov chain. When conditioned on the observations, the signal is a Markov chain in a random environment under the conditional measure. It is shown that this conditional signal is weakly ergodic when the signal is ergodic and the observations are nondegenerate. This permits a delicate exchange of the intersection and supremum of σ-fields, which is key for the stability of the nonlinear filter and partially resolves a long-standing gap in the proof of a result of Kunita [J. Multivariate Anal. 1 (1971) 365–393]. A similar result is obtained also in the continuous time setting. The proofs are based on an ergodic theorem for Markov chains in random environments in a general state space.

We consider random iterated function systems giving rise to Markov chains in random (stationary) environments. Conditions ensuring unique ergodicity and a “pure type” characterization of the limiting “randomly invariant” probability measure are provided. We also give a dimension formula and an algorithm for simulating exact samples from the limiting probability measure.

You are probably reading this book because you have an interesting source of data and you suspect it is not a linear one. Either you positively know it is nonlinear because you have some idea of what is going on in the piece of world that you are observing or you are led to suspect that it is because you have tried linear data analysis and you are unsatisfied with its results. Linear methods interpret all regular structure in a data set, such as a dominant frequency, through linear correlations (to be defined in Chapter 2 below). This means, in brief, that the intrinsic dynamics of the system are governed by the linear paradigm that small causes lead to small effects. Since linear equations can only lead to exponentially decaying (or growing) or (damped) periodically oscillating solutions, all irregular behaviour of the system has to be attributed to some random external input to the system. Now, chaos theory has taught us that random input is not the only possible source of irregularity in a system's output: nonlinear, chaotic systems can produce very irregular data with purely deterministic equations of motion in an autonomous way, i.e., without time dependent inputs. Of course, a system which has both, nonlinearity and random input, will most likely produce irregular data as well. Although we have not yet introduced the tools we need to make quantitative statements, let us look at a few examples of real data sets. They represent very different problems of data analysis where one could profit from reading this book since a treatment with linear methods alone would be inappropriate.

Additive N-step Markov chains as prototype model of symbolic stochastic dynamical systems with long-range correlations

Channel dispersion plays a fundamental role in assessing the backoff from capacity due to finite blocklength. This paper analyzes the channel dispersion for a simple channel with memory: the Gilbert-Elliott communication model in which the crossover probability of a binary symmetric channel evolves as a binary symmetric Markov chain, with and without side information at the receiver about the channel state. With side information, dispersion is equal to the average of the dispersions of the individual binary symmetric channels plus a term that depends on the Markov chain dynamics, which do not affect the channel capacity. Without side information, dispersion is equal to the spectral density at zero of a certain stationary process, whose mean is the capacity. In addition, the finite blocklength behavior is analyzed in the non-ergodic case, in which the chain remains in the initial state forever.

We study the problem of a two-user, single-channel cognitive radio network, in which the objective is to maximize the secondary user (SU) throughput subject to a constraint on the probability of collision experienced by the primary user (PU). The transmit/idle dynamics of the primary user is modeled as a binary Markov chain, and the secondary senses the channel and decides on its transmission and sensing strategy based on the estimated evolution of the primary user transmission state. Because of the Markov model of the primary user dynamics, the age of the information sensed by the secondary has an impact on its belief when it is not sensing. We apply a Lyapunov optimization algorithm to solve the constrained throughput optimization problem, which utilizes the AoI-dependent PU idle/transmit probability to make the SU's sense/transmit decision in each slot. We then apply the Lyapunov framework to identify the tradeoff between three fundamental information qualities: AoI, accuracy, and completeness. Characterizing these types of tradeoffs can be a useful intermediate step towards optimizing a variety of objectives.

The fluctuation of stock prices is modelled as a sequence of temporary equilibria on a financial market with different types of agents. I summarize joint work with M. Schweizer on the class of Ornstein-Uhlenbeck processes in a random environment which appears in the diffusion limit. Moreover, it is shown how the random environment may be generated by the interaction of a large set of agents modelled by Markov chains as they appear in the theory of probabilistic cellular automata.

We consider Markov processes, which describe, e.g. queueing network processes, in a random environment which influences the network by determining random breakdown of nodes, and the necessity of repair thereafter. Starting from an explicit steady-state distribution of product form available in the literature, we note that this steady-state distribution does not provide information about the correlation structure in time and space (over nodes). We study this correlation structure via one-step correlations for the queueing-environment process. Although formulas for absolute values of these correlations are complicated, the differences of correlations of related networks are simple and have a nice structure. We therefore compare two networks in a random environment having the same invariant distribution, and focus on the time behaviour of the processes when in such a network the environment changes or the rules for travelling are perturbed. Evaluating the comparison formulas we compare spectral gaps and asymptotic variances of related processes.

A general formulation of the stochastic model for a Markov chain in a random environment is given, including an analysis of the dependence relations between the environmental process and the controlled Markov chain, in particular the problem of feedback. Assuming stationary environments, the ergodic theory of Markov processes is applied to give conditions for the existence of finite invariant measure (equilibrium distributions) and to obtain ergodic theorems, which provide results on convergence of products of random stochastic matrices. Coupling theory is used to obtain results on direct convergence of these products and the structure of the tail σ-field. State properties including classification and communication properties are discussed.

Let S be the number of components in a finite discrete mixing distribution. We prove that the number of waves of panel being greater than or equal to 2S is a sufficient condition for global identification of a dynamic binary choice model in which all the parameters are heterogeneous. This model results in a mixture of S binary first‐order Markov Chains.

The output distribution formula for stack filters based on mirrored threshold decomposition is derived. This formula allows one to compute the cumulative distribution function of the output of a stack filter for a given input noise distribution. Mirrored threshold decomposition permits us to analyze the input-output characteristics of the stack filter in the binary domain. The sliding window operation of the stack filter is modeled by a deterministic finite automaton. The output distribution of the filter is obtained by interpreting the automaton as a Markov Chain whose transition probabilities depend on the probabilistic description of the binary input signal.

We consider a discrete time population model for which each individual alive at time n survives independently of everybody else at time n + 1 with probability β n . The sequence (βn) is i.i.d. and constitutes our random environment. Moreover, at every time n we add Z n individuals to the population. The sequence (Zn) is also i.i.d. We find sufficient conditions for null recurrence and transience (positive recurrence has been addressed by Neuts 1994 J. Appl. Probab. 31 48–58). We apply our results to a particular (Zn) distribution and deterministic β. This particular case shows a rather unusual phase transition in β in the sense that the Markov chain goes from transience to null recurrence without ever reaching positive recurrence.

A study of an n-unit system operating in a random environment