1 - Discrete state space Markov processes

The paper considers a class of complex control systems which modelled as a finite family of randomly coupled individual stochastic differential equations. Each individual equation describes an individual mode of complex system. Between these modes the discontinuous transitions are seen. This jumping process is modelled by homogeneous Markov chain with discrete set of states. In accordance with these special features the full state vector of the system under study is hybrid and contains both component taking values in continuous space and component taking values in a finite discrete set. At the moment of the jump of Markov chain the continuous state component can be changed by jump too, but its value after jump is uniquely dependent on the value before one. Based on power transformation technique the conditions of robust stability in the sense that the complex system is 2p-stable independently on the transition probabilities of discrete state component (Markov chain above) are obtained in both linear case and in the presence of Lur'e-type nonlinearities. The state feedback control law, which guarantees the robust 2p-stability of the closed-loop complex system in linear case, is also given. All the results are formulated in terms of linear matrix inequalities.

ABSTRACTThis article addresses the problem of repeats detection used in the comparison of significant repeats in sequences. The case of self-overlapping leftmost repeats for large sequences generated by an homogeneous stationary Markov chain has not been treated in the literature. In this work, we are interested by the approximation of the number of self-overlapping leftmost long enough repeats distribution in an homogeneous stationary Markov chain. Using the Chen–Stein method, we show that the number of self-overlapping leftmost long enough repeats distribution is approximated by the Poisson distribution. Moreover, we show that this approximation can be extended to the case where the sequences are generated by a m-order Markov chain.

This paper describes a Bayesian approach to determining the order of a finite state Markov chain whose transition probabilities are themselves governed by a homogeneous finite state Markov chain. It extends previous work on homogeneous Markov chains to more general and applicable hidden Markov models. The method we describe uses a Markov chain Monte Carlo algorithm to obtain samples from the (posterior) distribution for both the order of Markov dependence in the observed sequence and the other governing model parameters. These samples allow coherent inferences to be made straightforwardly in contrast to those which use information criteria. The methods are illustrated by their application to both simulated and real data sets.

Mathematically, stochastic hybrid systems can be described by Markov processes with discrete and continuous behaviours. This chapter provides in an incremental way the necessary mathematical background for understanding such complex systems. The chapter starts with discrete and continuous-time Markov chains. In this book, a Markov chain is understood as a stochastic process with the Markov property (or memoryless property, i.e. its future evolution depends only on the current state) defined on a discrete (finite or countable) state space. The Markov property is illustrated in the Chapman-Kolmogorov equation satisfied by the transition probabilities. Usually, in the continuous-time Markov case, the evolution of these transition probabilities is described by forward/backward Kolmogorov equations. These equations are expressed in terms of the stochastic matrix (called also infinitesimal generator) associated to the Markov chain. This is the matrix of transition rates and the practical use of Markov chains resides in our ability of handling it.The second part of this chapter is dedicated to Markov processes defined on continuous state spaces that can be equipped with additional structures as sigma-algebras, metrics, topologies, norms or others like specific algebraic structures (lattice, vector space). Characterisations of such processes are usually given using functional analysis operators like: the operator semigroup/resolvent, the infinitesimal generator or other operators that can be associated to a Markov process. At an abstract level, these operators describe the evolution the evolution of transition probabilities for a Markov process. Chapman-Kolmogorov equation, Kolmogorov equations and other properties of these operators (e.g. Dynkin formula, martingale problem) represent the natural tools for studying such Markov processes.KeywordsMarkov ChainTopological SpaceMarkov ProcessMarkov PropertyPolish SpaceThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

This paper deals with methods for constructing and solving large Markov chain models of computer system availability and reliability. A set of powerful high level modeling constructs is discussed that can be used to represent the failure and repair behavior of the components that comprise a system, including important component interactions. If time independent failure and repair rates are assumed then a time homogeneous continuous time Markov chain can be constructed automatically from the modeling constructs used to describe the system. Since, the size of Markov chains grows exponentially with the number of components modeled, simulation appears to be a practical way for solving models of large systems. However, the standard simulation takes very long simulation runs to estimate availability and reliability measures because the system failure event is a rare event. Therefore, variance reduction techniques which can aid in computing rare-event probabilities quickly are of interest. Specifically, the Importance Sampling technique has been found to be most useful. The modeling language and the simulation methods discussed in this paper have been implemented in a program package called the System Availability Estimator (SAVE).

We present in this paper a mathematical method based on stochastic comparisons of Markov chains in order to compute in a mobile network performance measures that are important Q parameters for users: the dropping handover of voice and the blocking probability of a new voice call. The key idea of this methodology is that given a complex system represented by a multi-dimensional Markov chain which is too large to be solved, we propose to reduce the state space, and so to define a new Markov chain which is a simplified version of the original one. This reduced Markov chain is defined as an aggregated one, which represents a stochastic bound for performance measures written as increasing reward function on the stationary distribution. The main steps of the construction of the aggregated Markov chain applied into mobile networks are presented in this paper. As the number of mobile users with different kinds of applications increases, the associated model is more complex and it can be represented by multi-dimensional continuous time Markov chain with a very large size. Thus, we define an aggregated Markov chain represented by a multi-dimensional birth and death process which is very easy to solve. We have proved that there is a weak ordering between the original Markov chain and the aggregated one using increasing set formalism. We have computed upper bounds of dropping handover and blocking probability for different values of input parameters. Numerical results prove that upper bounds give good results and so the stochastic methodology is an interesting mathematical tool for the performance evaluation of complex systems. Keywords: Mobile Networks, Quality of Service, Stochastic ordering, stochastic comparisons, Continuous Time Markov Chains.

In many applications, successive observations of a process, say X 1, X 2,…, have an inherent time component associated with them. For example, the X i could be the state of the weather at a particular location on the i th day, counting from some fixed day. In a simplistic model, the state of the weather could be “dry” or “wet,” quantified as, say, 0 and 1. It is hard to believe that in such an example, the sequence X 1, X 2,… could be mutually independent. The question then arises how to model the dependence among the X i ′s. Probabilists have numerous dependency models. A particular model that has earned a very special status is called the Markov chain. In a Markov chain model, we assume that the future, given the entire past and the present state of a process, depends only on the present state. In the weather example, suppose we want to assign a probability that tomorrow, say March 10, will be dry, and suppose that we have available to us the precipitation history for each day from January 1 to March 9. In a Markov chain model, our probability that March 10 will be dry will depend only on the state of the weather on March 9, even though the entire past precipitation history was available to us. As simple as it sounds, Markov chains are enormously useful in applications, perhaps more than any other specific dependency model. They also are independently relevant to statistical computing in very important ways. The topic has an incredibly rich and well-developed theory, with links to many other topics in probability theory. Familiarity with basic Markov chain terminology and theory is often considered essential for anyone interested in studying statistics and probability.We present an introduction to basic Markov chain theory in this chapter.KeywordsMarkov ChainStationary DistributionMarkov Chain ModelTransition Probability MatrixStationary Markov ChainThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Importance sampling is a Monte Carlo simulation technique in which the simulation distribution is different from the true underlying distribution. In order to obtain an unbiased Monte Carlo estimate of the desired parameter, simulated events are weighted to reflect their true relative frequency. In this paper, we consider the estimation via simulation of certain large deviations probabilities for time-homogeneous Markov chains. We first demonstrate that when the simulation distribution is also a homogeneous Markov chain, the estimator variance will vanish exponentially as the sample size n tends to∞. We then prove that the estimator variance is asymptotically minimized by the same exponentially twisted Markov chain which arises in large deviation theory, and furthermore, this optimization is unique among uniformly recurrent homogeneous Markov chain simulation distributions.

Importance sampling is a Monte Carlo simulation technique in which the simulation distribution is different from the true underlying distribution. In order to obtain an unbiased Monte Carlo estimate of the desired parameter, simulated events are weighted to reflect their true relative frequency. In this paper, we consider the estimation via simulation of certain large deviations probabilities for time-homogeneous Markov chains. We first demonstrate that when the simulation distribution is also a homogeneous Markov chain, the estimator variance will vanish exponentially as the sample size n tends to∞. We then prove that the estimator variance is asymptotically minimized by the same exponentially twisted Markov chain which arises in large deviation theory, and furthermore, this optimization is unique among uniformly recurrent homogeneous Markov chain simulation distributions.

We consider weak lumpability of finite homogeneous Markov chains, which is when a lumped Markov chain with respect to a partition of the initial state space is also a homogeneous Markov chain. We show that weak lumpability is equivalent to the existence of a direct sum of polyhedral cones that is positively invariant by the transition probability matrix of the original chain. It allows us, in a unified way, to derive new results on lumpability of reducible Markov chains and to obtain spectral properties associated with lumpability.

The known approaches to project management are typically based on the intuitive decisions by project managers, not on theoretical foundations, patterns and laws of project management. In order to solve this contradiction, it is necessary to conduct the generalization and develop applied aspects for using the Markov chains to represent and model the weakly structured project management systems. To construct the Markov chain, we used a modified system of project management, which is represented in the standard of project management and defines the scheme of interaction between project participants. A method for the transformation of this scheme into a homogeneous Markov chain with discrete states and time is developed. It is demonstrated that iteration solution of the system of equations of the Markov model makes it possible to construct a trajectory of development of virtual or actual project systems. We examined practical aspects in measuring the effectiveness of trajectory of project development by using the devised Markov model. An evaluation of results of changing the probability distribution of project states in the coordinates of probabilities of the system states by steps revealed a significant effect on the progress and efficiency of a project only in the case of variation of terms of cooperation within a project team.

Summary Markov chains are commonly used to model succession in plant and animal communities. Once fitted to data, these models are typically used to address ecological issues concerning future successional states. In some situations it may be of interest to use the present successional state to reconstruct past conditions. The properties of a time‐reversed Markov chain are reviewed and used to provide an expression for the conditional probability distribution of the most recent time that the chain was in a particular successional state given its present state. The estimation of this conditional probability is discussed and a parametric bootstrap is described for constructing a confidence interval. The calculations are illustrated using a published Markov chain model of succession in a rocky subtidal community. Synthesis and applications. The term succession refers to the progressive changes over time in the state of an ecological community. Although most analyses of succession focus on characterizing the future, in some situations interest centres on reconstructing the past. For example, forensic entomologists are commonly interested in estimating time of death from the insect community present on a corpse. When succession can be modelled as a Markov chain, the results obtained here can be used for this kind of reconstruction.

In this paper, we obtain Markovian bounds on a function of a homogeneous discrete time Markov chain. For deriving such bounds, we use well-known results on stochastic majorization of Markov chains and the Rogers–Pitman lumpability criterion. The proposed method of comparison between functions of Markov chains is not equivalent to generalized coupling method of Markov chains, although we obtain same kind of majorization. We derive necessary and sufficient conditions for existence of our Markovian bounds. We also discuss the choice of the geometric invariant related to the lumpability condition that we use.

In large-scale distributed systems, the interactions of many independent components may lead to emergent global behaviors with unforeseen, often detrimental, outcomes. The increasing importance of distributed systems such as clouds and computing grids will require analytical tools to understand and predict, complex system behavior to ensure system reliability. In previous work, we described how a piecewise homogeneous Discrete Time Markov chain representation of a computing grid can be systematically perturbed to predict situations that lead to performance degradations. While the execution time of this approach compared favorably with detailed large-scale simulation, a sizable number of perturbations of the model were needed to identify scenarios in which system performance degraded. Here, we evolve our original approach and describe two novel methods for quickly identifying portions of the Markov chain that are sensitive to perturbation. The first method involves finding minimal s-t cut sets, consisting of state transitions that disconnect all paths in a Markov chain from the initial to a desired end state. By perturbing state transitions in the cut set, it is possible to quickly identify scenarios in which system performance is adversely affected. We show this method can be applied to larger Markov models than the approach described in our earlier work. We then present a second method, in which the Spectral Expansion Theorem is used to analyze the eigensystem of a set of Markov transition probability matrices to predict which state transitions, if perturbed, are likely to adversely affect system performance. Results are presented for both methods to show that they can be used to identify the same failure scenarios presented in our earlier paper (as well as additional scenarios, using the first method), while reducing the number of perturbations needed. We argue that these methods provide a basis for creating practical tools for analysis of complex systems behavior in distributed systems.

From the analysis of experimentally observed variations in surface strains with loading in reinforced concrete beams, it is noted that there is a need to consider the evolution of strains (with loading) as a stochastic process. Use of Markov Chains for modeling stochastic evolution of strains with loading in reinforced concrete flexural beams is studied in this paper. A simple, yet practically useful, bi-level homogeneous Gaussian Markov Chain (BLHGMC) model is proposed for determining the state of strain in reinforced concrete beams. The BLHGMC model will be useful for predicting behavior/response of reinforced concrete beams leading to more rational design.