Comparison of Semi-Markov and Markov Processes

6 Dependence notions in reliability theory

The distribution theory for reward functions on semi-Markov processes has been of interest since the early 1960s. The relevant asymptotic distribution theory has been satisfactorily developed. On the other hand, it has been noticed that it is difficult to find exact distribution results which lead to the effective computation of such distributions. Note that there is no satisfactory exact distribution result for rewards accumulated over deterministic time intervals [0, t], even in the special case of continuous-time Markov chains. The present paper provides neat general results which lead to explicit closed-form expressions for the relevant Laplace transforms of general reward functions on semi-Markov and Markov additive processes.

The distribution theory for reward functions on semi-Markov processes has been of interest since the early 1960s. The relevant asymptotic distribution theory has been satisfactorily developed. On the other hand, it has been noticed that it is difficult to find exact distribution results which lead to the effective computation of such distributions. Note that there is no satisfactory exact distribution result for rewards accumulated over deterministic time intervals [0, t], even in the special case of continuous-time Markov chains. The present paper provides neat general results which lead to explicit closed-form expressions for the relevant Laplace transforms of general reward functions on semi-Markov and Markov additive processes.

A Markov Renewal Process (M.R.P.) is a process similar to a Markov chain, except that the time required to move from one state to another is not fixed, but is a random variable whose distribution may depend on the two states between which the transition is made. For an M.R.P. ofm (<∞) states we derive a goodness-of-fit test for a hypothetical matrix of transition probabilities. This test is similar to the test Bartlett has derived for Markov chains. We calculate the first two moments of the test statistic and modify it to fit the moments of a standard χ2. Finally, we illustrate the above procedure numeerically for a particular case of a two-state M.R.P.

If (XN)N=1,2,... is a sequence of Markov processes which solve the martingale problems for some operators (GN)N=1,2,..., it is a classical task to derive a limit result as N→∞, in particular a weak process limit with limiting operator G. For slow-fast systems XN=(VN,ZN) where VN is slow and ZN is fast, GN consists of two (or more) terms, and we are interested in weak convergence of VN to some Markov process V. In this case, for some f∈D(G), the domain of G, depending only on v, the limit Gf can sometimes be derived by using some gN→0 (depending on v and z), and study convergence of GN(f+gN)→Gf. We develop this method further in order to obtain functional Laws of Large Numbers (LLNs) and Central Limit Theorems (CLTs). We then apply our general result to various examples from Chemical Reaction Network theory. We show that we can rederive most limits previously obtained, but also provide new results in the case when the fast-subsystem is first order. In particular, we allow that fast species to be consumed faster than they are produced, and we derive a CLT for Hill dynamics with coefficient 2.

The tail decay of the hitting probabilities on level zero in M/G/1-type Markov renewal processes (MRPs) is studied. The Markov renewal process is transformed into a Markov chain so that the problem of tail decay is reformulated in terms of the decay of the coefficients of a suitable power series. The latter problem is reduced to analyze the analyticity domain of the power series. Two numerical methods are introduced to compute the convergence radius of the latter power series. Some numerical experiments are presented that show the effectiveness of the proposed algorithms.

In conjunction with the 15th International Conference of Numerical Analysis and Applied Mathematics (ICNAAM 2017)

We propose a system approach to the asymptotic analysis of stochastic systems in the scheme of series with averaging and diffusion approximation. Stochastic systems are defined by Markov processes with locally independent increments in a Euclidean space with random switchings that are described by jump Markov and semi-Markov processes. We use the asymptotic analysis of Markov and semi-Markov random evolutions. We construct the diffusion approximation using the asymptotic decomposition of generating operators and solutions of problems of singular perturbation for reducibly inverse operators.

Markov and Semi-Markov Processes with Fuzzy States. Part 1. Markov Processes

One presents the results obtained recently by the collaborators of the Institute of Mathematics, Academy of Sciences of the Ukrainian SSR, regarding limit theorems for additive functionals of Markov and semi-Markov processes. One makes use of the theory of inversion of singularly perturbed semigroups of operators in the phase extension scheme and of the methods of asymptotic analysis of singularly perturbed Markov renewal equations.

A fundamental identity, due to Miller (1961a), (1962a, b) and Kemperman (1961), is generalized to semi-Markov processes. Thus the identity applies to processes defined on a Markov chain with discrete state space and random walks with Markov dependent steps (Section 2). Wald's identity is discussed briefly in Section 3. Section 4 is a study of the maxima of partial sums, and Section 5 of maxima in a semi-Markov process.

At first there was the Markov property. The theory of stochastic processes, which can be considered as an exten- sion of probability theory, allows the modeling of the evolution of systems through the time. It cannot be properly understood just as pure mathemat- ics, separated from the body of experience and examples that have brought it to life. The theory of stochastic processes entered a period of intensive develop- ment, which is not finished yet, when the idea of the Markov property was brought in. Not even a serious study of the renewal processes is possible without using the strong tool of Markov processes. The modern theory of Markov processes has its origins in the studies by A. A: Markov (1856-1922) of sequences of experiments connected in a chain and in the attempts to describe mathematically the physical phenomenon known as Brownian mo- tion. Later, many generalizations (in fact all kinds of weakenings of the Markov property) of Markov type stochastic processes were proposed. Some of them have led to new classes of stochastic processes and useful applications. Let us mention some of them: systems with complete connections [90, 91, 45, 86]; K-dependent Markov processes [44]; semi-Markov processes, and so forth. The semi-Markov processes generalize the renewal processes as well as the Markov jump processes and have numerous applications, especially in relia- bility.

Chapter 1. Stepped Semi-Markov Processes. Chapter 2. Sequences of the First Exit Times and Regenerative Times. Chapter 3. Semi-Markov Processes of General Type. Chapter 4. Construction of the Process with Semi-Markov Transition Functions. Chapter 5. Semi-Markov Processes of Diffusion Type. Chapter 6. Time Change and Semi-Markov Processes. Chapter 7. Limit Theorems for Semi-Markov Processes. Chapter 8. Representing of the Semi-Markov Process as a Transformed Markov Process. Chapter 9. Semi-Markov Model of Chromatography. List of Authors. Index.

Reliability analysis of phased mission system with non-exponential and partially repairable components

Let be a non-decreasing Markov chain with a transition probability Q satisfying .Assuming that Q(x,By),where ,is non-increasing in x, and that Q(x,By+x) B is non-decreasing in x,we prove that ,where {Zi} is stochastically non-increasing and asymptotically i.i.d.,possessing a mixin property.We use this to show that (a.s) and that in probability,if ,where T(y) is the hitting time of (y,∞).We also show that is Markov renewal process where are sojourn times of