Transformations that Preserve Detailed Balance in Markov Models

We present in this paper a mathematical method based on stochastic comparisons of Markov chains in order to compute in a mobile network performance measures that are important Q parameters for users: the dropping handover of voice and the blocking probability of a new voice call. The key idea of this methodology is that given a complex system represented by a multi-dimensional Markov chain which is too large to be solved, we propose to reduce the state space, and so to define a new Markov chain which is a simplified version of the original one. This reduced Markov chain is defined as an aggregated one, which represents a stochastic bound for performance measures written as increasing reward function on the stationary distribution. The main steps of the construction of the aggregated Markov chain applied into mobile networks are presented in this paper. As the number of mobile users with different kinds of applications increases, the associated model is more complex and it can be represented by multi-dimensional continuous time Markov chain with a very large size. Thus, we define an aggregated Markov chain represented by a multi-dimensional birth and death process which is very easy to solve. We have proved that there is a weak ordering between the original Markov chain and the aggregated one using increasing set formalism. We have computed upper bounds of dropping handover and blocking probability for different values of input parameters. Numerical results prove that upper bounds give good results and so the stochastic methodology is an interesting mathematical tool for the performance evaluation of complex systems. Keywords: Mobile Networks, Quality of Service, Stochastic ordering, stochastic comparisons, Continuous Time Markov Chains.

The contribution first describes the implementation of Continuous-Time Markov Chains (CTMC) in the language Modelica; this enables the simulation by powerful solvers making typical Functional Safety applications numerically more efficient and reliable than with classically used Discrete-Time Markov Chains (DTMC). Secondly, the CTMC are extended to Multi-Phase Markov Chains (CTMC MP ) enabling the inclusion of maintenance (restoration) actions as well as online diagnostics, which is finally elaborated to the online prediction of an individual system's safety quantities.

This work develops asymptotic expansions for solutions of systems of backward equations of time-inhomogeneous Markov chains in continuous time. Owing to the rapid progress in technology and the increasing complexity in modeling, the underlying Markov chains often have large state spaces, which make the computational tasks infeasible. To reduce the complexity, two-time-scale formulations are used. By introducing a small parameter ɛ > 0 and using suitable decomposition and aggregation procedures, it is formulated as a singular perturbation problem. Both Markov chains having recurrent states only and Markov chains including also transient states are treated. Under certain weak irreducibility and smoothness conditions of the generators, the desired asymptotic expansions are constructed. Then error bounds are obtained.

We study a quantum entanglement switch that serves $k$ users in a star topology. We model variants of the system using Markov chains and standard queueing theory and obtain expressions for switch capacity and the expected number of qubits stored in memory at the switch. While it is more accurate to use a discrete-time Markov chain (DTMC) to model such systems, we quickly encounter practical constraints of using this technique and switch to using continuous-time Markov chains (CTMCs). Using CTMCs allows us to obtain a number of analytic results for systems in which the links are homogeneous or heterogeneous and for switches that have infinite or finite buffer sizes. In addition, we can model the effects of decoherence of quantum states fairly easily using CTMCs. We also compare the results we obtain from the DTMC against the CTMC in the case of homogeneous links and infinite buffer, and learn that the CTMC is a reasonable approximation of the DTMC. From numerical observations, we discover that decoherence has little effect on capacity and expected number of stored qubits for homogeneous systems. For heterogeneous systems, especially those operating close to stability constraints, buffer size and decoherence can have significant effects on performance metrics. We also learn that in general, increasing the buffer size from one to two qubits per link is advantageous to most systems, while increasing the buffer size further yields diminishing returns.

Pricing American Drawdown Options under Markov Models

Pricing American drawdown options under Markov models

EAPRAD: A MAC protocol for Enhancing Access Probability and Reducing Access Delay in VANETs

We consider continuous and impulse control of a Markov chain (MC) with a finite set of states in continuous time. Continuous control determines the intensity of transitions between MC states, while transition times and their directions are random. Nevertheless, sometimes it is necessary to ensure a transition that leads to an instantaneous change in the state of the MC. Since such transitions require different influences and can produce different effects on the state of the MC, such controls can be interpreted as impulse controls. In this work, we use the martingale representation of a controllable MC and give an optimality condition, which, using the principle of dynamic programming, is reduced to a form of quasi-variational inequality. The solution to this inequality can be obtained in the form of a dynamic programming equation, which for an MC with a finite set of states reduces to a system of ordinary differential equations with one switching line. We prove a sufficient optimality condition and give examples of problems with deterministic and random impulse action.

The use of probabilistic ideas by applied mathematicians has seen a continued increase in recent decades. Probability now appears frequently at the modeling stage. There is widespread interest in investigating the effects of noise and uncertainty. Probabilistic algorithms are routinely applied with much success to the solution of deterministic problems (think of Monte Carlo quadrature or stochastic gradient descent). Markov chains give a simple, widely used tool to describe systems that evolve randomly. They are also a very popular topic in applied mathematics courses. At times \(0\), \(1\), \(2\), łdots, a Markov chain jumps from its current state \(x\) to a randomly chosen new state \(y\); the set of possible states is discrete. The requirement that the jumping times be uniformly spaced is a clear limitation in many situations, and this shortcoming is avoided by considering continuous-time Markov chains. In the continuous-time setting, if the chain has reached state \(x\) at time \(t_i\), it will jump to the randomly chosen \(y\) at time \(t_i+1=t_i+\tau_i\), where the waiting time \(\tau_i>0\) is itself random. Markov chains in continuous time are featured in many applications, including chemistry, ecology, and epidemiology. In a chemical system containing \(n\) species \(S_1\), łdots, \(S_n\), each state corresponds to vector \((c_1\), łdots, \( c_n)\) where \(c_j\) is the number of molecules of species \(S_j\). The species may take part in a number of chemical reactions, let us say \(S_1+2S_2 \rightarrow S_3\), \(2S_1+3S_3 \rightarrow S_4+2S_6\), and so on. At random times, one or another of the \(m\) possible reactions will take place and the state will change. Such a stochastic, molecular approach typically provides a more accurate description of the system than deterministic treatments where the concentrations of the different species are regarded as continuous variables that evolve according to a set of differential equations. This is particularly true in systems where, for some species, the number \(c_j\) is low, as is the case in many biological reactions. The following Survey and Review paper, “Stationary Distributions of Continuous-Time Markov Chains: A Review of Theory and Truncation-Based Approximations,” has been written by Juan Kuntz, Philipp Thomas, Guy-Bart Stan, and Mauricio Barahona. Section 2 provides a reader-friendly introduction to continuous-time Markov chains and their stationary distributions; these are important because they determine the long-time behavior of the chain. A salient future is that the authors present, in an accessible way, results that do not assume the chain to be irreducible (roughly speaking, irreducibility means that the chain is not the juxtaposition of two or more smaller chains that do not talk to each other; irreducibility simplifies the math, but is not a reasonable hypothesis in some applications). After that, the authors concentrate on how to compute invariant distributions. The paper contains numerical results, an extensive bibliography, and a detailed discussion of open problems.

Jump statistics, sojourn times, fluctuation dynamics and ergodic behavior for Markov processes in continuous time with a finite number of states

A general framework is proposed for pricing both continuously and discretely monitored Asian options under one-dimensional Markov processes. For each type (continuously monitored or discretely monitored), we derive the double transform of the Asian option price in terms of the unique bounded solution to a related functional equation. In the special case of continuous-time Markov chain (CTMC), the functional equation reduces to a linear system that can be solved analytically via matrix inversion. Thus the Asian option prices under a one-dimensional Markov process can be obtained by first constructing a CTMC to approximate the targeted Markov process model, and then computing the Asian option prices under the approximate CTMC by numerically inverting the double transforms. Numerical experiments indicate that our pricing method is accurate and fast under popular Markov process models, including the CIR model, the CEV model, Merton’s jump diffusion model, the double-exponential jump diffusion model, the variance gamma model, and the CGMY model.

CHAPTER 6 - Continuous-Time Markov Chains

We consider the susceptible-infective (SI) epidemiological model, a variant of the Kermack-McKendrick models, and let the contact rate be a function of the number of infectives, an indicator of disease spread during the course of the epidemic. We represent the resultant model as a continuous-time Markov chain. The result is a pure death (or birth) process with state-dependent rates, for which we find the probability distribution of the associated Markov chain by solving the Kolmogorov forward equations. This model is used to find the analytic solution of the SI model as well as the distribution of the epidemic duration. We use the maximum likelihood method to estimate contact rates based on observations of inter-infection time intervals. We compare the stochastic model to the corresponding deterministic models through a numerical experiment within a typical household. We also incorporate different intervention policies for vaccination, antiviral prophylaxis, isolation, and treatment considering both full and partial adherence to interventions among individuals.

Background and purpose Recent methodological reviews of evaluations of behaviour change interventions in public health have highlighted that the decay in effectiveness over time has been mostly overlooked, potentially leading to suboptimal decision-making. While, in principle, discrete-time Markov chains—the most commonly used modelling approach—can be adapted to account for decay in effectiveness, this framework inherently lends itself to strong model simplifications. The application of formal and more appropriate modelling approaches has been supported, but limited progress has been made to date. The purpose of this paper is to encourage this shift by offering a practical guide on how to model decay in effectiveness using a continuous-time Markov chain (CTMC)-based approach.MethodsA CTMC approach is demonstrated, with a contextualized tutorial being presented to facilitate learning and uptake. A worked example based on the stylized case study in physical activity promotion is illustrated with accompanying R code.DiscussionThe proposed framework presents a relatively small incremental change from the current modelling practice. CTMC represents a technical solution which, in absence of relevant data, allows for formally testing the sensitivity of results to assumptions regarding the long-term sustainability of intervention effects and improving model transparency.Conclusions The use of CTMC should be considered in evaluations where decay in effectiveness is likely to be a key factor to consider. This would enable more robust model-based evaluations of population-level programmes to promote behaviour change and reduce the uncertainty surrounding the decision to invest in these public health interventions.

We investigate the construction of a partial absorbing continuous-time Markov chain (CTMC) using a heuristic algorithm aimed at approximate transient analysis. The accuracy of transient state probabilities is indicated by the probability of absorbing state(s) at the specified time moment. A key challenge is the construction of a partial CTMC that minimizes the probability of reaching the absorbing state(s). The generation of all possible partial CTMCs is too computationally demanding, in general. Thus, we turn to investigation of heuristic algorithms that chose to include one state at a time based on limited information (i.e., the partial chain that is already constructed) and without any assumptions about the structure of the underlying CTMC. We consider three groups of such algorithms: naive, based on state characterization by the shortest path (obtained by Dijkstra method) and based on exact/approximate state probabilities. After introducing the algorithms, we discuss the problem of optimal partial CTMC construction and provide several examples. Then we compare the algorithm performance by constructing the partial CTMCs for two models: car sharing system and a randomly generated CTMC. Our obtained numerical results suggest that heuristic algorithms using state characterization via the shortest path offer a balance between accuracy and computational effort.