Ramaswami's duality and probabilistic algorithms for determining the rate matrix for a structured GI/M/1 Markov chain

In this paper, we consider continuous‐time Markov chains with a finite state space under nonlinear expectations. We define so‐called Q‐operators as an extension of Q‐matrices or rate matrices to a nonlinear setup, where the nonlinearity is due to model uncertainty. The main result gives a full characterization of convex Q‐operators in terms of a positive maximum principle, a dual representation by means of Q‐matrices, time‐homogeneous Markov chains under convex expectations, and a class of nonlinear ordinary differential equations. This extends a classical characterization of generators of Markov chains to the case of model uncertainty in the generator. We further derive an explicit primal and dual representation of convex semigroups arising from Markov chains under convex expectations via the Fenchel–Legendre transformation of the generator. We illustrate the results with several numerical examples, where we compute price bounds for European contingent claims under model uncertainty in terms of the rate matrix.

Mathematically, stochastic hybrid systems can be described by Markov processes with discrete and continuous behaviours. This chapter provides in an incremental way the necessary mathematical background for understanding such complex systems. The chapter starts with discrete and continuous-time Markov chains. In this book, a Markov chain is understood as a stochastic process with the Markov property (or memoryless property, i.e. its future evolution depends only on the current state) defined on a discrete (finite or countable) state space. The Markov property is illustrated in the Chapman-Kolmogorov equation satisfied by the transition probabilities. Usually, in the continuous-time Markov case, the evolution of these transition probabilities is described by forward/backward Kolmogorov equations. These equations are expressed in terms of the stochastic matrix (called also infinitesimal generator) associated to the Markov chain. This is the matrix of transition rates and the practical use of Markov chains resides in our ability of handling it.The second part of this chapter is dedicated to Markov processes defined on continuous state spaces that can be equipped with additional structures as sigma-algebras, metrics, topologies, norms or others like specific algebraic structures (lattice, vector space). Characterisations of such processes are usually given using functional analysis operators like: the operator semigroup/resolvent, the infinitesimal generator or other operators that can be associated to a Markov process. At an abstract level, these operators describe the evolution the evolution of transition probabilities for a Markov process. Chapman-Kolmogorov equation, Kolmogorov equations and other properties of these operators (e.g. Dynkin formula, martingale problem) represent the natural tools for studying such Markov processes.KeywordsMarkov ChainTopological SpaceMarkov ProcessMarkov PropertyPolish SpaceThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

We study a supply chain consisting of a single manufacturer and two retailers. The manufacturer produces goods on a make-to-order basis, while both retailers maintain an inventory and use a periodic replenishment rule. As opposed to the traditional (r, S) policy, where a retailer at the end of each period orders the demand seen during the previous period, we assume that the retailers dampen their demand variability by smoothing the order size. More specifically, the order placed at the end of a period is equal to ß times the demand seen during the last period plus (1?ß) times the previous order size, with ß ? (0, 1] the smoothing parameter. We develop a GI/M/1-type Markov chain with only two nonzero blocks A 0 and A d to analyze this supply chain. The dimension of these blocks prohibits us from computing its rate matrix R in order to obtain the steady state probabilities. Instead we rely on fast numerical methods that exploit the structure of the matrices A 0 and A d , i.e., the power method, the Gauss-Seidel iteration and GMRES, to approximate the steady state probabilities. Finally, we provide various numerical examples that indicate that the smoothing parameters can be set in such a manner that all the involved parties benefit from smoothing. We consider both homogeneous and heterogeneous settings for the smoothing parameters.

We describe a computational procedure for evaluating the quasi-stationary distributions of a continuous-time Markov chain. Our method, which is an ‘iterative version' of Arnoldi's algorithm, is appropriate for dealing with cases where the matrix of transition rates is large and sparse, but does not exhibit a banded structure which might otherwise be usefully exploited. We illustrate the method with reference to an epidemic model and we compare the computed quasi-stationary distribution with an appropriate diffusion approximation.

The aim of this paper is to introduce a new model of a financial asset prices distribution. It is known that the probability distribution of an asset prices or returns is unknown in reality. The general model of asset prices based on continuous time Markov chains is proposed. For this reason the interarrivals between two price states are approximated by mixture of exponential distributions. Numerical-analytic approach is used to obtain the probability distribution of asset prices. The developed software allows creating the space of an asset prices, the matrix of transition rates among states, a system of equations to find the steady state probabilities of price states and solves the system of equations by method of imbedded Markov chains.

In this paper, we use the Markov chain censoring technique to study infinite state Markov chains whose transition matrices possess block-repeating entries. We demonstrate that a number of important probabilistic measures are invariant under censoring. Informally speaking, these measures involve first passage times or expected numbers of visits to certain levels where other levels are taboo;they are closely related to the so-called fundamental matrix of the Markov chain which is also studied here. Factorization theorems for the characteristic equation of the blocks of the transition matrix are obtained. Necessary and sufficient conditions are derived for such a Markov chain to be positive recurrent, null recurrent, or transient based either on spectral analysis, or on a property of the fundamental matrix. Explicit expressions are obtained for key probabilistic measures, including the stationary probability vector and the fundamental matrix, which could be potentially used to develop various recursivealgorithms for computing these measures.

A new method based on fault chains and the Markov process for analyzing time-delay power system stability is proposed in this article. First, reasonable fault chains are formed using the fault chain generation method based on network power flow. Second, by combining the fault chains and Markov processes, a class of Lyapunov–Krasovskii functionals considering the Markov jump is established. Free weighting items constructed with the Markov process transfer rate matrix and Newton–Leibniz formula, respectively, are then introduced to the weak infinitesimal operators in the Lyapunov–Krasovskii functional. Meanwhile, to lower the conservativeness of the method, the whole time-varying time-delay interval has been divided into two subintervals. Finally, the time-delay upper bound that the power system can bear in the case of cascading faults is calculated by means of the generalized eigenvalue problem. The efficiency and feasibility of the proposed method in solving the maximum time delay in the case of cascading faults have been validated by the dynamic time-domain simulation results. Comparative results with other methods show the proposed method has low conservativeness.

Many approaches to tackle the state-space explosion problem of Markov chains are based on the notion of lumpability (a.k.a. probabilistic bisimulation), which allows computation of measures using the quotient Markov chain, which, in some cases, has much smaller state space than the original one. We present a new signature-based algorithm for computing the optimal (i.e., smallest possible) quotient Markov chain, prove its correctness, and implement it symbolically for Markov chains represented as Multi-Terminal BDDs (MTBDDs). The algorithm is very time-efficient because we translate the core operation of the algorithm, i.e., the computation of the signatures, into symbolic operations. Our experiments on various configurations of three example models with different levels of lump ability show that the algorithm (1) handles significantly larger state spaces than an explicit algorithm, (2) outperforms a very efficient explicit algorithm for significantly lump able Markov chains while it is not prohibitively slower in the worst case, and (3) outperforms our previous optimal symbolic algorithm [10] in terms of running time although it has higher space requirement for most of the configurations.

Many approaches to tackle the state-space explosion problem of Markov chains are based on the notion of lumpability (a.k.a. probabilistic bisimulation), which allows computation of measures using the quotient Markov chain, which, in some cases, has much smaller state space than the original one. We present a new signature-based algorithm for computing the optimal (i.e., smallest possible) quotient Markov chain, prove its correctness, and implement it symbolically for Markov chains represented as Multi-Terminal BDDs (MTBDDs). The algorithm is very time-efficient because we translate the core operation of the algorithm, i.e., the computation of the signatures, into symbolic operations. Our experiments on various configurations of three example models with different levels of lump ability show that the algorithm (1) handles significantly larger state spaces than an explicit algorithm, (2) outperforms a very efficient explicit algorithm for significantly lump able Markov chains while it is not prohibitively slower in the worst case, and (3) outperforms our previous optimal symbolic algorithm [10] in terms of running time although it has higher space requirement for most of the configurations.

The use of probabilistic ideas by applied mathematicians has seen a continued increase in recent decades. Probability now appears frequently at the modeling stage. There is widespread interest in investigating the effects of noise and uncertainty. Probabilistic algorithms are routinely applied with much success to the solution of deterministic problems (think of Monte Carlo quadrature or stochastic gradient descent). Markov chains give a simple, widely used tool to describe systems that evolve randomly. They are also a very popular topic in applied mathematics courses. At times \(0\), \(1\), \(2\), łdots, a Markov chain jumps from its current state \(x\) to a randomly chosen new state \(y\); the set of possible states is discrete. The requirement that the jumping times be uniformly spaced is a clear limitation in many situations, and this shortcoming is avoided by considering continuous-time Markov chains. In the continuous-time setting, if the chain has reached state \(x\) at time \(t_i\), it will jump to the randomly chosen \(y\) at time \(t_i+1=t_i+\tau_i\), where the waiting time \(\tau_i>0\) is itself random. Markov chains in continuous time are featured in many applications, including chemistry, ecology, and epidemiology. In a chemical system containing \(n\) species \(S_1\), łdots, \(S_n\), each state corresponds to vector \((c_1\), łdots, \( c_n)\) where \(c_j\) is the number of molecules of species \(S_j\). The species may take part in a number of chemical reactions, let us say \(S_1+2S_2 \rightarrow S_3\), \(2S_1+3S_3 \rightarrow S_4+2S_6\), and so on. At random times, one or another of the \(m\) possible reactions will take place and the state will change. Such a stochastic, molecular approach typically provides a more accurate description of the system than deterministic treatments where the concentrations of the different species are regarded as continuous variables that evolve according to a set of differential equations. This is particularly true in systems where, for some species, the number \(c_j\) is low, as is the case in many biological reactions. The following Survey and Review paper, “Stationary Distributions of Continuous-Time Markov Chains: A Review of Theory and Truncation-Based Approximations,” has been written by Juan Kuntz, Philipp Thomas, Guy-Bart Stan, and Mauricio Barahona. Section 2 provides a reader-friendly introduction to continuous-time Markov chains and their stationary distributions; these are important because they determine the long-time behavior of the chain. A salient future is that the authors present, in an accessible way, results that do not assume the chain to be irreducible (roughly speaking, irreducibility means that the chain is not the juxtaposition of two or more smaller chains that do not talk to each other; irreducibility simplifies the math, but is not a reasonable hypothesis in some applications). After that, the authors concentrate on how to compute invariant distributions. The paper contains numerical results, an extensive bibliography, and a detailed discussion of open problems.

Hitting times, number of jumps, and occupation times for continuous-time finite state Markov chains

In this paper, a general stabilization problem of stochastic delay systems is realized by a disordered controller and studied by exploiting the disorder-dependent approach. Different from the traditional results, the stabilizing controller here experiences a disorder between control gains and system states. Firstly, the above disorder is described by the robust method, whose probability distribution is embodied by a Markov process with two modes. Based on this description, a kind of disordered controller having special uncertainties and depending on a Markov process is proposed. Then, by exploiting a disorder-dependent Lyapunov functional, two respective conditions for the existence of such a disordered controller are provided with LMIs. Moreover, the presented results are further extended to a general case that the corresponding transition rate matrix of the disordered controller has uncertainties. Finally, a numerical example is exploited to demonstrate the effectiveness and superiority of the proposed methods.

Article Free Access Share on Exact sampling and approximate counting techniques Author: Mark Huber School of Operations Research and Industrial Engineering, Cornell University, Ithaca, NY School of Operations Research and Industrial Engineering, Cornell University, Ithaca, NYView Profile Authors Info & Claims STOC '98: Proceedings of the thirtieth annual ACM symposium on Theory of computingMay 1998 Pages 31–40https://doi.org/10.1145/276698.276709Published:23 May 1998Publication History 25citation666DownloadsMetricsTotal Citations25Total Downloads666Last 12 Months66Last 6 weeks15 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteeReaderPDF

In this paper, the deleterious effects of obesity, type II diabetes, and insulin resistance, systolic and diastolic hypertension on the rate of progression of fibrosis in patients with non-alcoholic fatty liver disease (NAFLD) are illustrated using a new approach utilizing the Poisson regression to model the transition rate matrix. The observed counts in the transition count matrix are used as the response variables and the covariates are the risk factors for fatty liver. Then, the estimated counts from running the Poisson regression are used to estimate the transition rates using the continuous-time Markov chains (CTMCs) followed by exponentiation of the estimated rate matrix to obtain the transition probability matrix at specific time points. A depicted, hypothetical, observational, prospective longitudinal study of 150 participants followed up every year for a total of 29 years recording their demographic characteristics and their timeline follow-up is demonstrated. The findings revealed that insulin resistance expressed by HOMA2-IR had the most deleterious effects among other factors on increasing the rate of fibrosis progression from state 1 to state 2, from state 2 to state 3, and from state 3 to state 4. The higher the level of HOMA2-IR is, the more rapid the rate of progression is. This analysis helps the health policymakers and medical insurance managers to allocate the financial and human resources for investigating and treating high-risk patients with NAFLD. In addition, this analysis can be used by pharmaceutical companies to conduct longitudinal studies to assess the effectiveness of the newly emerging anti-fibrotic drugs.

The authors consider Bayesian analysis for continuous‐time Markov chain models based on a conditional reference prior. For such models, inference of the elapsed time between chain observations depends heavily on the rate of decay of the prior as the elapsed time increases. Moreover, improper priors on the elapsed time may lead to improper posterior distributions. In addition, an infinitesimal rate matrix also characterizes this class of models. Experts often have good prior knowledge about the parameters of this matrix. The authors show that the use of a proper prior for the rate matrix parameters together with the conditional reference prior for the elapsed time yields a proper posterior distribution. The authors also demonstrate that, when compared to analyses based on priors previously proposed in the literature, a Bayesian analysis on the elapsed time based on the conditional reference prior possesses better frequentist properties. The type of prior thus represents a better default prior choice for estimation software.