An Introduction to Markov Chain Monte Carlo

We study new formulae based on Lya- punov exponents for entropy, mutual information, and capacity of finite state discrete time Markov chan- nels. We also develop a method for directly com- puting mutual information and entropy using contin- uous state space Markov chains. We show that the entropy rate for a symbol sequence is equal to the primary Lyapunov exponent for a product of random matrices. We then develop a continuous state space Markov chain formulation that allows us to directly compute entropy rates as expectations with respect to the Markov chain's stationary distribution. We also show that the stationary distribution is a continuous function of the input symbol dynamics. This continu- ity allows the channel capacity to be written in terms of Lyapunov exponents.

This review article is concerned to provide a global context to several works on the fitting of continuous time nonhomogeneous Markov chains with finite state space and also to point out some selected aspects of two techniques previously introduced—estimation and calibration—relevant for applications and used to fit a continuous time Markov chain model to data by the adequate selection of parameters. The denomination estimation suits the procedure better when statistical techniques—e.g., maximum likelihood estimators—are employed, while calibration covers the case where, for instance, some optimisation technique finds a best approximation parameter to ensure good model fitting. For completeness, we provide a short summary of well-known important notions and results formulated for nonhomogeneous Markov chains that, in general, can be transferred to the homogeneous case. Then, as an illustration for the homogeneous case, we present a selected Billingsley’s result on parameter estimation for irreducible chains with finite state space. In the nonhomogeneous case, we quote two recent results, one of the calibration type and the other with more of a statistical flavour. We provide an ample set of bibliographic references so that the reader wanting to pursue her/his studies will be able to do so more easily and productively.

Markov Chain Monte Carlo (MCMC) is widely applied to fit complex, practical models in various disciplines, such as engineering, physics, agriculture, etc. Indeed, when the target density $f$ is not available in closed form, a Markov chain is constructed with stationary density $f$ for forming Monte Carlo estimators of means with respect to $f$. Although a Harris ergodic Markov chain will converge to the target distribution as the Monte Carlo sample size tends to infinity, and the population means can be consistently estimated by the corresponding MCMC estimators, in practice, it is important to estimate the standard errors associated with these Monte Carlo estimates due to the availability of only a finite sample. Therefore, the geometric ergodicity of a Markov Chain is highly desired as it is the most standard method for guaranteeing a central limit theorem (CLT) for MCMC estimators and finding its standard errors. Among the different MCMC algorithms, Metropolis-Hastings (MH) algorithms are predominant. This thesis establishes conditions under which MH algorithms with a position-dependent proposal covariance matrix will or will not have the geometric convergence rate. Indeed, we provide these results for MH algorithms where the normal proposal density has a general mean function $c(x)$ and a covariance matrix $G(x)$ depending on the current position $x$ of the Markov chain. For modern variants of the popular Metropolis adjusted Langevin algorithms (MALA), like the manifold MALA (MMALA) that adapt to the geometry of the target distributions, the proposal covariance matrix changes in every iteration. Thus, we provide conditions for the geometric ergodicity of different variations of the Langevin algorithms, e.g., MMALA, position-dependent MALA (PMALA), and the pre-conditioned MALA (PCMALA). We also provide conditions guaranteeing geometric ergodicity of the pre-conditioned unadjusted Langevin algorithms (PCULA). Generalized linear mixed models (GLMMs), where the response variable is allowed to follow a non-Gaussian distribution, are popular statistical models, although their likelihood functions are not available in closed form. GLMMs can handle the over-dispersion often present in real-world data, and MCMC methods are generally used to fit frequentist and Bayesian GLMMs. Since the likelihood function for GLMMs is not available in closed form, Monte Carlo EM or Monte Carlo maximum likelihood methods are often used for making inferences from GLMMs. Simulation from the conditional (on the data) distribution of the random effects is, in turn, needed for implementing the Monte Carlo EM or Monte Carlo maximum likelihood methods. In this thesis, our general conditions for geometric ergodicity of variants of Langevin algorithms are also verified in the context of conditional simulation from the two most popular GLMMs, namely the binomial GLMM with the logit link and the Poisson GLMM with the log link. These results have important practical implications as users can now confidently fit these models by attaching asymptotically valid standard errors with their Markov chain-based Monte Carlo estimates. Spatial generalized linear mixed models (SGLMMs) are GLMMs where the random effects form a spatial process and are useful for modeling spatially correlated data. We compare the performance of different variants of MALA mentioned above as well as other popular MH algorithms like Random Walk Metropolis in the context of conditional simulation from SGLMMs. Next, we consider MCMC algorithms for full Bayesian analysis of GLMMs. Blocking and reparameterization are techniques used to improve the efficiency and convergence properties of MCMC algorithms. We combine these strategies with pre-conditioning to develop efficient Langevin algorithms for Bayesian GLMMs and SGLMMs. We illustrate our algorithms using extensive real and simulated data analyses.

The Markov Chain Monte Carlo (MCMC) methods have become indispensable tools in modern statistical computation, enabling researchers to approximate complex probability distributions that are otherwise intractable. This paper focus on MCMC in Statistics and Probability area which is used to draw samples from a probability distribution. In order to introduce this algorithm in a relatively light and straightforward way, this paper breaks the content into two parts: Markov Chain and MCMC, and brings in stochastic process, Markov property, Ordinary Monte Carlo, and Monte Carlo Integration in succession. On this foundation, this paper formally introduces two algorithms of MCMC that are Metropolis-Hasting Algorithm and Gibbs Sampling, the most famous and significant algorithms of MCMC. With this approach, this paper concludes both the advantages and disadvantages of these two algorithms, laying the foundation of the further study and the deeper utilizations of MCMC and, hopefully, contributing to providing a certain degree of assistance for scholars.

In this article, a multiuser cooperative nonorthogonal multiple access (NOMA) network is considered. In order to improve their battery lifetime, the near-user (NU) Internet of Things (IoT) nodes relay information to the far-user (FU) IoT nodes using only the energy harvested from the ambience. The harvested energy at all the NU IoT nodes is stored in energy buffers. To improve the throughput performance, the best NU and best FU (BNBF) user selection scheme is employed. Moreover, unlike conventional orthogonal multiple access (OMA) networks, an adaptive NOMA network is considered in this article that switches between direct NOMA, NOMA with relaying, and OMA modes to maximize throughput. To improve accuracy and reduce the computational complexity, the energy buffer states at the NU IoT nodes are modeled using a continuous state-space Markov chain (CSMC) instead of discrete state-space Markov chain (DSMC). We derive the limiting distributions of the stored energy in energy buffers with either the best effort policy (BEP) or the on–off policy (OOP) applied for buffer energy management. Expressions for throughput are derived for both harvest-store-use (HSU) and harvest-use (HU) architectures. The Monte Carlo simulations are used to validate the derived analytical expressions.

As themany examples in this book illustrate,Markov chainMonte Carlo (MCMC)methods have revolutionized Bayesian statistical analyses. Rather than using off-the-shelf models and methods, we can use MCMC to fit application-specific models that are designed to account for the particular complexities of a problem. These complex multilevel models are becoming more prevalent throughout the natural, social, and engineering sciences largely because of the ease of using standard MCMC methods such as the Gibbs and MetropolisHastings (MH) samplers. Indeed, the ability to easily fit statistical models that directly represent the complexity of a data-generationmechanismhas arguably lead to the increased popularity of Bayesian methods in many scientific disciplines. Although simple standard methods work surprisingly well in many problems, neitherthe Gibbs nor theMH sampler can directly handle problemswith very high posterior correlations among theparameters. Themarginal distribution of a givenparameter ismuchmore variable than the corresponding full conditional distribution in this case, causing the Gibbs sampler to take small steps. With MH a proposal distribution that does not account for the posterior correlation either has far toomuchmass in regions of low posterior probability or has such smallmarginal variances that only small steps are proposed, causing high rejection rates and/or high autocorrelations in the resultingMarkov chains. Unfortunately, accounting for the posterior correlation requires more information about the posterior distribution than is typically available when the proposal distribution is constructed. Much work has been devoted to developing computational methods that extend theusefulness of these standard tools in the presence of high correlations. For Gibbs sampling, for example, it is nowwell known that blocking or grouping steps (Liu et al., 1994), nesting steps (van Dyk, 2000), collapsing or marginalizing parameters (Liu et al., 1994; Meng and vanDyk, 1999), incorporatingauxiliaryvariables (BesagandGreen, 1993), certainparameter transformations (Gelfand et al., 1995; Yu and Meng, 2011), and parameter expansion (Liu andWu,1999) canall beused to improve the convergenceof certain samplers.Byembedding an MH sampler within the Gibbs sampler and updating one parameter at a time (i.e. the well-known Metropolis-within-Gibbs sampler in the terminology of Gilks et al., 1995), the same strategies can be used to improve MH samplers.

ABSTRACTIn epidemic modelling, the emergence of a disease is characterized by the low numbers of infectious individuals. Environmental randomness impacts outcomes such as the outbreak or extinction of the disease in this case. This randomness can be accounted for by modelling the system as a continuous time Markov chain, . The probability of extinction given some initial state is the probability of hitting a subset of the state space associated with extinction for the initial state. This hitting probability can be studied by passing to the discrete time Markov chain (DTMC), . An approach is presented to approximate a DTMC on a countably infinite state space by a DTMC on a finite state space for the purpose of solving general hitting problems. This approach is applied to approximate the probability of disease extinction in an epidemic model. It is also applied to evaluate a heterogeneous disease control strategy in a metapopulation.

Bayesian statistics has attracted people's interest again recently because of the application of Markov Chain Monte Carlo (MCMC) theory. In particle filtering, the diversification of particles disappears in the process of importance sampling. However, this problem can be solved using Metropolis-Hastings (MH) sampling usually used in MCMC theory. As a modification to added MCMC (AMCMC)—an improved MCMC particle filter that can track variable number of targets at the same time, a new approach to optimize those rejected samples in MH sampling process by Mean Shift algorithm is proposed in this paper. Because the operation rate of particles in AMCMC is increased, the circles of sampling needed for the convergence of Markov chain is reduced. It is shown by experiment that, the optimized algorithm has better tracking performance under the condition of fewer particles.

In many practices of bridge asset management, life cycle costs are estimated by statistical deterioration prediction models based upon monitoring data collected through inspection activities. In many applications, it is, however, often the case that the validity of statistical deterioration prediction models is flawed by an inadequate stock of inspection dates. In this paper, a systematic methodology is presented to provide estimates of the deterioration process for bridge managers based upon empirical judgments at early stages by experts, and whereby revisions may be made as new data are obtained through later inspections. More concretely, Bayesian estimation methodology is developed to improve the estimation of Markov transition probability of the multi-stage exponential Markov model by Markov chain Monte Carlo method using Gibbs sampling. The paper concludes with an empirical example, using the real world monitoring data, to demonstrate the applicability of the model and its Bayesian estimation method in the case of incomplete monitoring data.

PurposeThe application of statistical inversion theory provides a powerful approach for solving estimation problems including the ability for uncertainty quantification (UQ) by means of Markov chain Monte Carlo (MCMC) methods and Monte Carlo integration. This paper aims to analyze the application of a state reduction technique within different MCMC techniques to improve the computational efficiency and the tuning process of these algorithms.Design/methodology/approachA reduced state representation is constructed from a general prior distribution. For sampling the Metropolis Hastings (MH) Algorithm and the Gibbs sampler are used. Efficient proposal generation techniques and techniques for conditional sampling are proposed and evaluated for an exemplary inverse problem.FindingsFor the MH-algorithm, high acceptance rates can be obtained with a simple proposal kernel. For the Gibbs sampler, an efficient technique for conditional sampling was found. The state reduction scheme stabilizes the ill-posed inverse problem, allowing a solution without a dedicated prior distribution. The state reduction is suitable to represent general material distributions.Practical implicationsThe state reduction scheme and the MCMC techniques can be applied in different imaging problems. The stabilizing nature of the state reduction improves the solution of ill-posed problems. The tuning of the MCMC methods is simplified.Originality/valueThe paper presents a method to improve the solution process of inverse problems within the Bayesian framework. The stabilization of the inverse problem due to the state reduction improves the solution. The approach simplifies the tuning of MCMC methods.

Osteoporosis (OP) is a common metabolic bone disease with high incidence, and is recognized as a major public health problem worldwide. It is essential to clarify the pathogenesis of the disease for improving the diagnosis, prevention and treatment of OP. The aim of this study was to clarify the pivotal pathways and hub genes in OP using Gibbs sampling. The gene expression profile datasets were obtained from Gene Expression Omnibus (GEO) database. The pathways were enriched by Kyoto Encyclopedia of Genes and Genomes (KEGG) with genes intersection ≥5 based on gene expression profile data. Then, the acquired pathways were converted into Markov chains (MC). Gibbs sampling was conducted to obtain a new MC. In addition, the average probabilities of each pathway in two states containing human mesenchymal stem cells (hMSC) _middle-aged and hMSC_elderly were calculated through Markov chain Monte Carlo (MCMC) algorithm. Moreover, gene expression variation was taken into account to adjust the probability. Pivotal pathways were identified under adjusted posterior value >0.8. Then, Gibbs sampling was implemented to find hub genes from pathways. There were 280 pathways determined by the gene intersection ≥5. Gibbs sampling identified two disturbed pathways (pathways in cancer and influenza A) and two hub genes (cyclin A1 and WNT2) under the adjusted probability >0.8. Gene expression analysis showed that all the disturbed pathways and hub genes had increased expression levels in hMSC_middle-aged samples compared with hMSC_elderly samples. We identified two pivotal pathways and two hub genes in OP using Gibbs sampling. The results contribute to the understanding of underlying pathogenesis and could be considered as potential biomarkers for the therapy of OP.

Groundwater pollution source identification using Metropolis-Hasting algorithm combined with Kalman filter algorithm

This paper presents an overview of Markov Chain Monte Carlo (MCMC) methods for statistical inference and applications. The article begins by describing ordinary Monte Carlo methods, which in principle has the same goals as the MCMC but can hardly be implemented in practice. Following that basic Markov Chain Monte Carlo is discussed, which is founded on the Hastings algorithm and includes Metropolis method and the Gibbs sampler as special cases. Finally, various special applications of Markov Chain Monte Carlo methods are briefly mentioned and some recent development of MCMC are covered in final remarks section.KeywordsReal EstateMarkov Chain Monte CarloGibbs SamplerDetailed BalanceMarkov Chain Monte Carlo MethodThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

The prediction of numbers of newborns is an important issue in hospital management. Relying on the inherent non-aftereffect property, discrete-time Markov chain (DTMC) is a candidate for solving the problem. But the classical DTMC is unable to handle the uncertainty of states, especially when the state space is not discrete, which would lead to instable predicted results. In order to overcome the limitation of the existing DTMC model, a belief Markov chain (BMC) model is proposed by synthesizing the classical DTMC and Dempster-Shafer theory effectively. Depending on the advantages of Dempster-Shafer theory in expressing uncertainty, the proposed BMC model is capable of dealing with various uncertainties, which improves and perfects the classical DTMC model. An illustrative example demonstrates the effectiveness of the proposed model. Moreover, a comparison between the proposed BMC model and the classical and fuzzy states modified DTMC models is given to show the superiority of the proposed model against the other two. Finally, the stability of the proposed model has been proven.

A Markov process in discrete time with a finite state space is controlled by choosing the transition probabilities from a given convex family of distributions depending on the present state. The immediate cost is prescribed for each choice and it is required to minimise the average expected cost over an infinite future. The paper considers a special case of this general problem and provides the foundation for a general solution. The main result is that an optimal policy exists if each state of the system can be reached with positive probability from any other state by choosing a suitable policy.