Generalized Bayesian Inference for Regression-Type Models with an Intractable Normalizing Constant

Simulating from distributions with intractable normalizing constants has been a long-standing problem in machine learning. In this letter, we propose a new algorithm, the Monte Carlo Metropolis-Hastings (MCMH) algorithm, for tackling this problem. The MCMH algorithm is a Monte Carlo version of the Metropolis-Hastings algorithm. It replaces the unknown normalizing constant ratio by a Monte Carlo estimate in simulations, while still converges, as shown in the letter, to the desired target distribution under mild conditions. The MCMH algorithm is illustrated with spatial autologistic models and exponential random graph models. Unlike other auxiliary variable Markov chain Monte Carlo (MCMC) algorithms, such as the Møller and exchange algorithms, the MCMH algorithm avoids the requirement for perfect sampling, and thus can be applied to many statistical models for which perfect sampling is not available or very expensive. The MCMH algorithm can also be applied to Bayesian inference for random effect models and missing data problems that involve simulations from a distribution with intractable integrals.

Bayesian inference for models with intractable likelihood functions represents a challenging suite of problems in modern statistics. In this work we analyse the Conway-Maxwell-Poisson (COM-Poisson) distribution, a two parameter generalisation of the Poisson distribution. COM-Poisson regression modelling allows the flexibility to model dispersed count data as part of a generalised linear model (GLM) with a COM-Poisson response, where exogenous covariates control the mean and dispersion level of the response. The major difficulty with COM-Poisson regression is that the likelihood function contains multiple intractable normalising constants and is not amenable to standard inference and Markov Chain Monte Carlo (MCMC) techniques. Recent work by Chanialidis et al. (2018) has seen the development of a sampler to draw random variates from the COM-Poisson likelihood using a rejection sampling algorithm. We provide a new rejection sampler for the COM-Poisson distribution which significantly reduces the central processing unit (CPU) time required to perform inference for COM-Poisson regression models. An extension of this work shows that for any intractable likelihood function with an associated rejection sampler it is possible to construct unbiased estimators of the intractable likelihood which proves useful for model selection or for use within pseudo-marginal MCMC algorithms (Andrieu and Roberts, 2009). We demonstrate all of these methods on a real-world dataset of takeover bids.

Identification of contaminant sources in rivers is crucial for river protection and emergency response. This study presents an innovative approach for identifying river pollution sources by using Bayesian inference and cellular automata (CA) modeling. A general Bayesian framework is proposed that combines the CA model with observed data to identify unknown sources of river pollution. To reduce the computational burden of the Bayesian inference, a CA contaminant transport model is developed to efficiently simulate pollutant concentration values in the river. These simulated concentration values are then used to calculate the likelihood function of available measurements. The Markov chain Monte Carlo (MCMC) method is used to produce the posterior distribution of contaminant source parameters, which is a sampling-based method that enables the estimation of complex posterior distributions. The suggested methodology is applied to a real case study of the Fen River in Yuncheng City, Shanxi Province, Northern China, and it estimates the release time, release mass, and source location with relative errors below 19%. The research indicates that the proposed methodology is an effective and flexible way to identify the location and concentrations of river contaminant sources.

Sampling technique is one of the asymptotically unbiased estimation approaches for inference in Bayesian probabilistic models. Markov chain Monte Carlo (MCMC) is a kind of sampling methods, which is widely used in the inference of complex probabilistic models. However, current MCMC methods can incur high autocorrelation of samples, which means that the samples generated by MCMC samplers are far from independent. In this paper, we propose the neural networks Langevin Monte Carlo (NNLMC) which makes full use of the flexibility of neural networks and the high efficiency of the Langevin dynamics sampling to construct a new MCMC sampling method. We propose the new update function to generate samples and employ appropriate loss functions to improve the performance of NNLMC during the process of sampling. We evaluate our method on a large diversity of challenging distributions and real datasets. Our results show that NNLMC is able to sample from the target distribution with low autocorrelation and rapid convergence, and outperforms the state-of-the-art MCMC samplers.

ABSTRACTA regression model with skew-normal errors provides a useful extension for ordinary normal regression models when the dataset under consideration involves asymmetric outcomes. In this article, we explore the use of Markov Chain Monte Carlo (MCMC) methods to develop a Bayesian analysis for joint location and scale nonlinear models with skew-normal errors, which relax the normality assumption and include the normal one as a special case. The main advantage of these class of distributions is that they have a nice hierarchical representation that allows the implementation of MCMC methods to simulate samples from the joint posterior distribution. Finally, simulation studies and a real example are used to illustrate the proposed methodology.

Airport air traffic is one of the most important and hardest one among all the airport data forecasts. In this paper, the Markov Chain Monte Carlo (MCMC) method of applied theory of statistics has been introduced into the aviation sector, and the discussion on airport air traffic forecast has been conducted taking Shanghai airport as the application background. MCMC method is designed to solve the problem parameters expectations where some baysian inference can not be directly calculated. This paper conducts the iterative computation using WinBUGS, EViews is used to estimate the initial iteration parameter of the regression model, and it is on this basis that the regression equation under the MCMC method is obtained.

Chapter 56 Computationally intensive methods for integration in econometrics

Estimation of dynamic mixture distributions is a difficult task, because the density contains an intractable normalizing constant. To overcome this difficulty, we develop an approach that maximizes, by means of the cross‐entropy method, a Monte Carlo approximation of the log‐likelihood function. The proposed noisy cross‐entropy approach is unsupervised, since it does not require the specification of a threshold between the distributions. Moreover, it bypasses the evaluation of the normalizing constant, combining good statistical properties with a modest computational burden. Both simulation‐based evidence and empirical applications suggest that noisy cross‐entropy estimation is comparable or preferable to existing methods in terms of statistical efficiency, but is less demanding from the computational point of view.

Bayesian inference for normal regression models, including sensitivity analysis, model comparison and error in variables under non-informative and conjugate prior for the model parameters has received considerable attention in the last decades. From a distributional point of view the results can be extended in several directions. One is by considering a wider class of prior distributions for the parameters of the model. Another, is by considering alternative distributions for the error terms. Usually, the results with non-conjugate priors rely heavily on MCMC methods. On the other hand, many extensions has been obtained by considering the so called dependent elliptical model, which is often used in linear regression analysis to accommodate the kurtosis of the error terms and to accommodate outliers. Bayesian inference with multivariate elliptical models was initially presented in Chu (1973). Posteriorly, Meinhold and Singpurwalla (1989) consider a robustification of Kalman filters by using the multivariate Student-t distribution. This distribution was also used by Zellner (1976), who considered a Bayesian treatment of linear regression models under non-informative prior distributions.

Clustering is a fundamental problem in statistics and machine learning, whose solution is commonly computed by the Expectation-Maximization (EM) method, which finds a locally optimal solution for an objective function called log-likelihood. Since the surface of the log-likelihood function is non convex, a stochastic search with Markov Chain Monte Carlo (MCMC) methods can help escaping locally optimal solutions. In this article, we tackle two fundamental conflicting goals: Finding higher quality solutions and achieving faster convergence. With that motivation in mind, we introduce an efficient algorithm that combines elements of the EM and MCMC methods to find clustering solutions that are qualitatively better than those found by the standard EM method. Moreover, our hybrid algorithm allows tuning model parameters and understanding the uncertainty in their estimation. The main issue with MCMC methods is that they generally require a very large number of iterations to explore the posterior of each model parameter. Convergence is accelerated by several algorithmic improvements which include sufficient statistics, simplified model parameter priors, fixing covariance matrices and iterative sampling from small blocks of the data set. A brief experimental evaluation shows promising results.

Bayesian inference for nonlinear diffusions, observed at discrete times, is a challenging task that has prompted the development of a number of algorithms, mainly within the computational statistics community. We propose a new direction, and accompanying methodology—borrowing ideas from statistical physics and computational chemistry—for inferring the posterior distribution of latent diffusion paths and model parameters, given observations of the process. Joint configurations of the underlying process noise and of parameters, mapping onto diffusion paths consistent with observations, form an implicitly defined manifold. Then, by making use of a constrained Hamiltonian Monte Carlo algorithm on the embedded manifold, we are able to perform computationally efficient inference for a class of discretely observed diffusion models. Critically, in contrast with other approaches proposed in the literature, our methodology is highly automated, requiring minimal user intervention and applying alike in a range of settings, including: elliptic or hypo-elliptic systems; observations with or without noise; linear or non-linear observation operators. Exploiting Markovianity, we propose a variant of the method with complexity that scales linearly in the resolution of path discretisation and the number of observation times. Python code reproducing the results is available at http://doi.org/10.5281/zenodo.5796148.

ABSTRACTSinh-normal/independent distributions are a class of symmetric heavy-tailed distributions that include the sinh-normal distribution as a special case, which has been used extensively in Birnbaum–Saunders regression models. Here, we explore the use of Markov Chain Monte Carlo methods to develop a Bayesian analysis in nonlinear regression models when Sinh-normal/independent distributions are assumed for the random errors term, and it provides a robust alternative to the sinh-normal nonlinear regression model. Bayesian mechanisms for parameter estimation, residual analysis and influence diagnostics are then developed, which extend the results of Farias and Lemonte [Bayesian inference for the Birnbaum-Saunders nonlinear regression model, Stat. Methods Appl. 20 (2011), pp. 423-438] who used the Sinh-normal/independent distributions with known scale parameter. Some special cases, based on the sinh-Student-t (sinh-St), sinh-slash (sinh-SL) and sinh-contaminated normal (sinh-CN) distributions are discussed in detail. Two real datasets are finally analyzed to illustrate the developed procedures.

A tutorial introduction to Bayesian inference for stochastic epidemic models using Markov chain Monte Carlo methods

Variational Bayes approximation of factor stochastic volatility models

Parameter estimation and uncertainty analysis for a watershed model