The transformed Variational Bayes approximation

Cycle slips strongly impact the performance of phase tracking algorithm leading, in the worst case, to a permanent loss of lock of the signal. In this paper, we propose a new nonlinear phase estimator to obtain more robust tracks. The latter stems from a Variational Bayes (VB) approximation used within the optimal Bayesian filtering formulation in case of high-order phase dynamics. A comparison with a more conventional technique, namely a Kalman filter based PLL (Phase Lock Loop), is performed in terms of mean square error of the phase estimate and mean time to first slip. Results show that the proposed method outperforms the conventional linear filter with respect to both metrics, especially at low signal-to-noise ratio.

Factors models are commonly used to analyze high-dimensional data in both single-study and multi-study settings. Bayesian inference for such models relies on Markov chain Monte Carlo (MCMC) methods, which scale poorly as the number of studies, observations, or measured variables increase. To address this issue, we propose new variational inference algorithms to approximate the posterior distribution of Bayesian latent factor models using the multiplicative gamma process shrinkage prior. The proposed algorithms provide fast approximate inference at a fraction of the time and memory of MCMC-based implementations while maintaining comparable accuracy in characterizing the data covariance matrix. We conduct extensive simulations to evaluate our proposed algorithms and show their utility in estimating the model for high-dimensional multi-study gene expression data in ovarian cancers. Overall, our proposed approaches enable more efficient and scalable inference for factor models, facilitating their use in high-dimensional settings. An R package VIMSFA implementing our methods is available on GitHub (github.com/blhansen/VI-MSFA). Supplementary materials for this article are available online.

This paper proposes an adaptive noise suppression method for non-stationary noise based on the Bayesian estimation method. The following conditions are assumed: (1) speech and noise samples are statistically independent, and they follow auto-regressive (AR) processes. (2) The prior distribution of the parameters of the noise AR model of a current frame is identical to the posterior distribution of those parameters calculated in the previous frame. Under these conditions, the proposed method approximates the joint posterior distribution of the AR model parameters and the speech samples by using the variational Bayesian method. Furthermore, we describe an efficient implementation by assuming that all involved covariance matrices have the Toeplitz structure. The proposed method was tested on real speech and noise signals and compared with other noise suppression methods.

Linear regression with measurement error in the covariates is a heavily studied topic, however, the statistics/econometrics literature is almost silent to estimating a multi-equation model with measurement error. This paper considers a seemingly unrelated regression model with measurement error in the covariates and introduces two novel estimation methods: a pure Bayesian algorithm (based on Markov chain Monte Carlo techniques) and its mean field variational Bayes (MFVB) approximation. The MFVB method has the added advantage of being computationally fast and can handle big data. An issue pertinent to measurement error models is parameter identification, and this is resolved by employing a prior distribution on the measurement error variance. The methods are shown to perform well in multiple simulation studies, where we analyze the impact on posterior estimates for different values of reliability ratio or variance of the true unobserved quantity used in the data generating process. The paper further implements the proposed algorithms in an application drawn from the health literature and shows that modeling measurement error in the data can improve model fitting.

Variational inference has become one of the most widely used methods in latent variable modeling. In its basic form, variational inference employs a fully factorized variational distribution and minimizes its Kullback–Leibler divergence to the posterior. As the minimization can only be carried out approximately, this approximation induces a bias. In this paper, we revisit perturbation theory as a powerful way of improving the variational approximation. Perturbation theory relies on a form of Taylor expansion of the log marginal likelihood, vaguely in terms of the log ratio of the true posterior and its variational approximation. While first order terms give the classical variational bound, higher-order terms yield corrections that tighten it. However, traditional perturbation theory does not provide a lower bound, making it inapt for stochastic optimization. In this paper, we derive a similar yet alternative way of deriving corrections to the evidence lower bound that resemble perturbation theory, but that result in a valid bound. We show in experiments on Gaussian processes and variational autoencoders that the new bounds are more mass covering, and that the resulting posterior covariances are closer to the true posterior and lead to higher likelihoods on held-out data.

Mixture models are commonly used in the statistical segmentation of images. For example, they can be used for the segmentation of structural medical images into different matter types, or of statistical parametric maps into activating and nonactivating brain regions in functional imaging. Spatial mixture models have been developed to augment histogram information with spatial regularization using Markov random fields (MRFs). In previous work, an approximate model was developed to allow adaptive determination of the parameter controlling the strength of spatial regularization. Inference was performed using Markov Chain Monte Carlo (MCMC) sampling. However, this approach is prohibitively slow for large datasets. In this work, a more efficient inference approach is presented. This combines a variational Bayes approximation with a second-order Taylor expansion of the components of the posterior distribution, which would otherwise be intractable to Variational Bayes. This provides inference on fully adaptive spatial mixture models an order of magnitude faster than MCMC. We examine the behavior of this approach when applied to artificial data with different spatial characteristics, and to functional magnetic resonance imaging statistical parametric maps.

In this paper, we propose a robust cubature Kalman filter (CKF) for nonlinear state-space models with unknown state and measurement noise covariance matrix. We study situations in which sensors are independent of each other. Therefore, the unknown measurement noise variance is modeled as an unknown inverse gamma (IG) distribution. The Gaussian-Student-t-inverse-Wishart (GSTIW) mixture distribution is used to model the one-step prediction distribution. Modeling generates numerous unknown parameters. Therefore, we adopt the statistical linearization method to linearize the observation function and then estimate the state and parameters separately to reduce the computational burden of estimating unknown parameters, significantly improving the algorithm’s efficiency. Finally, using the variational Bayesian method, a novel and efficient robust CKF based on the IG distribution and GSTIW mixture distributions (IG-GSTIW-CKF) is obtained. Simulation and experimental results show that the proposed method has better estimation accuracy than several advanced algorithms when sensors are independent. In addition, the efficiency of the proposed algorithm is significantly higher than that of other CKF-based methods.

Variational Bayesian inference for network autoregression models

Scanning devices acquire geometric information from the surface of an object in the form of a 3D point set. Such point sets, as any data obtained by means of physical measurement, contain some noise. To create an accurate model of the scanned object, this noise should be resolved before or during the process of surface reconstruction. In this paper, we develop a statistical technique to estimate the noise in a scanned point set. The noise is represented as normal distributions with zero mean and their variances determine the amount of the noise. These distributions are estimated with a variational Bayesian method, which is known to provide more robust estimations than point estimate methods, such as maximum likelihood and maximum a posteriori. Validation experiments and further tests with real scan data show that the proposed technique can accurately estimate the noise in a 3D point set.

Variational Bayes approximation of factor stochastic volatility models

is studied, which allows transformations and distortions on the regressor to be handled. Many important signal processing problems are amenable to this Extended AR (i.e. EAR) model. It is shown that Bayesian identification and prediction of the EAR model can be performed recursively, in common with the AR model itself. The EAR model does, however, require that the transformation be known. When it is unknown, the associated transformation space is represented by a finite set of candidates. What follows is a Mixture-based EAR model, i.e. the MEAR model. An approximate identification algorithm for MEAR is developed, using a restricted Variational Bayes (VB) procedure. It preserves the efficient recursive update of sufficient statistics. The MEAR model is applied to the robust identification of AR processes corrupted by outliers and burst noise respectively, and to click removal for speech. Index Terms — Bayesian identification, probabilistic mixtures, sufficient statistics, recursive identification, Variational Bayes,

Aiming at a comprehensive and concise tutorial survey, recap of variational inference and reinforcement learning with Probabilistic Graphical Models are given with detailed derivations. Reviews and comparisons on recent advances in deep reinforcement learning are made from various aspects. We offer detailed derivations to a taxonomy of Probabilistic Graphical Model and Variational Inference methods in deep reinforcement learning, which serves as a complementary material on top of the original contributions.

Item response theory (IRT) model is a widely appreciated statistical method in exploring the relationship between individual latent traits and item responses. In this paper, a sparse IRT model is established to address the sparsity of factor loadings. A global and local shrinkage prior is imposed to penalize the factor loadings: the global parameter controls the amount of shrinkage at the column levels, while the local parameter adjusts the penalty of factor loadings within each column. We develop a variational Bayesian procedure to conduct posterior inference. By exploiting a stochastic representation for logistic function, we frame sparse IRT model as a mixture model mixing with Pólya-Gamma distribution. Such a strategy admits a conjugate posterior for the latent quantity, thus leading to a straightforward posterior computation. We assess the performance of the proposed method via a simulation study. A real example related to personality assessment is analysed to illustrate the usefulness of methodology.

It is now widely accepted that knowledge can be acquired from networks by clustering their vertices according to the connection profiles. Many methods have been proposed and in this paper we concentrate on the Stochastic Block Model (SBM). The clustering of vertices and the estimation of SBM model parameters have been subject to previous work, and numerous inference strategies such as variational expectation maximization (EM) and classification EM have been proposed. However, SBM still suffers from a lack of criteria to estimate the number of components in the mixture. To our knowledge, only one model-based criterion, Integrated Complete-data Likelihood (ICL), has been derived for SBM in the literature. It relies on an asymptotic approximation of the integrated complete-data likelihood and recent studies have shown that it tends to be too conservative in the case of small networks. To tackle this issue, we propose a new criterion that we call Integrated Likelihood Variational Bayes (ILvb), based on a non-asymptotic approximation of the marginal likelihood. We describe how the criterion can be computed through a variational Bayes EM algorithm.

Non-negative matrix factorization is used to find a basic matrix and a weight matrix to approximate the non-negative matrix. It has proven to be a powerful low-rank decomposition technique for non-negative multivariate data. However, its performance largely depends on the assumption of a fixed number of features. This work proposes a new probabilistic non-negative matrix factorization which factorizes a non-negative matrix into a low-rank factor matrix with constraints and a non-negative weight matrix. In order to automatically learn the potential binary features and feature number, a deterministic Indian buffet process variational inference is introduced to obtain the binary factor matrix. Further, the weight matrix is set to satisfy the exponential prior. To obtain the real posterior distribution of the two factor matrices, a variational Bayesian exponential Gaussian inference model is established. The comparative experiments on the synthetic and real-world datasets show the efficacy of the proposed method.