Inference and Learning for Generative Capsule Models.

It is now widely accepted that knowledge can be acquired from networks by clustering their vertices according to the connection profiles. Many methods have been proposed and in this paper we concentrate on the Stochastic Block Model (SBM). The clustering of vertices and the estimation of SBM model parameters have been subject to previous work, and numerous inference strategies such as variational expectation maximization (EM) and classification EM have been proposed. However, SBM still suffers from a lack of criteria to estimate the number of components in the mixture. To our knowledge, only one model-based criterion, Integrated Complete-data Likelihood (ICL), has been derived for SBM in the literature. It relies on an asymptotic approximation of the integrated complete-data likelihood and recent studies have shown that it tends to be too conservative in the case of small networks. To tackle this issue, we propose a new criterion that we call Integrated Likelihood Variational Bayes (ILvb), based on a non-asymptotic approximation of the marginal likelihood. We describe how the criterion can be computed through a variational Bayes EM algorithm.

Introduction Variational methods are a key component of the approximate inference and learning toolbox. These methods fill an important middle ground, retaining distributional information about uncertainty in latent variables, unlike maximum a posteriori methods, and yet generally requiring less computational time than Markov chain Monte Carlo methods. In particular the variational expectation maximisation (vEM) and variational Bayes algorithms, both involving variational optimisation of a free-energy, are widely used in time series modelling. Here, we investigate the success of vEM in simple probabilistic time series models. First we consider the inference step of vEM, and show that a consequence of the well-known compactness property of variational inference is a failure to propagate uncertainty in time, thus limiting the usefulness of the retained distributional information. In particular, the uncertainty may appear to be smallest precisely when the approximation is poorest. Second, we consider parameter learning and analytically reveal systematic biases in the parameters found by vEM. Surprisingly, simpler variational approximations (such as mean-field) can lead to less bias than more complicated structured approximations. The variational approach We begin this chapter with a brief theoretical review of the variational expectation maximisation algorithm, before illustrating the important concepts with a simple example in the next section. The vEM algorithm is an approximate version of the expectation maximisation (EM) algorithm [4]. Expectation maximisation is a standard approach to finding maximum likelihood (ML) parameters for latent variable models, including hidden Markov models and linear or non-linear state space models (SSMs) for time series.

A new approach is proposed for the planar array failure diagnosis from a set of far-field measurements. Under the framework of Bayesian learning, a Bernoulli–Gaussian (BG) model is constructed to exploit the binary prior information of the unknown failure elements. Subsequently, the variational expectation maximization (EM) strategy is used to perform the Bayesian inference. Moreover, the damped generalized approximate message passing (GAMP) is further adopted to circumvent the matrix inversion in the variational EM procedure, thus making the overall process computationally very efficient. A set of representative numerical experiments are provided to validate the accuracy, efficiency, and flexibility of the proposed method.

For nonlinear two-dimensional spatial data, a 2D Markov-switching unilateral autoregressive model is defined. Strict stationarity is studied and parameter estimation via the variational expectation maximisation (VEM) algorithm is performed. First, a 2D Markov random field (MRF) is constructed for which an imposed causality allows to establish an analogy between this 2D MRF and the Markov chain representation. Based on the proposed 2D MRF, the 2D MS-AR model is defined under some necessary assumptions and useful notations. Finally, parameter estimation of the model is discussed paving the way to broad exploitation perspectives of the proposed 2D MS-AR processes to efficiently model several phenomena exhibiting a structural break in spatial data.

We recently introduced the high-resolution nonnegative matrix factorization (HR-NMF) model for analyzing mixtures of nonstationary signals in the time-frequency domain, and highlighted its capability to both reach high spectral resolution and reconstruct high quality audio signals. In order to estimate the model parameters and the latent components, we proposed to resort to an expectation-maximization (EM) algorithm based on a Kalman filter/smoother. The approach proved to be appropriate for modeling audio signals in applications such as source separation and audio inpainting. However, its computational cost is high, dominated by the Kalman filter/smoother, and may be prohibitive when dealing with high-dimensional signals. In this paper, we consider two different alternatives, using the variational Bayesian EM algorithm and two mean-field approximations. We show that, while significantly reducing the complexity of the estimation, these novel approaches do not alter its quality.

In this paper, the variational expectation-maximization (VEM) algorithm is used to provide channel estimates in a space-time decoder. The proposed estimator works for many types of space-time codes (STC), including full-rate full-diversity (FRFD) codes, Bell Laboratories layered space-time architecture (BLAST) and orthogonal STC's. The principle idea is to treat the channel coefficients as the unknown parameters, and the transmitted symbols as the unobserved variables in the EM algorithm. The posterior distribution of the symbols is approximated by a factorized distribution, whose Kullback-Liebler divergence with the true distribution is minimized, thereby explaining the ``variational'' aspect of the technique. The channel estimates may then be used for coherent decoding -- here we use the K-best detector for a 2×2 system. Simulation results show that the new channel estimator can reduce system complexity greatly compared with traditional schemes, while achieving near-ideal performance.

User modeling plays an important role in delivering customized web services to the users and improving their engagement. However, most user models in the literature do not explicitly consider the temporal behavior of users. More recently, continuous-time user modeling has gained considerable attention and many user behavior models have been proposed based on temporal point processes. However, typical point process-based models often considered the impact of peer influence and content on the user participation and neglected other factors. Gamification elements are among those factors that are neglected, while they have a strong impact on user participation in online services. In this article, we propose interdependent multi-dimensional temporal point processes that capture the impact of badges on user participation besides the peer influence and content factors. We extend the proposed processes to model user actions over the community-based question and answering websites, and propose an inference algorithm based on Variational-Expectation Maximization that can efficiently learn the model parameters. Extensive experiments on both synthetic and real data gathered from Stack Overflow show that our inference algorithm learns the parameters efficiently and the proposed method can better predict the user behavior compared to the alternatives.

Hyperspectral image clustering is an important and challenging problem, which aims to group image pixels according to the land cover information extracted from the spectrum. The spectrum observed at adjacent pixels are often highly-correlated, and leveraging such spatial correlation can greatly improve the clustering accuracy. Markov Random Field (MRF) is a powerful model to characterize such correlation. However, in this model the spatial parameter β often needs to be manually tuned, which brings difficulty in finding an optimal value. In this paper, we propose a novel hyperspectral clustering algorithm, which is able to learn parameter β from the data and thus achieves better performance. Specifically, we model the spectral information with Gaussian mixture model, and use variational expectation maximization method to complete the parameter estimation and clustering task. Experiments on both synthetic and real-world data sets verify the effectiveness of the proposed algorithm.

Clustering is a crucial step in scientific data analysis and engineering systems. Thus, an efficient cluster analysis method often remains a key challenge. In this paper, we introduce a general purpose exemplar-based clustering method called (MEGA), which performs a novel message-passing strategy based on variational expectation–maximization and generalized arc-consistency techniques. Unlike message passing clustering methods, MEGA formulates the message-passing schema as E- and M-steps of variational expectation–maximization based on a reparameterized factor graph. It also exploits an adaptive variant of generalized arc consistency technique to perform a variational mean-field approximation in E-step to minimize a Kullback–Leibler divergence on the model evidence. Dissimilar to density-based clustering methods, MEGA has no sensitivity to initial parameters. In contrast to partition-based clustering methods, MEGA does not require pre-specifying the number of clusters. We focus on the binary-variable factor graph to model the clustering problem but MEGA is applicable to other graphical models in general. Our experiments on real-world problems demonstrate the efficiency of MEGA over existing prominent clustering algorithms such as Affinity propagation, Agglomerative, DBSCAN, K-means, and EM.

In this paper, we use a variational Bayesian framework for color image segmentation. Each image is represented in the Luv color coordinate system before being segmented by the variational algorithm. The model chosen to describe the color images is a Gaussian mixture model. The parameter estimation uses variational learning by taking into account the uncertainty in parameter estimation. In the variational Bayesian approach, we integrate over distributions of parameters. We propose a maximum log-likelihood initialization approach for the variational expectation-maximization (VEM) algorithm and we apply it to color image segmentation. The segmentation task in our approach consists of the estimation of the distribution hyperparameters.

An on-line variational Bayesian model for multi-person tracking from cluttered scenes

Object tracking is an ubiquitous problem in computer vision with many applications in human-machine and human-robot interaction, augmented reality, driving assistance, surveillance, etc. Although thoroughly investigated, tracking multiple persons remains a challenging and an open problem. In this paper, an online variational Bayesian model for multiple-person tracking is proposed. This yields a variational expectation-maximization (VEM) algorithm. The computational efficiency of the proposed method is due to closed-form expressions for both the posterior distributions of the latent variables and for the estimation of the model parameters. A stochastic process that handles person birth and person death enables the tracker to handle a varying number of persons over long periods of time. The proposed method is benchmarked using the MOT 2016 dataset.

This paper proposes a joint maximum likelihood and Bayesian methodology for estimating Gaussian mixture models. In Bayesian inference, the distributions of parameters are modeled, characterized by hyperparameters. In the case of Gaussian mixtures, the distributions of parameters are considered as Gaussian for the mean, Wishart for the covariance, and Dirichlet for the mixing probability. The learning task consists of estimating the hyperparameters characterizing these distributions. The integration in the parameter space is decoupled using an unsupervised variational methodology entitled variational expectation-maximization (VEM). This paper introduces a hyperparameter initialization procedure for the training algorithm. In the first stage, distributions of parameters resulting from successive runs of the expectation-maximization algorithm are formed. Afterward, maximum-likelihood estimators are applied to find appropriate initial values for the hyperparameters. The proposed initialization provides faster convergence, more accurate hyperparameter estimates, and better generalization for the VEM training algorithm. The proposed methodology is applied in blind signal detection and in color image segmentation.

Pronunciation prediction with Default&Refine

In this paper, we propose a saliency detection model for RGB-D images based on the deep features of RGB images and depth images within a Bayesian framework. By analysing 3D saliency in the case of RGB images and depth images, the class-conditional mutual information is computed for measuring the dependence of deep features extracted using a convolutional neural network; then, the posterior probability of the RGB-D saliency is formulated by applying Bayes’ theorem. By assuming that deep features are Gaussian distributions, a discriminative mixed-membership naive Bayes (DMNB) model is used to calculate the final saliency map. The Gaussian distribution parameters can be estimated in the DMNB model by using a variational inference-based expectation maximization algorithm. The experimental results on RGB-D images from the NLPR dataset and NJU-DS400 dataset show that the proposed model performs better than other existing models.