Bayesian Regularization for Graphical Models With Unequal Shrinkage

The ability to theoretically model the propagation of photon noise through PET and SPECT tomographic reconstruction algorithms is crucial in evaluating the reconstructed image quality as a function of parameters of the algorithm. In a previous approach for the important case of the iterative ML - EM (maximum-likelihood - expectation-maximization) algorithm, judicious linearizations were used to model theoretically the propagation of a mean image and a covariance matrix from one iteration to the next. Our analysis extends this approach to the case of MAP (maximum a posteriori) - EM algorithms, where the EM approach incorporates prior terms. We analyse in detail two cases: a MAP - EM algorithm incorporating an independent gamma prior, and a one-step-late (OSL) version of a MAP - EM algorithm incorporating a multivariate Gaussian prior, for which familiar smoothing priors are special cases. To validate our theoretical analyses, we use a Monte Carlo methodology to compare, at each iteration, theoretical estimates of mean and covariance with sample estimates, and show that the theory works well in practical situations where the noise and bias in the reconstructed images do not assume extreme values.

Motivated by the many applications associated with estimation of sparse multivariate models, the estimation of sparse directional connectivity between the imperfectly measured nodes of a network is studied. Node dynamics and interactions are assumed to follow a multivariate autoregressive model driven by noise, and the observations are a noisy linear combination of the underlying node activities. The corresponding maximum a posteriori (MAP) problem is derived to estimate system parameters. Due to the intractability of the MAP problem, the expectation maximization (EM) framework is used to iteratively implement the MAP estimation. To impose sparsity, the EM algorithm is augmented with an $\ell _1$ regularization of the connectivity matrix. Multiple techniques have been used to lower the computational complexity. Importantly, an efficient coordinate descent algorithm utilizing a closed-form solution is designed to solve the $\ell _1$ -regularized sub-problem of the EM. Cholesky factors of the unknown covariance matrices are used directly in the optimization process in order to impose positive definiteness and guarantee the functionality of the $\ell _1$ optimization. The algorithm is first applied to synthetic data to evaluate the estimation accuracy. A comparison with the previous work over an extensive set of configurations shows that our method is superior under moderate to high sparsity. Finally, the algorithm is applied to real temperature data obtained from 98 stations across the U.S. mainland to identify the predictive interactions between the time series.

This paper presents a new two-step pseudo maximum a posteriori (MAP) segmentation method for the Markov random field (MRF)-modeled image because the exact MAP estimation is hard to implement due to intractable complexity. The expectation maximization (EM) and Markov Chain Monte Carlo (MCMC) methods are adopted to estimate the parameters for the MRF model due to their comparatively good performance. Although the image segmentation algorithms via graph cuts have become very popular nowadays, our proposed algorithm still performs significantly better in automatic identification and segmentation of fuzzy images than them, which is shown by the quantitative results on synthesized images. In practical applications, the proposed two-step pseudo MAP method is superior in segmenting the fuzzy laser images reflected from the weld pool surfaces during the P-GMAW welding process.

Coherent processing of synthetic aperture radar (SAR) data makes SAR images highly susceptible to signal‐dependent granular noise known as speckle. For accurate texture estimation, the maximum a posteriori (MAP) approach is found to be efficient among multitude of despeckling approaches reported in the available literature. However, the formulations of MAP estimator for texture are available in ‘intensity format’. The authors formulate ‐MAP and ‐MAP estimators for texture in ‘amplitude format’, considering clutter to be and distributed. The parameters associated with and distributions are estimated using Mellin transform and second‐kind statistics. Furthermore, an MAP‐based filter is proposed here for mixture of Cauchy–Exponential compound Gaussian (CE‐CG) distributed clutter data where the texture is modelled using CE mixture model. For estimation of CE model parameters, they employ expectation–maximisation (EM) algorithm. Finally, the ‐MAP, ‐MAP, and CE‐MAP estimation of texture both in ‘amplitude format’ and ‘intensity format’ are performed on single look real SAR clutter data and also on multilook synthetic data. Experimental results illustrate the superiority of CE‐MAP filter and moreover that the MAP estimation in ‘amplitude format’ achieves better mean preservation and speckle suppression compared with the same in ‘intensity format’.

Updating cover type maps using sequential indicator simulation

In this paper, we present two finite-dimensional iterative algorithms for maximum a posteriori (MAP) state sequence estimation of bilinear systems. Bilinear models are appealing in their ability to represent or approximate a broad class of nonlinear systems. Our iterative algorithms for state estimation are based on the expectation-maximization (EM) algorithm and outperform the widely used extended Kalman smoother (EKS). Unlike the EKS, these EM algorithms are optimal (in the MAP sense) finite-dimensional solutions to the state sequence estimation problem for bilinear models. We also present recursive (on-line) versions of the two algorithms and show that they outperform the extended Kalman filter (EKF). Our main conclusion is that the EM-based algorithms presented in this paper are novel nonlinear filtering methods that perform better than traditional methods such as the EKF.

A Bayesian estimation approach for enhancing speech signals which have been degraded by statistically independent additive noise is motivated and developed. In particular, minimum mean square error (MMSE) and maximum a posteriori (MAP) signal estimators are developed using hidden Markov models (HMMs) for the clean signal and the noise process. It is shown that the MMSE estimator comprises a weighted sum of conditional mean estimators for the composite states of the noisy signal, where the weights equal the posterior probabilities of the composite states given the noisy signal. The estimation of several spectral functionals of the clean signal such as the sample spectrum and the complex exponential of the phase is also considered. A gain-adapted MAP estimator is developed using the expectation-maximization algorithm. The theoretical performance of the MMSE estimator is discussed, and convergence of the MAP estimator is proved. Both the MMSE and MAP estimators are tested in enhancing speech signals degraded by white Gaussian noise at input signal-to-noise ratios of from 5 to 20 dB. >

We consider the identification problem for a linear system when the variances of the system and/or the observation noises take on two unknown values. It is assumed that the system and observation noises can be represented in terms of Bernoulli sequences and two random variables with specified but unknown variances. We propose an algorithm to simultaneously estimate the unknown variances and the switching parameters by applying the EM (expectation-maximization) algorithm, avoiding non-linear optimizations that arise in the Maximum Likelihood (ML) and the Maximum A Posteriori (MAP) estimations. Several numerical examples are given to illustrate the effectiveness of the present algorithm.

Languages are transmitted from person to person and generation to generation via a process of iterated learning: people learn a language from other people who once learned that language themselves. We analyze the consequences of iterated learning for learning algorithms based on the principles of Bayesian inference, assuming that learners compute a posterior distribution over languages by combining a prior (representing their inductive biases) with the evidence provided by linguistic data. We show that when learners sample languages from this posterior distribution, iterated learning converges to a distribution over languages that is determined entirely by the prior. Under these conditions, iterated learning is a form of Gibbs sampling, a widely-used Markov chain Monte Carlo algorithm. The consequences of iterated learning are more complicated when learners choose the language with maximum posterior probability, being affected by both the prior of the learners and the amount of information transmitted between generations. We show that in this case, iterated learning corresponds to another statistical inference algorithm, a variant of the expectation-maximization (EM) algorithm. These results clarify the role of iterated learning in explanations of linguistic universals and provide a formal connection between constraints on language acquisition and the languages that come to be spoken, suggesting that information transmitted via iterated learning will ultimately come to mirror the minds of the learners.

Imaging performance evaluation in depth-of-interaction PET with a new method of sinogram generation:A Monte Carlo simulation study

Forward Decoding Kernel Machines (FDKM) combine largemargin classifiers with Hidden Markov Models (HMM) for Maximum a Posteriori (MAP) adaptive sequence estimation. State transitions in the sequence are conditioned on observed data using a kernel-based probability model, and forward decoding of the state transition probabilities with the sum-product algorithm directly produces the MAP sequence. The parameters in the probabilistic model are trained using a recursive scheme that maximizes a lower bound on the regularized cross-entropy. The recursion performs an expectation step on the outgoing state of the transition probability model, using the posterior probabilities produced by the previous maximization step. Similar to Expectation-Maximization (EM), the FDKM recursion deals effectively with noisy and partially labeled data.We also introduce a multi-class support vector machine for sparse conditional probability regression, GiniSVM based on a quadratic formulation of entropy. Experiments with benchmark classification data show that GiniSVM generalizes better than other multi-class SVM techniques. In conjunction with FDKM, GiniSVM produces a sparse kernel expansion of state transition probabilities, with drastically fewer non-zero coefficients than kernel logistic regression. Preliminary evaluation of FDKM with GiniSVM on a subset of the TIMIT speech database reveals significant improvements in phoneme recognition accuracy over other SVM and HMM techniques.

Recently, an iterative algorithm has been presented for estimating the parameters of partially observed continuous-time processes [1]. In this note we concentrate on continuous-time ARMA processes observed in white noise. A maximum a-posteriori (MAP) estimator is defined for the trajectory of the parameters' random process. This approach enables the MAP estimation of randomly slowly varying parameters, and extends the conventional treatment of time-invariant parameters. The iterative algorithm derived for the MAP estimation, increases the posterior probability of the parameters in each iteration, and converges to a stationary point of the posterior probability functional. Each iteration involves a standard linear smoother followed by a finite-dimensional linear system, and thus is easily implemented.

Over the past years there has been considerable interest in statistically optimal reconstruction of cross-sectional images from tomographic data. In particular, a variety of such algorithms have been proposed for maximum a posteriori (MAP) reconstruction from emission tomographic data. While MAP estimation requires the solution of an optimization problem, most existing reconstruction algorithms take an indirect approach based on the expectation maximization (EM) algorithm. We propose a new approach to statistically optimal image reconstruction based on direct optimization of the MAP criterion. The key to this direct optimization approach is greedy pixel-wise computations known as iterative coordinate decent (ICD). We propose a novel method for computing the ICD updates, which we call ICD/Newton-Raphson. We show that ICD/Newton-Raphson requires approximately the same amount of computation per iteration as EM-based approaches, but the new method converges much more rapidly (in our experiments, typically five to ten iterations). Other advantages of the ICD/Newton-Raphson method are that it is easily applied to MAP estimation of transmission tomograms, and typical convex constraints, such as positivity, are easily incorporated.

This reference is for an abstract only. A full paper was not submitted for this conference. Abstract AVO (amplitude-varying-with-offset) is a seismic prestack inversion technique for estimating elastic parameters of the subsurface. The seismic AVO inversion problem can be formulated as a linearized or nonlinear inverse problem. It is a multidimensional and highly ill-posed inverse problem. It is affected by strong noise and measurement uncertainty. Therefore, in seismic AVO analysis, the goal is not only to find the model that best-fits the data, but also to characterize the uncertainty of the analysis. Uncertainty characterization of seismic analysis makes the geophysical interpretation more reliable. The Bayesian formalism generally constitutes a powerful approach for solving many seismic inverse problems. It allows combining available prior knowledge with the information contained in the seismic data. The solution of a Bayesian inverse problem is given by the joint posterior distribution of model parameters. Stochastic approximations of the true posterior are exemplified by the MCMC method and more recently by the particle filtering (PF) method. For most seismic inverse problems, stochastic approximation techniques are computationally unappealing. The maximum a posteriori (MAP) is the simplest Bayesian method for estimating model parameters. It provides the explanation that maximizes the posterior distribution. The maximum likelihood (ML) is a popular non-Bayesian method for estimating model parameters. It provides the explanation that maximizes the likelihood of the data. It assumes that the model parameters are deterministic and omits the pertinent prior knowledge on them. In practice, both MAP and ML point estimators have severe problems with over-fitting the data and model order estimation. In this paper, the recently-developed variational Bayesian learning is proposed as a new method for solving the linearized seismic AVO inversion problem. Variational Bayesian learning allows the true joint posterior distribution of model parameters to be approximated by a simpler approximating ensemble for which the required inferences are tractable. The main advantage in resorting to variational Bayesian learning is its robustness to the over-fitting problem, which is major in practice. This probabilistic machine learning method allows finding an optimum balance between the representation power and the model complexity of the data.

In this paper, the expectation-maximization (EM) algorithm for maximum a posteriori (MAP) estimation of a random vector is applied to the problem of symbol detection for continuous phase modulation signals transmitted over time-selective Rayleigh-fading channels. This results in a soft-in-soft-out detection algorithm suitable for iterative detection/decoding schemes. Simulation results show that the error performance provided by the proposed solution is very close to that of a MAP detector endowed with an ideal knowledge of the channel state both in uncoded and coded transmissions.