Cross-Validation EM Training for Robust Parameter Estimation

EM algorithm is a very popular iteration-based method to estimate the parameters of Gaussian Mixture Model from a large observation set. However, in most cases, EM algorithm is not guaranteed to converge to the global optimum. Instead, it stops at some local optimums, which can be much worse than the global optimum. Therefore, it is usually required to run multiple procedures of EM algorithm with different initial configurations and return the best solution. To improve the efficiency of this scheme, we propose a new method which can estimate an upper bound on the logarithm likelihood of the local optimum, based on the current configuration after the latest EM iteration. This is accomplished by first deriving some region bounding the possible locations of local optimum, followed by some upper bound estimation on the maximum likelihood. With this estimation, we can terminate an EM algorithm procedure if the estimated local optimum is definitely worse than the best solution seen so far. Extensive experiments show that our method can effectively and efficiently accelerate conventional multiple restart EM algorithm.

In a recent contribution, assuming pilot-only based channel estimation, we proposed an improved receiver which reduces the impact of channel uncertainty on the decoder performance. Here, we focus on semi-blind channel estimation based on expectation-maximization (EM) algorithm and propose an improved receiver design by an appropriate use of the channel estimation error statistics. This is done by means of an equivalent formulation of the EM algorithm from which we extract the statistics of the estimation errors. This additional soft information is used at each iteration of the receiver in a Bayesian framework to improve the performance of iterative detection. Our approach is compared to the conventional EM algorithm where the channel estimation error statistics are not available and where the receiver uses the channel estimate as if it was the perfect channel (also known as mismatched detection). Numerical results in the context of orthogonal frequency-division multiplexing show that the proposed approach outperforms the classically-used EM algorithm in terms of bit error rates.

Thermoacoustic tomography (TAT) is an emerging imaging technique with potential for a wide range of biomedical imaging applications. In this correspondence, we propose an infinite family of weighted expectation maximization (EM) algorithms for reconstruction of images from temporally truncated TAT measurement data. The weighted EM algorithms are equivalent mathematically to the conventional EM algorithm, but are shown to propagate data inconsistencies in different ways. Using simulated and experimental TAT measurement data, we demonstrate that suitable choices of weighted EM algorithms can effectively mitigate image artifacts that are attributable to temporal truncation of the TAT data function.

Integrating model-based machine learning methods into deep neural architectures allows one to leverage both the expressive power of deep neural nets and the ability of model-based methods to incorporate domain-specific knowledge. In particular, many works have employed the expectation maximization (EM) algorithm in the form of an unrolled layer-wise structure that is jointly trained with a backbone neural network. However, it is difficult to discriminatively train the backbone network by backpropagating through the EM iterations as they are prone to the vanishing gradient problem. To address this issue, we propose Highway Expectation Maximization Networks (HEMNet), which is comprised of unrolled iterations of the generalized EM (GEM) algorithm based on the Newton-Rahpson method. HEMNet features scaled skip connections, or highways, along the depths of the unrolled architecture, resulting in improved gradient flow during backpropagation while incurring negligible additional computation and memory costs compared to standard unrolled EM. Furthermore, HEMNet preserves the underlying EM procedure, thereby fully retaining the convergence properties of the original EM algorithm. We achieve significant improvement in performance on several semantic segmentation benchmarks and empirically show that HEMNet effectively alleviates gradient decay.

This paper considers the problem of the joint channel and carrier frequency offset (CFO) estimation for asynchronous amplify-and-forward (AF) two-way relay networks (TWRNs). This problem has not been solved to the best of our knowledge, and the presence of timing offsets between the two terminals, in addition to the frequency offsets, makes it necessary to develop signal sampling and joint estimation schemes that explicitly take into account both offsets. As potential solutions, we investigate two estimation strategies. The first strategy is pilot-based estimation, and in this context, we develop the maximum-likelihood joint CFO and channel estimator. Then, we consider the semi-blind estimation strategy, proposing a novel estimator based on expectation maximization (EM). As an alternative to the EM, we also investigate decision-directed (DD) semi-blind estimation. The semi-blind Cramer-Rao bound (CRB) is also obtained as a benchmark. Extensive experiments are used to compare the different estimation methods, showing that the proposed EM algorithm yields significantly lower mean-squared error than both the pilot-based estimator and the DD estimator and converges to the obtained CRB. It also yields much better symbol error rates. This is achieved at a reasonable computational complexity, and only a limited number of EM iterations are sufficient for convergence. The main contribution of our work is the development of the EM estimator, the corresponding CRB, and the use of simulations to demonstrate the superior performance of the proposed EM algorithm compared to other potential solutions.

This paper presents novel expectation-maximization (EM) algorithms for estimating the nonnegative matrix factorization model with Itakura-Saito divergence. Indeed, the common EM-based approach exploits the space-alternating generalized EM (SAGE) variant of EM but it usually performs worse than the conventional multiplicative algorithm. We propose to explore more exhaustively those algorithms, in particular the choice of the methodology (standard EM or SAGE variant) and the latent variable set (full or reduced). We then derive four EM-based algorithms, among which three are novel. Speech separation experiments show that one of those novel algorithms using a standard EM methodology and a reduced set of latent variables outperforms its SAGE variants and competes with the conventional multiplicative algorithm.

In this paper, an unsupervised image segmentation algorithm is proposed, which combines spatial constraints with a kernel fuzzy c-means (KFCM) clustering algorithm. Conventional KFCM clustering segmentation algorithm does not incorporate the spatial context information of image, which makes it sensitive to the noise and intensity variations. In order to overcome the shortcomings, the contents of image is characterized by Gaussian mixture model, and the parameters of model are estimated by modified expectation maximization (EM) algorithm, which overcomes the classical EM algorithm drawbacks that easily trap in local maxima and be susceptible to initial value. According to the maximum a posterior theorem, we can get the pixel maximum posterior probability. We redefine the objective function of the KFCM algorithm which incorporates the pixel maximum posteriori probability, by minimizing the fuzzy objective function, the fuzzy segmentation algorithm is derived. The experimental results on a synthetic image and a real magnetic resonance image show that the proposed algorithm is more effective than the conventional FCM algorithm without local spatial constraints.

Inference of haplotypes is important in genetic epidemiology studies. However, all large genotype data sets have errors due to the use of inexpensive genotyping machines that are fallible and shortcomings in genotyping scoring softwares, which can have an enormous impact on haplotype inference. In this article, we propose two novel strategies to reduce the impact induced by genotyping errors in haplotype inference. The first method makes use of double sampling. For each individual, the "GenoSpectrum" that consists of all possible genotypes and their corresponding likelihoods are computed. The second method is a genotype clustering algorithm based on multi-genotyping data, which also assigns a "GenoSpectrum" for each individual. We then describe two hybrid EM algorithms (called DS-EM and MG-EM) that perform haplotype inference based on "GenoSpectrum" of each individual obtained by double sampling and multi-genotyping data. Both simulated data sets and a quasi real-data set demonstrate that our proposed methods perform well in different situations and outperform the conventional EM algorithm and the HMM algorithm proposed by Sun, Greenwood, and Neal (2007, Genetic Epidemiology 31, 937-948) when the genotype data sets have errors.

In this paper, a unified framework of iterative algebraic reconstruction for emission computed tomography (ECT) and its application to positron emission tomography (PET) is presented. The unified framework is based on an algebraic image restoration model and contains conventional iterative algebraic reconstruction algorithms: ART, SIRT, Landweber iteration (LWB), the generalized Landweber iteration (GLWB), the steepest descent method (STP), as well as iterative filtered backprojection (IFBP) reconstruction algorithms: Chang's method, Walters' method, and a modified iterative MAP. The framework provides an effective tool to systematically study conventional iterative algebraic algorithms and IFBP algorithms. Based on this framework, conventional iterative algebraic algorithms and IFBP algorithms are generalized. It is shown from the algebraic point of view that IFBP algorithms are not only excellent methods for correction of attenuation (either uniform or nonuniform) but are also good general iterative reconstruction algorithms (they can be applied to either attenuated or attenuation-free projections and converge very fast). The convergence behavior of iterative algebraic algorithms is discussed and insight is drawn into the fast convergence property of IFBP algorithms. A simulated PET system is used to evaluate IFBP algorithms and LWB in comparison with the maximum likelihood estimation via expectation maximization algorithm (MLE-EM) and the filtered backprojection (FBP) algorithm. The simulation results indicate that for both attenuation-free projection and attenuated projection cases IFBP algorithms have a significant computational advantage over LWB and MLE-EM, and have performance advantages over FBP in terms of contrast recovery and/or noise-to-signal ratios (NSRs) in regions of interest.

This paper presents a new approximate Bayesian estimator for enhancing a noisy speech signal. The speech model is assumed to be a Gaussian mixture model (GMM) in the log-spectral domain. This is in contrast to most current models in frequency domain. Exact signal estimation is a computationally intractable problem. We derive three approximations to enhance the efficiency of signal estimation. The Gaussian approximation transforms the log-spectral domain GMM into the frequency domain using minimal Kullback-Leiber (KL)-divergency criterion. The frequency domain Laplace method computes the maximum a posteriori (MAP) estimator for the spectral amplitude. Correspondingly, the log-spectral domain Laplace method computes the MAP estimator for the log-spectral amplitude. Further, the gain and noise spectrum adaptation are implemented using the expectation-maximization (EM) algorithm within the GMM under Gaussian approximation. The proposed algorithms are evaluated by applying them to enhance the speeches corrupted by the speech-shaped noise (SSN). The experimental results demonstrate that the proposed algorithms offer improved signal-to-noise ratio, lower word recognition error rate, and less spectral distortion.

Two Level (TL) models allow the total variation in the outcome to be decomposed as level one and level two or ‘individual and group’ variance components. Two Level Mixture (TLM) models can be used to explore unobserved heterogeneity that represents different qualitative relationships in the outcome. In this paper, we extend the standard TL model by introducing constraints to guide the TLM algorithm towards a more appropriate data partitioning. Our constraints-based methods combine the mixing proportions estimated by parametric Expectation Maximization (EM) of the outcome and the random component from the TL model. This forms new two level mixing conditional (TLMc) approach by means of prior information. The new framework advantages are: 1. avoiding trial and error tactic used by TLM for choosing the best BIC (Bayesian Information Criterion), 2. permitting meaningful parameter estimates for distinct classes in the coefficient space and finally 3. allowing smaller residual variances. We show the benefit of our method using overweight and obesity from Body Mass Index (BMI) for students in year 6. We apply these methods on hierarchical BMI data to estimate student multiple deprivation and school Club effects.

Recent literature has suggested that haplotype inference through close relatives, especially from nuclear families, can be an alternative strategy in determining linkage phase and estimating haplotype frequencies. In the case of no possibility to obtain genotypes for parents, and only full-sib information being used, a new approach is suggested to infer phase and to reconstruct haplotypes. We present a maximum-likelihood method via an expectation-maximization algorithm, called FSHAP, using only full-sib information when parent information is not available. FSHAP can deal with families with an arbitrary number of children, and missing parents or missing genotypes can be handled as well. In a simulation study we compare FSHAP with another existing expectation-maximization (EM)-based approach (FAMHAP), the conditioning approach implemented in FBAT and GENEHUNTER, which is only pedigree based and assumes linkage equilibrium. In most situations, FSHAP has the smallest discrepancy of haplotype frequency estimation and the lowest error rate in haplotype reconstruction, only in some cases FAMHAP yields comparable results. GENEHUNTER produces the largest discrepancy, and FBAT produces the highest error rate in offspring in most situations. Among the methods compared, FSHAP has the highest accuracy in reconstructing the diplotypes of the unavailable parents. Potential limitations of the method, e.g., in analyzing very large haplotypes, are indicated and possible solutions are discussed.

Bayesian principal geodesic analysis for estimating intrinsic diffeomorphic image variability.

A reconstructed image in positron emission tomography (PET) should be such that its likelihood, assuming a Poisson model, is high given the observed detector readings. The expectation maximization (EM) methodology leads to an iterative algorithm, called ML-EM, that converges in the limit to an image that maximizes this likelihood. An undesirable property of the algorithm is that it produces images with irregular high amplitude patterns as the number of iterations increases. One approach to alleviate these high amplitude patterns is to use a stopping rule that terminates the process before the appearance of the undesirable high amplitude patterns; one recently-proposed stopping rule results in the method called MLEM-STOP. This paper takes a different approach by applying the recently developed superiorization methodology to ML-EM. Superiorization is an automated procedure for turning an iterative algorithm for producing images that satisfy a primary criterion (in our case that of having a high likelihood given the observed detector readings) into its superiorized version that will be as good as the original algorithm according to the primary criterion, but will in addition produce images that are also good according to a secondary criterion. The approach is demonstrated for two secondary criteria, one provided by an assumed Gaussian prior distribution and the other based on total variation minimization. It is demonstrated that the superiorization methodology achieves its aim for both these criteria. It is further shown by a study, using statistical hypothesis testing on simulated collection of PET data from the human head, that for either secondary criterion the superiorized version of ML-EM outperforms MLEM-STOP for the task of estimating activity within neuroanatomical structures.

Recent advances are presented for multiframe blind deconvolution (MFBD) of ground based telescope imagery for low-earth orbit objects. The iterative algorithm uses the maximum likelihood estimation optimization criterion. It is modeled from a previous well-known algorithm called the expectation-maximization (EM) algorithm. New renditions of the algorithm simplify the phase reconstruction, thereby reducing the complexity of the original EM algorithm. Examples are shown, with and without adaptive optics (AO). The system is being designed for on-the-fly streaming video operation.