Estimating local optimums in EM algorithm over Gaussian mixture model

In a distribution power network, the load model has no certain pattern or predicted behaviour due to large range of data and changes in energy consumption for end-user consumers. Thus, a powerful analysis based on probabilistic structure is required. For this paper Gaussian Mixture Model (GMM) has been used. GMM is a powerful probability model that allows different types of load distributions to be presented as a combination of several Gaussian distributions. The parameters of GMM is unknown for large random data such as real load data and these parameters can be identified by Expectation Maximization (EM) algorithm. This paper presents a method to evaluate probabilistic load data concerning the time-evolution of any type of distribution load for any duration of time. The proposed method is explained through generated load data of 100 residential houses for duration of one year.

The expectation maximisation (EM) algorithm is an iterative maximum likelihood procedure often used for estimating the parameters of a mixture model. Theoretically, increases in the likelihood function are guaranteed as the algorithm iteratively improves upon previously derived parameter estimates. The algorithm is considered to converge when all parameter estimates become stable and no further improvements can be made to the likelihood value. However, to reduce computational time, it is often common practice for the algorithm to be stopped before complete convergence using heuristic approaches. In this paper, we consider various stopping criteria and evaluate their effect on fitting Gaussian mixture models (GMMs) to patient length of stay (LOS) data. Although the GMM can be successfully fitted to positively skewed data such as LOS, the fitting procedure often requires many iterations of the EM algorithm. To our knowledge, no previous study has evaluated the effect of different stopping criteria on fitting GMMs to skewed distributions. Hence, the aim of this paper is to evaluate the effect of various stopping criteria in order to select and justify their use within a patient spell classification methodology. Results illustrate that criteria based on the difference in the likelihood value and on the GMM parameters may not always be a good indicator for stopping the algorithm. In fact we show that the values of the difference in the variance parameters should be used instead, as these parameters are the last to stabilise. In addition, we also specify threshold values for the other stopping criteria.

The Expectation-Maximization algorithm is a very popular approach for estimating the parameters of Gaussian mixture models (GMMs). A known issue with GMM estimation is its sensitivity to outliers, which can lead to poor estimation performance depending on the dataset under consideration. A common approach to deal with this issue is robust estimation, which typically consists of reducing the influence of the outliers on the estimators by weighting the impact of some samples of the dataset considered as outliers. In an unsupervised context, it is difficult to know which sample from the database corresponds to a normal observation. To that extent, we propose to use within the EM algorithm an outlier detection step that attributes an anomaly score to each sample of the database in an unsupervised way. A modified Bayesian Information Criterion is also introduced to efficiently select the appropriate amount of outliers contained in a dataset. The proposed method is tested on a benchmark remote sensing dataset coming from the UCI Machine Learning Repository. The experimental results show the interest of the proposed robustification when compared to other benchmark imputation procedures.

Many diseases show dichotomous phenotypic variation but do not follow a simple Mendelian pattern of inheritance. Variances of these binary diseases are presumably controlled by multiple loci and environmental variants. A least-squares method has been developed for mapping such complex disease loci by treating the binary phenotypes (0 and 1) as if they were continuous. However, the least-squares method is not recommended because of its ad hoc nature. Maximum Likelihood (ML) and Bayesian methods have also been developed for binary disease mapping by incorporating the discrete nature of the phenotypic distribution. In the ML analysis, the likelihood function is usually maximized using some complicated maximization algorithms (e.g. the Newton-Raphson or the simplex algorithm). Under the threshold model of binary disease, we develop an Expectation Maximization (EM) algorithm to solve for the maximum likelihood estimates (MLEs). The new EM algorithm is developed by treating both the unobserved genotype and the disease liability as missing values. As a result, the EM iteration equations have the same form as the normal equation system in linear regression. The EM algorithm is further modified to take into account sexual dimorphism in the linkage maps. Applying the EM-implemented ML method to a four-way-cross mouse family, we detected two regions on the fourth chromosome that have evidence of QTLs controlling the segregation of fibrosarcoma, a form of connective tissue cancer. The two QTLs explain 50-60% of the variance in the disease liability. We also applied a Bayesian method previously developed (modified to take into account sex-specific maps) to this data set and detected one additional QTL on chromosome 13 that explains another 26% of the variance of the disease liability. All the QTLs detected primarily show dominance effects.

The accurate modeling of uncertain variables, such as wind power and load uncertainty, is an essential foundation for the planning operation and scheduling decisions in power systems. Currently, Gaussian mixture model (GMM) and Dirichlet process mixture model (DPMM) are the primary techniques used to characterize uncertainties in power systems, which are commonly solved by expectation-maximization (EM) algorithm. However, for the massive data of uncertain variables, the algorithm encounters challenges in accurately obtaining GMM and DPMM with a lower time consumption. To address this issue, we propose a method for GMM uncertainty modeling in power systems considering the mutual assistance of latent variables. Specifically, the GMM of uncertain variables is first constructed, and the conditional probability is employed to characterize the mutual assistance of latent variables. Then, an improved EM algorithm is developed to obtain the optimal GMM parameters, in which the expectation step (E-step) and maximization step (M-step) of the algorithm are revised using the conditional probability. Importantly, the closed-form solutions for GMM parameters are rederived in the revised E-step and M-step. Finally, the proposed uncertainty modeling method is compared with the traditional GMM and DPMM on actual wind power and load data from Australia. The proposed method performs the efficiency and accuracy in characterizing uncertainty.

An optimal maximal ratio combiner (MRC) based on the expectation-maximization (EM) algorithm is devel- oped for noisy constant envelope signals transmitted over a Rayleigh fading channel. Instead of using a transmitted pilot signal with the data to estimate the combiner gains, the EM algorithm is used to perform this estimation. In the developed MRC, estimation of the transmitted data sequence is performed also by the EM algorithm. Estimation using the EM algo- rithm provides an iterative solution to the maximum likelihood (ML) approach. Therefore, the resulting receiver is opti- mum and does not suffer from the difficulties resulted from direct application of the ML procedure. One of these difficul- ties is the computational complexity which depends exponentially on the data sequence length. Introducing an iterative structure in the developed MRC achieves a linear computational complexity and enables efficient data extraction by the Viterbi algorithm when trellis coding is used.

Joint modeling of survival and longitudinal data has been studied extensively in the recent literature. The likelihood approach is one of the most popular estimation methods employed within the joint modeling framework. Typically, the parameters are estimated using maximum likelihood, with computation performed by the expectation maximization (EM) algorithm. However, one drawback of this approach is that standard error (SE) estimates are not automatically produced when using the EM algorithm. Many different procedures have been proposed to obtain the asymptotic covariance matrix for the parameters when the number of parameters is typically small. In the joint modeling context, however, there may be an infinite-dimensional parameter, the baseline hazard function, which greatly complicates the problem, so that the existing methods cannot be readily applied. The profile likelihood and the bootstrap methods overcome the difficulty to some extent; however, they can be computationally intensive. In this paper, we propose two new methods for SE estimation using the EM algorithm that allow for more efficient computation of the SE of a subset of parametric components in a semiparametric or high-dimensional parametric model. The precision and computation time are evaluated through a thorough simulation study. We conclude with an application of our SE estimation method to analyze an HIV clinical trial dataset.

In this paper, a computationally efficient algorithm is presented for joint maximum likelihood (ML) timing and carrier phase estimation of OFDM systems employing M-PSK modulation scheme with additive Gaussian noise, based on the expectation-maximization (EM) algorithm. A nondata-aided (NDA) scheme is considered for the joint timing and phase synchronizer which maximizes the low SNR limit of the likelihood function averaged over the M-PSK signal constellation. For this, an EM algorithm is derived which estimates the timing offset and the phase rotations of each subcarrier iteratively and which converges to the true ML estimation of the unknown timing and phase. It is shown that the algorithm becomes independent of the signal-to-noise ratio for both low and high SNR cases. The algorithm is applied to the QPSK modulated OFDM systems and it is concluded that for SNR values greater than 10 dB the convergence is achieved in first iteration and for SNR values less than 10 dB, at most in three iterations. It is also concluded that the convergence is independent of the initial starting points.

This paper addresses the problem of initialization of the expectation-maximization (EM) algorithm for maximum likelihood estimation of Gaussian mixture models. In order to avoid local maxima of the likelihood function, a genetic algorithm (GA) which searches for best initial conditions of the EM algorithm is proposed. In the GA, a chromosome represents a set of initial conditions, in which initial mean vectors of mixture components are feature vectors chosen from the training set. The chromosome also encodes variances of initial spherical covariance matrices of mixture components. To evaluate each chromosome in the GA we run the EM algorithm until convergence and use the obtained log likelihood as the fitness. In computational experiments our approach was applied to clustering problem and tested on two datasets from the image processing domain. The results indicate that our method outperforms the standard multiple restart EM algorithm and is at least comparable to the state-of-the art random swap EM method.

Condition assessment of cables by pattern recognition of vehicle-induced cable tension ratio

A new maximum likelihood training algorithm is proposed that compensates for weaknesses of the EM algorithm by using cross-validation likelihood in the expectation step to avoid overtraining. By using a set of sufficient statistics associated with a partitioning of the training data, as in parallel EM, the algorithm has the same order of computational requirements as the original EM algorithm. Analyses using a GMM with artificial data show the proposed algorithm is more robust for overtraining than the conventional EM algorithm. Large vocabulary recognition experiments on Mandarin broadcast news data show that the method makes better use of more parameters and gives lower recognition error rates than EM training.

The EM algorithm is a very powerful optimization method and has become popular in many fields. Unfortunately, EM is only a local optimization method and can get stuck in sub-optimal solutions. While more and more contemporary data/model combinations yield multiple local optima, there have been only very few attempts at making EM suitable for global optimization. In this paper we review the basic EM algorithm, its properties and challenges, and we focus in particular on its randomized implementation. The randomized EM implementation promises to solve some of the contemporary data/model challenges, and it is particularly well-suited for a wedding with global optimization ideas, since most global optimization paradigms are also based on the principles of randomization. We review some of the challenges of the randomized EM implementation and present a new algorithm that combines the principles of EM with that of the Genetic Algorithm. While this new algorithm shows some promising results for clustering of an online auction database of functional objects, the primary goal of this work is to bridge a gap between the field of statistics, which is home to extensive research on the EM algorithm, and the field of operations research, in which work on global optimization thrives, and to stimulate new ideas for joint research between the two.

A classical technique for reconstruction of emission tomography (ET) images from measured projections is based on the maximum likelihood (ML) estimation, achieved with the expectation maximization (EM) algorithm. We incorporate the wavelet transform (WT) and total variation (TV) based penalties into the ML framework, and compare performance of the EM algorithm and the previously proposed conjugate barrier (CB) algorithm. Using the WT- and TV-based penalties allows one to embed regularization procedures into the iterative process. In the case of the WT-based penalty, we impose a subset of wavelet coefficients with a desired resolution on the objective function. It appears that the CB algorithm outperforms substantially the EM algorithm in penalized reconstruction. The properties of the optimization algorithms along with WTand TV-based regularization are demonstrated on image reconstructions of a synthetic brain phantom, and the quality of reconstruction is compared with standard methods.

Salient object detection in a real-time environment demands high accuracy with less computation time. It is a changeling task to investigate a saliency model which improves saliency detection accuracy as well as reduces computation time. In this paper, we propose a hybrid model that improves detection accuracy with low computational time. The input image is simplified and clustered at multiple scales using SLIC and K-means clustering algorithms respectively. Gaussian Mixture Model (GMM) is developed on various color components of the digital image at multiple scales. The parameters of GMM are learnt using Expectation Maximization (EM) algorithm. The spatial variance of each color component is determined using GMM parameters and hence object position is estimated. Further, spatial variance of color components and object position is exploited to compute saliency map at a scale level. Afterwards, all the saliency maps generated across various scales are linearly combined to produce the final saliency map. The performance of the proposed model is compared in terms of Precision, Recall, F-Measure, Area under the Curve (AUC), Receiver Operating Characteristics (ROC) and Mean Absolute Error (MAE). Extensive experiments on six publicly available datasets viz. MSRA10K, DUT-OMRON, ECSSD, PASCAL-S, SED2, and THUR15K show that the proposed model outperforms or comparable against 11 state-of-the-art methods of the last decade. The key features of the proposed method are object completeness and efficiency in terms of computational time.

This paper describes a local and distributed expectation maximization algorithm for learning parameters of Gaussian mixture models (GMM) in large peer-to-peer (P2P) environments. The algorithm can be used for a variety of well-known data mining tasks in distributed environments such as clustering, anomaly detection, target tracking, and density estimation to name a few, necessary for many emerging P2P applications in bioinformatics, webmining and sensor networks. Centralizing all or some of the data to build global models is impractical in such P2P environments because of the large number of data sources, the asynchronous nature of the P2P networks, and dynamic nature of the data/network. The proposed algorithm takes a two-step approach. In the monitoring phase, the algorithm checks if the model `quality' is acceptable by using an efficient local algorithm. This is then used as a feedback loop to sample data from the network and rebuild the GMM when it is outdated. We present thorough experimental results to verify our theoretical claims.