Identification of target clusters by using the restricted normal mixture model

Information geometry of the EM and em algorithms for neural networks

SummaryPrivacy is becoming increasingly important in collaborative data analysis, especially those involving personal or sensitive information commonly arising from health and commercial settings. The aim of privacy preserving statistical algorithms is to allow inference to be drawn on the joint data without disclosing private data held by each party. This paper presents a privacy‐preserving expectation–maximization (PPEM) algorithm for carrying out maximum likelihood estimation of the parameters of mixture models. We address the scenario of horizontally partitioned data distributed among three or more parties. The PPEM algorithm is a two‐cycle iterative distributed algorithm for fitting mixture models under privacy‐preserving requirements. A distinct advantage of PPEM is that it does not require a trusted third party for cooperative learning, unlike most existing schemes that implement a master/slave hierarchy. By adopting a ring topology and adding random noises to messages before encryption, PPEM helps prevent information leakage in the case of corrupted parties. Furthermore, in contrast to existing works, which typically assume a Honest‐but‐Curious adversary, we consider the much stronger case of a Malicious adversary. For illustration, PPEM is applied to two of the most popular mixture models, namely, the normal mixture model (NMM) andt‐mixture model (tMM), and their effectiveness is analyzed through a security analysis. A real data example is also presented to evaluate the computational complexity and accuracy of PPEM relative to its non‐privacy‐preserving version.

Many diseases show dichotomous phenotypic variation but do not follow a simple Mendelian pattern of inheritance. Variances of these binary diseases are presumably controlled by multiple loci and environmental variants. A least-squares method has been developed for mapping such complex disease loci by treating the binary phenotypes (0 and 1) as if they were continuous. However, the least-squares method is not recommended because of its ad hoc nature. Maximum Likelihood (ML) and Bayesian methods have also been developed for binary disease mapping by incorporating the discrete nature of the phenotypic distribution. In the ML analysis, the likelihood function is usually maximized using some complicated maximization algorithms (e.g. the Newton-Raphson or the simplex algorithm). Under the threshold model of binary disease, we develop an Expectation Maximization (EM) algorithm to solve for the maximum likelihood estimates (MLEs). The new EM algorithm is developed by treating both the unobserved genotype and the disease liability as missing values. As a result, the EM iteration equations have the same form as the normal equation system in linear regression. The EM algorithm is further modified to take into account sexual dimorphism in the linkage maps. Applying the EM-implemented ML method to a four-way-cross mouse family, we detected two regions on the fourth chromosome that have evidence of QTLs controlling the segregation of fibrosarcoma, a form of connective tissue cancer. The two QTLs explain 50-60% of the variance in the disease liability. We also applied a Bayesian method previously developed (modified to take into account sex-specific maps) to this data set and detected one additional QTL on chromosome 13 that explains another 26% of the variance of the disease liability. All the QTLs detected primarily show dominance effects.

Along with advances in technology, matrix data, such as medical/industrial images, have emerged in many practical fields. These data usually have high dimensions and are not easy to cluster due to their intrinsic correlated structure among rows and columns. Most approaches convert matrix data to multi dimensional vectors and apply conventional clustering methods to them, and thus, suffer from an extreme high-dimensionality problem as well as a lack of interpretability of the correlated structure among row/column variables. Recently, a regularized model was proposed for clustering matrix-valued data by imposing a sparsity structure for the mean signal of each cluster. We extend their approach by regularizing further on the covariance to cope better with the curse of dimensionality for large size images. A penalized matrix normal mixture model with lasso-type penalty terms in both mean and covariance matrices is proposed, and then an expectation maximization algorithm is developed to estimate the parameters. The proposed method has the competence of both parsimonious modeling and reflecting the proper conditional correlation structure. The estimators are consistent, and their limiting distributions are derived. We applied the proposed method to simulated data as well as real datasets and measured its clustering performance with the clustering accuracy (ACC) and the adjusted rand index (ARI). The experiment results show that the proposed method performed better with higher ACC and ARI than those of conventional methods.

In standard interval mapping (IM) of quantitative trait loci (QTL), the QTL effect is described by a normal mixture model. When this assumption of normality is violated, the most commonly adopted strategy is to use the previous model after data transformation. However, an appropriate transformation may not exist or may be difficult to find. Also this approach can raise interpretation issues. An interesting alternative is to consider a skew-normal mixture model in standard IM, and the resulting method is here denoted as skew-normal IM. This flexible model that includes the usual symmetric normal distribution as a special case is important, allowing continuous variation from normality to non-normality. In this paper we briefly introduce the main peculiarities of the skew-normal distribution. The maximum likelihood estimates of parameters of the skew-normal distribution are obtained by the expectation-maximization (EM) algorithm. The proposed model is illustrated with real data from an intercross experiment that shows a significant departure from the normality assumption. The performance of the skew-normal IM is assessed via stochastic simulation. The results indicate that the skew-normal IM has higher power for QTL detection and better precision of QTL location as compared to standard IM and nonparametric IM.

Microarray technology is a useful tool for monitoring the expressed levels of thousands of genes simultaneously. Recently, mixture modelling has been used to extract information from expressed genes. It utilizes two separate steps to estimate the number of classes and model parameters, respectively, which however may be time-consuming and fall into sub-optimal solutions. In this paper, we therefore apply an one-step approach, namely rival penalized expectation-maximization (RPEM) algorithm, to microarray data analysis. The RPEM algorithm is capable of estimating the parameters of the normal mixture model, meanwhile determining the number of classes automatically. The numerical results have shown the effectiveness of this technique on real microarray data analysis.

In some applications of data mining, clustering analysis of directional data is often involved. In this case, conventional model-based clustering methods are not suitable for fitting the data of such type of structure. Therefore, a Dirichlet Process Mixture Model based on von Mises-Fisher distribution was proposed for the clustering analysis of directional data. The main motivation is that as a non-parametric Bayesian model, The Dirichlet process can automatically determine the complexity of the mixture model when the number of data categories is unknown. We use the accelerated Variational inference algorithm to quickly estimate the parameters involved in the model, which enables the method to be applied in applications with large data scale. The validity of the proposed model was verified by using different scale simulation data and clustering analysis of gene expression data.

Mixture modeling is a popular technique for identifying unobserved subpopulations (e.g., components) within a data set, with Gaussian (normal) mixture modeling being the form most widely used. Generally, the parameters of these Gaussian mixtures cannot be estimated in closed form, so estimates are typically obtained via an iterative process. The most common estimation procedure is maximum likelihood via the expectation-maximization (EM) algorithm. Like many approaches for identifying subpopulations, finite mixture modeling can suffer from locally optimal solutions, and the final parameter estimates are dependent on the initial starting values of the EM algorithm. Initial values have been shown to significantly impact the quality of the solution, and researchers have proposed several approaches for selecting the set of starting values. Five techniques for obtaining starting values that are implemented in popular software packages are compared. Their performances are assessed in terms of the following four measures: (1) the ability to find the best observed solution, (2) settling on a solution that classifies observations correctly, (3) the number of local solutions found by each technique, and (4) the speed at which the start values are obtained. On the basis of these results, a set of recommendations is provided to the user.

ABSTRACTThis article introduces a new asymmetric distribution constructed by assuming the multivariate normal mean-variance mixture model. Called normal mean-variance mixture of the Lindley distribution, we derive some mathematical properties of the new distribution. Also, a feasible maximum likelihood estimation procedure using the EM algorithm and the asymptotic standard errors of parameter estimates are developed. The performance of the proposed distribution is illustrated by means of real datasets and simulation analysis.

In a distribution power network, the load model has no certain pattern or predicted behaviour due to large range of data and changes in energy consumption for end-user consumers. Thus, a powerful analysis based on probabilistic structure is required. For this paper Gaussian Mixture Model (GMM) has been used. GMM is a powerful probability model that allows different types of load distributions to be presented as a combination of several Gaussian distributions. The parameters of GMM is unknown for large random data such as real load data and these parameters can be identified by Expectation Maximization (EM) algorithm. This paper presents a method to evaluate probabilistic load data concerning the time-evolution of any type of distribution load for any duration of time. The proposed method is explained through generated load data of 100 residential houses for duration of one year.

The Fuzzy C Means (FCM) and Expectation Maximization (EM) algorithms are the most prevalent methods for automatic segmentation of MR brain images into three classes Gray Matter (GM), White Matter (WM) and Cerebrospinal Fluid (CSF). The major difficulties associated with these conventional methods for MR brain image segmentation are the Intensity Non-uniformity (INU) and noise. In this paper, EM and FCM with spatial information and bias correction are proposed to overcome these effects. The spatial information is incorporated by convolving the posterior probability during E-Step of the EM algorithm with mean filter. Also, a method of pixel re-labeling is included to improve the segmentation accuracy. The proposed method is validated by extensive experiments on both simulated and real brain images from standard database. Quantitative and qualitative results depict that the method is superior to the conventional methods by around 25% and over the state-of-the art method by 8%.

A new online clustering method based on not only reinforcement and competitive learning but also pursuit algorithm (Pursuit Reinforcement Competitive Learning: PRCL) as well as learning automata is proposed for reaching a relatively stable clustering solution in comparatively short time duration. UCI repository data which are widely used for evaluation of clustering performance in usual is used for a comparative study among the existing conventional online clustering methods of Reinforcement Guided Competitive Learning: RGCL, Sustained RGCL: SRGCL, Vector Quantization, and the proposed PRCL. The results show that the clustering accuracy of the proposed method is superior to the conventional methods. More importantly, it is found that the proposed PRCL is much faster than the conventional methods. The proposed method is then applied to the evacuation simulation study. It is found that the proposed method is much faster than the conventional method of vector quatization to find the most appropriate evacuation route. Due to the fact that the proposed PRCL method allows finding the most appropriate evacuation route, collisions among peoples who have to evacuate for the proposed method is much less than that of vector quatization.

Remaining useful life (RUL) estimation is the key of prognostics and health management (PHM) technology and is an effective way to ensure the safe and reliable operation of equipment. Aiming at the lack of historical data and prior information for the newly developed small-sample systems, an adaptive RUL estimation method based on the expectation maximization (EM) algorithm is proposed with three-source variability. First, a degradation model based on a Wiener process is established to incorporate three-source variability and dynamic sampling interval, and the analytical solution of RUL distribution is derived in the sense of the first hitting time. Second, an adaptive parameter estimation method based on the EM algorithm is proposed to update the model parameters by using the condition monitoring (CM) data from one working system running up to the current moment. Finally, a practical example of a gyroscope in an inertial navigation system is provided to substantiate the effectiveness and superiority of the proposed method. The results indicate that the proposed method can efficiently improve the accuracy of the RUL estimation.

In Recent days Semi supervised image segmentation techniques play a noteworthy role in image processing. Semi supervised image segmentation needs both labeled data and unlabeled data. It means that a Small amount of human assistance or Prior information is given during clustering process. This paper discusses an enhanced semi supervised image segmentation method from labeled image. It uses both a background selection marker and fore ground object selection marker separately. The EM (Expectation Maximization) algorithm is used for clustering along with must link constraints. The proposed method is applied for natural images using MATLAB 7. Thus the proposed method extracts Object of Interest (OOI) from OONI (Object of Not Interest) efficiently and the experimental results are compared with Standard K Means and EM Algorithm also. The results show that the proposed system gives better results than the other two methods. It may also be suitable for object extraction from natural images and medical image analysis.

This paper considers the investigation of the optimal number of clusters for datasets that are modeled as the Gaussian mixture. For that purpose, the adaptive method that is based on the modified Expectation Maximization (EM) algorithm is developed. The modification is conducted within the hidden variable of the standard EM algorithm. Assuming that data are multivariate normally distributed, where each component of the Gaussian mixture corresponds to one cluster, the modification is provided by utilizing the fact that the Mahalanobis distance of samples follows a Chi-square distribution. Besides, the quantity measure is constructed in order to determine number of clusters. The proposed method is presented in several numerical examples.