Two Entropy-Based Methods for Learning Unsupervised Gaussian Mixture Models

The expectation maximisation (EM) algorithm is an iterative maximum likelihood procedure often used for estimating the parameters of a mixture model. Theoretically, increases in the likelihood function are guaranteed as the algorithm iteratively improves upon previously derived parameter estimates. The algorithm is considered to converge when all parameter estimates become stable and no further improvements can be made to the likelihood value. However, to reduce computational time, it is often common practice for the algorithm to be stopped before complete convergence using heuristic approaches. In this paper, we consider various stopping criteria and evaluate their effect on fitting Gaussian mixture models (GMMs) to patient length of stay (LOS) data. Although the GMM can be successfully fitted to positively skewed data such as LOS, the fitting procedure often requires many iterations of the EM algorithm. To our knowledge, no previous study has evaluated the effect of different stopping criteria on fitting GMMs to skewed distributions. Hence, the aim of this paper is to evaluate the effect of various stopping criteria in order to select and justify their use within a patient spell classification methodology. Results illustrate that criteria based on the difference in the likelihood value and on the GMM parameters may not always be a good indicator for stopping the algorithm. In fact we show that the values of the difference in the variance parameters should be used instead, as these parameters are the last to stabilise. In addition, we also specify threshold values for the other stopping criteria.

The accurate modeling of uncertain variables, such as wind power and load uncertainty, is an essential foundation for the planning operation and scheduling decisions in power systems. Currently, Gaussian mixture model (GMM) and Dirichlet process mixture model (DPMM) are the primary techniques used to characterize uncertainties in power systems, which are commonly solved by expectation-maximization (EM) algorithm. However, for the massive data of uncertain variables, the algorithm encounters challenges in accurately obtaining GMM and DPMM with a lower time consumption. To address this issue, we propose a method for GMM uncertainty modeling in power systems considering the mutual assistance of latent variables. Specifically, the GMM of uncertain variables is first constructed, and the conditional probability is employed to characterize the mutual assistance of latent variables. Then, an improved EM algorithm is developed to obtain the optimal GMM parameters, in which the expectation step (E-step) and maximization step (M-step) of the algorithm are revised using the conditional probability. Importantly, the closed-form solutions for GMM parameters are rederived in the revised E-step and M-step. Finally, the proposed uncertainty modeling method is compared with the traditional GMM and DPMM on actual wind power and load data from Australia. The proposed method performs the efficiency and accuracy in characterizing uncertainty.

The expectation-maximization (EM) algorithm is a popular approach for parameter estimation of finite mixture model (FMM). A drawback of this approach is that the number of components of the finite mixture model is not known in advance, nevertheless, it is a key issue for EM algorithms. In this paper, a penalized minimum matching distance-guided EM algorithm is discussed. Under the framework of Greedy EM, a fast and accurate algorithm for estimating the number of components of the Gaussian mixture model (GMM) is proposed. The performance of this algorithm is validated via simulative experiments of univariate and bivariate Gaussian mixture models.

In this paper, we use a variational Bayesian framework for color image segmentation. Each image is represented in the Luv color coordinate system before being segmented by the variational algorithm. The model chosen to describe the color images is a Gaussian mixture model. The parameter estimation uses variational learning by taking into account the uncertainty in parameter estimation. In the variational Bayesian approach, we integrate over distributions of parameters. We propose a maximum log-likelihood initialization approach for the variational expectation-maximization (VEM) algorithm and we apply it to color image segmentation. The segmentation task in our approach consists of the estimation of the distribution hyperparameters.

Efficient similarity search for uncertain data is a challenging task in many modern data mining applications like image retrieval, speaker recognition and stock market analysis. A common way to model the uncertainty of data objects is using probability density functions in the form of Gaussian Mixture Models (GMMs), which have an ability to approximate arbitrary distribution. However, due to the possible unequal length of mixture models, the use of existing index techniques has serious problems for the objects modeled by GMMs. Either the techniques cannot handle GMMs or they have too many limitations. Hence, we propose a dynamic index structure, Gaussian Component based Index (GCI), for GMMs. GCI decomposes GMMs into the single, pairs, or n-lets of Gaussian components, stores these components into well studied index trees such as U-tree and Gauss-Tree, and refines the corresponding GMMs in a conservative but tight way. GCI supports both k-most-likely queries and probability threshold queries by means of Matching Probability. Extensive experimental evaluations of GCI demonstrate a considerable speed-up of similarity search on both synthetic and real-world data sets.

Model-based clustering approaches concern the paradigm of exploratory data analysis relying on the finite mixture model to automatically find a latent structure governing observed data. They are one of the most popular and successful approaches in cluster analysis. The mixture density estimation is generally performed by maximizing the observed-data log-likelihood by using the expectation-maximization (EM) algorithm. However, it is well-known that the EM algorithm initialization is crucial. In addition, the standard EM algorithm requires the number of clusters to be known a priori. Some solutions have been provided in [31], [12] for model-based clustering with Gaussian mixture models for multivariate data. In this paper we focus on model-based curve clustering approaches, when the data are curves rather than vectorial data, based on regression mixtures. We propose a new robust EM algorithm for clustering curves. We extend the model-based clustering approach presented in [31] for Gaussian mixture models, to the case of curve clustering by regression mixtures, including polynomial regression mixtures as well as spline or B-spline regressions mixtures. Our approach both handles the problem of initialization and the one of choosing the optimal number of clusters as the EM learning proceeds, rather than in a twofold scheme. This is achieved by optimizing a penalized log-likelihood criterion. A simulation study confirms the potential benefit of the proposed algorithm in terms of robustness regarding initialization and funding the actual number of clusters.

The traditional Expectation-Maximization (EM) training of Gaussian Mixture Model (GMM) is essentially a batch mode procedure which requires the multiple data samples with the sufficient size to update the model parameters. This severely limits the deployment and adaptation of GMM in many real-time online systems since the newly observed data samples are expected to be incorporated into the system upon available via retraining the model. This paper presents a new online incremental EM training procedure of GMM, which aims to perform the EM training incrementally and so can adapt GMM online sample by sample. The proposed method is extended on two kinds of EM algorithms for GMM, namely, Split-and-Merge EM and the traditional EM. Experiments on both the synthetic data and a speech processing task show the advantages and efficiency of the new method.

Active curve axis Gaussian mixture models

In many modern applications such as biometric identification systems, sensor networks, medical imaging, geology, and multimedia databases, the data objects are not described exactly. Therefore, recent solutions propose to model data objects by probability density functions(pdf). Since a pdf describing an uncertain object is often not explicitly known, approximation techniques like Gaussian mixture models(GMM) need to be employed. In this paper, we introduce a method for efficiently indexing and querying GMMs allowing fast object retrieval for arbitrary shaped pdf. We consider probability ranking queries which are very important for probabilistic similarity search. Our method stores the components and weighting functions of each GMM in an index structure. During query processing the mixture models are dynamically reconstructed whenever necessary. In an extensive experimental evaluation, we demonstrate that GMMs yield a compact and descriptive representation of video clips. Additionally, we show that our new query algorithm outperforms competitive approaches when answering the given probabilistic queries on a database of GMMs comprising about 100.000 single Gaussians.

Summary form only given. This paper presents a probabilistic approach for statistical modelling of the loads in distribution networks. In a distribution network, the Probability Density Functions (pdfs) of loads at different buses show a number of variations and cannot be represented by any specific distribution. The approach presented in this paper represents all the load pdfs through Gaussian Mixture Model (GMM). The Expectation Maximization (EM) algorithm is used to obtain the parameters of the mixture components. The performance of the method is demonstrated on a 95-bus generic distribution network model.

We demonstrate that regularisation by convolution, a smoothing technique based on the convolution of Gaussian kernels with Gaussian mixture models (Molina and Niranjan, 1997), is equivalent to jittering (the addition of noise to the input data of the mixture model) (Bishop, 1995) in the case of probability density estimation. We also demonstrate that regularisation by convolution is consistent with linear transformations of the input space when used with full covariance matrix Gaussian components. Regularisation by convolution is illustrated on the probability density estimation of ink in ancient manuscript letters (British library Beowulf manuscript).

The high proportion abnormal data in photovoltaic power generation output has the characteristics of overlapping and multi Gaussian distribution. How to accurately remove the high proportion abnormal data in photovoltaic power generation output is a key issue in achieving accurate prediction of photovoltaic power. The article analyzes the characteristics of high proportion abnormal data in photovoltaic power generation output, and studies a detection method for high proportion abnormal data in photovoltaic power generation output based on Gaussian mixture model. We have established an algorithm model for the Gaussian mixture model, using the Expectation Maximization (EM) algorithm to estimate the parameters of the Gaussian mixture model, and using the algorithm model to detect and eliminate high proportion abnormal data in photovoltaic power generation output. Actual case analysis and comparative experiments show that the method proposed in the article can accurately detect high proportion abnormal data from multiple classifications, making it more suitable for detecting high proportion abnormal data in photovoltaic power generation output compared to traditional abnormal data detection methods.

The topic of colour image segmentation has been and still is a hot issue in areas such as computer vision and image processing because of its wide range of practical applications. The urge has led to the development of numerous colour image segmentation algorithms to extract salient objects from colour images. However, because of the diverse imaging conditions in varying application domains, accuracy and robustness of several state-of-the-art colour image segmentation algorithms still leave room for further improvement. This dissertation reports on the development of a new image segmentation algorithm based on perceptual colour difference saliency along with binary morphological operations. The algorithm consists of four essential processing stages which are colour image transformation, luminance image enhancement, salient pixel computation and image artefact filtering. The input RGB colour image is first transformed into the CIE L*a*b colour image to achieve perceptual saliency and obtain the best possible calibration of the transformation model. The luminance channel of the transformed colour image is then enhanced using an adaptive gamma correction function to alleviate the adverse effects of illumination variation, low contrast and improve the image quality significantly. The salient objects in the input colour image are then determined by calculating saliency at each pixel in order to preserve spatial information. The computed saliency map is then filtered using the morphological operations to eliminate undesired factors that are likely present in the colour image. A series of experiments was performed to evaluate the effectiveness of the new perceptual colour difference saliency algorithm for colour image segmentation. This was accomplished by testing the algorithm on a large set of a hundred and ninety images acquired from four distinct publicly available benchmarks corporal. The accuracy of the developed colour image segmentation algorithm was quantified using four widely used statistical evaluation metrics in terms of precision, F-measure, error and Dice. Promising results were obtained despite the fact that the experimental images were selected from four different corporal and in varying imaging conditions. The results have indeed demonstrated that the performance of the newly developed colour image segmentation algorithm is consistent with an improved performance compared to a number of other saliency and non- saliency state-of-the-art image segmentation algorithms.

Wildfires have increased in frequency, duration and size in the western United States (U.S.) over the past decades. These trends are projected to continue, with negative consequences for air quality across the U.S. Wildfires emit large quantities of particles and gases that serve as air pollutants and their precursors, and can lead to severe air quality conditions over large spatial and long temporal scales. Characterization of the chemical constituents in smoke as a function of combustion conditions, fuel type, and fuel component is an important step towards improving the prediction of air quality effects from fires and evaluating mitigation strategies. Building on the comprehensive characterization of gaseous non-methane organic compounds (NMOCs) identified in laboratory and field studies, a supervised pattern recognition algorithm was developed that successfully identified unique chemical speciation profiles among similar fuel types common in western coniferous forests. The algorithm was developed using laboratory data from single fuel species and tested on simplified synthetic fuel mixtures. The fuel types in the synthetic mixtures were differentiated but as the relative mixing proportions became more similar, the differentiation became poorer. Using the results from the pattern recognition algorithm, a classification model based on linear discriminant analysis was trained to differentiate smoke samples based on the contribution(s) of dominant fuel type(s). The classification model was applied to field data and despite the complexity of contributing fuels, and the presence of fuels "unknown" to the classifier, the dominant sources/fuel types were identified correctly. The pattern recognition and classification algorithms are a promising approach for identifying the types of fuels contributing to smoke samples and facilitating selection of appropriate chemical speciation profiles for predictive air quality modeling, using a highly reduced suite of measured NMOCs. Utility and performance of the pattern recognition and classification algorithms can be improved by expanding the training and test sets to include data from a broader range of single and mixed fuel types.

In the last decades, Gaussian Mixture Models (GMMs) have attracted considerable interest in data mining and pattern recognition. A GMM-based clustering algorithm models a dataset with a mixture of multiple Gaussian components and estimates the model parameters using the Expectation-Maximization (EM) algorithm. Recently, a new Locally Consistent GMM (LCGMM) has been proposed to improve the clustering performance by exploiting the local manifold structure of the data using a p nearest neighbor graph. In addition to the underlying manifold structure, many other forms of prior knowledge may guide the clustering process and improve the performance. In this paper, we introduce a Semi-Supervised LCGMM (Semi-LCGMM), where the prior knowledge is provided in the form of class labels of partial data. In particular, the new Semi-LCGMM incorporates the prior knowledge into the maximum likelihood function of the original LCGMM, and the model parameters are estimated using the EM algorithm. It is worth noting that, in our algorithm, each class may be modeled by multiple Gaussian components while in the unsupervised setting each class is modeled by a single Gaussian component. Our algorithm has shown promising results in many different applications, including clustering breast cancer data, heart disease data, handwritten digit images, human face images, and image segmentation.