Eigenvalues and constraints in mixture modeling: geometric and computational issues

A common and old problem in statistics is the separation of a heterogeneous population into more homogeneous subpopulations. A wide variety of approaches and techniques for tackling this problem now exist. The finite mixture model is one such technique used to analyse a given data set and create natural groups. The main advantage of mixture models is that they offer a very flexible and relatively easy method of fitting to a data set.The main focus of this thesis will be the clustering of data with the aid of mixed variable mixture models. There has been a lot of research and many papers written with regard to the analysis of continuous variables. This thesis, however, will be concerned with the techniques that will be able to analyse data from continuous to mixed variables, a combination of continuous and discrete variables.The expectation-maximization (EM) algorithm of Dempster et al. (1977) will be used in this thesis for the clustering of mixed variable data. As demonstrated effectively in numerous publications, this algorithm solves estimation of the model parameters iteratively under an advantageous property of monotonic convergence. An extension of the EM algorithm, the ECM algorithm introduced by Meng and Rubin (1993), will also be considered in this thesis for itsrole within more complicated mixed mixture models.On the application of mixed mixture models, this thesis will use a joint distribution specified by the conditional distribution of the continuous variables, given the values of the discrete variables times the marginal distribution of the latter. There are three models - naive, logistic and multinomial - considered for the discrete variables and two models - independent and location - considered for the continuous variables. These models are combined to make a total of sixmodels for the analysis of mixed variable data.The program EMM, its design and development will be discussed in this thesis. Many real mixed data sets and some simulations will also be analysed to investigate how these proposed methods will compare with existing methods. Some of these data sets will be transformed so as to create a suitable input for the program and hence provide a comparable output so that comparisons can be made.The program EMM is written in the FORTRAN computer language. This means that it is well organised with each segment arranged in separate sub-routines. This allows for easy modification to any part of the program or alternatively the addition of a new subroutine so as to increase the capabilities of the model.

Technological advances led to the generation of large scale complex data. Thus, extraction and retrieval of information to automatically discover latent pattern have been largely studied in the various domains of science and technology. Consequently, machine learning experienced tremendous development and various statistical approaches have been suggested. In particular, data clustering has received a lot of attention. Finite mixture models have been revealed to be one of the flexible and popular approaches in data clustering. Considering mixture models, three crucial aspects should be addressed. The first issue is choosing a distribution which is flexible enough to fit the data. In this paper, a model based on multivariate Beta distributions is proposed. The two other challenges in mixture models are estimation of model's parameters and model complexity. To tackle these challenges, variational inference techniques demonstrated considerable robustness. In this paper, two methods are studied, namely, batch and online variational inferences and the models are evaluated on four medical applications including image segmentation of colorectal cancer, multi‐class colon tissue analysis, digital imaging in skin lesion diagnosis and computer aid detection of Malaria.

A mixture of SDB skew- t factor analyzers

On model-based clustering of skewed matrix data

In recent years, work has been carried out on clustering gene expression microarray data. Some approaches are developed from an algorithmic viewpoint whereas others are developed via the application of mixture models. In this article, a family of eight mixture models which utilizes the factor analysis covariance structure is extended to 12 models and applied to gene expression microarray data. This modelling approach builds on previous work by introducing a modified factor analysis covariance structure, leading to a family of 12 mixture models, including parsimonious models. This family of models allows for the modelling of the correlation between gene expression levels even when the number of samples is small. Parameter estimation is carried out using a variant of the expectation-maximization algorithm and model selection is achieved using the Bayesian information criterion. This expanded family of Gaussian mixture models, known as the expanded parsimonious Gaussian mixture model (EPGMM) family, is then applied to two well-known gene expression data sets. The performance of the EPGMM family of models is quantified using the adjusted Rand index. This family of models gives very good performance, relative to existing popular clustering techniques, when applied to real gene expression microarray data. The reduced, preprocessed data that were analysed are available at www.paulmcnicholas.info

This chapter introduces several fundamental models and algorithms for probabilistic clustering, including mixture models, Expectation-Maximization (EM) algorithm, and probabilistic topic models. It focuses on general framework of modeling, the probabilistic explanation, the standard algorithms to learn the model, and its applications. Mixture models are probabilistic models which are increasingly used to find the clusters for univariate and multivariate data. The chapter begins the discussion of mixture models, in which the values of the discrete latent variables can be interpreted as the assignments of data points to specific components of the mixture. It uses the Gaussian mixture model to motivate the EM algorithm in an informal way, and then gives a more general view of the EM algorithm, which is a standard learning algorithm for many probabilistic models. The chapter presents two popular probabilistic topic models, i.e., probabilistic latent semantic analysis (PLSA) and latent Dirichlet allocation (LDA), for document clustering and analysis.

Mixture modeling using elliptical distributions promises enhanced robustness, flexibility, and stability over the widely employed Gaussian mixture model (GMM). However, existing studies based on the elliptical mixture model (EMM) are restricted to several specific types of elliptical probability density functions, which are not supported by general solutions or systematic analysis frameworks; this significantly limits the rigor in the design and power of EMMs in applications. To this end, we propose a novel general framework for estimating and analyzing the EMMs, achieved through the Riemannian manifold optimization. First, we investigate the relationships between Riemannian manifolds and elliptical distributions, and the so established connection between the original manifold and a reformulated one indicates a mismatch between these manifolds, a major cause of failure of the existing optimization for solving general EMMs. We next propose a universal solver that is based on the optimization of a redesigned cost and prove the existence of the same optimum as in the original problem; this is achieved in a simple, fast and stable way. We further calculate the influence functions of the EMM as theoretical bounds to quantify robustness to outliers. Comprehensive numerical results demonstrate the ability of the proposed framework to accommodate EMMs with different properties of individual functions in a stable way and with fast convergence speed. Finally, the enhanced robustness and flexibility of the proposed framework over the standard GMM are demonstrated both analytically and through comprehensive simulations.

This special issue of Cognitive Computation includes 11 original articles, which have been selected among the highest quality submissions to the 2012 Brain Inspired Cognitive Systems (BICS 2012) Conference. BICS 2012 provided a high-level international forum for scientists, engineers, and educators to present the state-of-the-art research on brain inspired computing and cognitive systems, with applications in multi-disciplinary fields. The conference featured plenary speeches given by worldwide renowned scholars, regular sessions with broad coverage, and some special sessions focusing on interesting topics for the related scientific community. Based on the recommendation of symposium organizers and reviewers, a number of authors were invited to resubmit an extended version of their contributions, originally submitted as conference papers, for this special issue of Cognitive Computation. All these journal articles went through the same rigorous review procedure by at least three independent experts before being accepted for publication. This special issue focuses on recent advancements in the field of brain inspired computing. The selected 11 articles can be divided into two main groups. The first group consists of four papers, and it is more oriented to new architectures and algorithms, whereas the second group contains the remaining seven papers, dealing with challenging applications in diverse areas. The special issue starts with the first group of papers and with the contribution by Kozma and Puljic, titled ‘‘Learning Effects in Coupled Arrays of Cellular Neural Oscillators,’’ where the authors analyze the spatio-temporal dynamics of coupled neural oscillatory arrays. In particular, a neuropercolation model encompassing the Freeman principles of neurodynamics is used, and the modulation effects due to distributed input biases in interconnected excitatory-inhibitory oscillators spanning the lattice graph are investigated. It is shown that the proposed neuropercolation model is able to generate large-scale synchronized, narrow-band oscillations in response to learned stimuli, as observed in EEG and ECoG experiments. The second contribution by Xunan Zhang et al., addresses the Bayesian classification problem in the presence of incomplete data. The usual approach adopted in the literature consists in simply ignoring the samples with missing values or imputing missing values before classification. This is not really effective when the amount of missing values is considerable and/or the data acquisition phase is expensive. The authors thus propose an innovative expectation–maximization-based learning algorithm, applying it to a multivariate Gaussian mixture model and a multiple kernel density estimator. Such a technique is able to avoid list-wise deletion or mean imputation in solving classification tasks with incomplete data. Experimental tests on several benchmark problems and also on practical classification tasks on the lithology identification of hydrothermal minerals and license plate character recognition have shown that the proposed approach shows remarkable classification S. Squartini (&) Department of Information Engineering, UniversitaPolitecnicadelle Marche, Via BrecceBianche 1, 60131 Ancona, Italy e-mail: s.squartini@univpm.it

Reviewed by: Verb clusters: A study of Hungarian, German and Dutch ed. by Katalin É. Kiss, Henk van Riemsdijk Hubert Haider Verb clusters: A study of Hungarian, German and Dutch. Ed. by Katalin É. Kiss and Henk van Riemsdijk. Amsterdam: John Benjamins, 2004. Pp. vi, 514. ISBN 1588115070. $174 (Hb). This volume assembles fifteen contributions: a detailed introduction to verb clustering in Dutch, German, and Hungarian by the editors; two questionnaire studies on German and Hungarian clustering data by Susanne Wurmbrand and by Kriszta Szendröi and Ildikó Toth, respectively; a review of clustering theories by Jonathan David Bobaljik; two demonstrations by Michael Brody and by Edwin Williams of how to apply their theory to verb clustering; five papers with a primary focus on Hungarian by Szendröi, Anikó Csirmaz, Gabor Alberti, Csaba Olsvay, and Huba Bartos; and four papers that model clustering and clause union in a systematic comparative perspective (Hungarian, Dutch, German) by Peter Ackema, Marcel den Dikken, É. Kiss, and Tóth. Verb clustering still is an insufficiently understood syntactic phenomenon. From the perspective of English (or any other VO language), clustering constructions like those of West Germanic OV languages appear to be out of the ordinary. However, clustering is not a peculiarity of West Germanic. A non-Indo-European language like Hungarian displays clustering properties with virtually the same classes of verbs and with strikingly similar syntactic effects. In the past three decades, a great variety of different syntactic analyses have been designed to capture clustering and its manifold side effects. The predominant strategy has been a derivational one. Clustering is seen as the result of syntactic processes that operate on structures of verbal complementation, and in particular, on verbal projections that are complements of verbs. Various kinds of syntactic tools have been tried out: Verbs may raise out of their projection and adjoin to a higher verb or to a functional head, or verbal projections are emptied by removing everything except the verbal head, before they are moved, or a given complement structure is restructured by means of ‘reanalysis’. What these tools have in common is that they could be implemented in the grammar of any language. In other words, the various approaches remain silent on a crucial question: Why is clustering and clause union typologically restricted to OV languages and why should verbs cluster at all? A descriptively adequate clustering theory should take a stand on the appropriateness of the generalization that clustering correlates with head-final verbal projections (see Bobaljik’s paper [End Page 647] ‘Clustering theories’, 121–45; Haider 2003). Head-initial languages do not cluster the verbs of stacked verbal projections, but in OV languages, stacked head-final projections seem to generally cluster. For clustering properties of OV languages of different language families, see Sells 1990 on Korean and Japanese. The clustering symptoms involve at least the following properties: first, clustered verbs are a compact syntactic unit. The verbs are adjacent. Intervening nonverbal elements except for verbal particles and particle-like elements are ungrammatical. Second, if a language displays variable orders for nonfinite verbs in a sequence of verbal projections, the language is a clustering language. The first property is an indicator of the syntactic organization of a verb cluster. The second property, namely reordering variation, is a concomitant effect of head-final clustering. Head-final clusters may be partially, and in many cases optionally, turned into head-initial clusters. Consistently head-initial languages (i.e. VO languages and VSO languages) do not cluster, nor do they reorder nonfinite verbs of stacked verbal projections, VP-transparency phenomena notwithstanding (Roberts 1997). Hungarian is clustering but it is certainly not a standard VO language. According to É. Kiss 1987 and literature cited there, the Hungarian VP is ‘non-configurational’, that is, neither strictly head-final nor strictly head-initial. As noted in the introductory section (36), ‘typologists do tend to classify Hungarian as an OV language on the basis of various properties’. A third quality of clustering constructions is the clause union effect. The main evidence comes from elements that are...

Nowadays smartphones get increasingly popular which also attracted hackers. With the increasing capabilities of such phones, more and more malicious softwares targeting these devices have been developed. Malwares can seriously damage an infected device within seconds. This paper focus on the aggregation of a popular probabilistic model: the Gaussian mixture model, for a dynamic detection of the abnormal behavior in smartphone applications. More precisely, we propose to apply a mixture model estimation technique on the behavior of applications, for density modeling and data clustering. The mixture models of the different smartphones are then aggregated to estimate the global model that reflecting the probability density of the global data set. Furthermore, we carry out a model-based clustering outlier detection to compute an anomaly score for each application, leading to identify the malware applications. Initial experiments results prove the efficiency and the accuracy of the model-based clustering in detecting abnormal applications with a low false alerts rate.

Finite mixture models are an important tool in the statistical analysis of data, for example in data clustering. The optimal parameters of a mixture model are usually computed by maximizing the log-likelihood functional via the Expectation-Maximization algorithm. We propose an alternative approach based on the theory of Mean Field Games, a class of differential games with an infinite number of agents. We show that the solution of a finite state space multi-population Mean Field Games system characterizes the critical points of the log-likelihood functional for a Bernoulli mixture. The approach is then generalized to mixture models of categorical distributions. Hence, the Mean Field Games approach provides a method to compute the parameters of the mixture model, and we show its application to some standard examples in cluster analysis.

Model-based clustering methods for continuous data are well established and commonly used in a wide range of applications. However, model-based clustering methods for categorical data are less standard. Latent class analysis is a commonly used method for model-based clustering of binary data and/or categorical data, but due to an assumed local independence structure there may not be a correspondence between the estimated latent classes and groups in the population of interest. The mixture of latent trait analyzers model extends latent class analysis by assuming a model for the categorical response variables that depends on both a categorical latent class and a continuous latent trait variable; the discrete latent class accommodates group structure and the continuous latent trait accommodates dependence within these groups. Fitting the mixture of latent trait analyzers model is potentially difficult because the likelihood function involves an integral that cannot be evaluated analytically. We develop a variational approach for fitting the mixture of latent trait models and this provides an efficient model fitting strategy. The mixture of latent trait analyzers model is demonstrated on the analysis of data from the National Long Term Care Survey (NLTCS) and voting in the U.S. Congress. The model is shown to yield intuitive clustering results and it gives a much better fit than either latent class analysis or latent trait analysis alone.

This paper considers a hierarchical Bayesian analysis of regression models using a class of Gaussian scale mixtures. This class provides a robust alternative to the common use of the Gaussian distribution as a prior distribution in particular for estimating the regression function subject to uncertainty about the constraint. For this purpose, we use a family of rectangular screened multivariate scale mixtures of Gaussian distribution as a prior for the regression function, which is flexible enough to reflect the degrees of uncertainty about the functional constraint. Specifically, we propose a hierarchical Bayesian regression model for the constrained regression function with uncertainty on the basis of three stages of a prior hierarchy with Gaussian scale mixtures, referred to as a hierarchical screened scale mixture of Gaussian regression models (HSMGRM). We describe distributional properties of HSMGRM and an efficient Markov chain Monte Carlo algorithm for posterior inference, and apply the proposed model to real applications with constrained regression models subject to uncertainty.

Mixture models have been widely used for data clustering. However, commonly used mixture models are generally of a parametric form (e.g., mixture of Gaussian distributions or GMM), which significantly limits their capacity in fitting diverse multidimensional data distributions encountered in practice.We propose a non-parametric mixture model (NMM) for data clustering in order to detect clusters generated from arbitrary unknown distributions, using non-parametric kernel density estimates. The proposed model is non-parametric since the generative distribution of each data point depends only on the rest of the data points and the chosen kernel. A leave-one-out likelihood maximization is performed to estimate the parameters of the model. The NMM approach, when applied to cluster high dimensional text datasets significantly outperforms the state-of-the-art and classical approaches such as K-means, Gaussian Mixture Models, spectral clustering and linkage methods.

A modified procedure for mixture-model clustering of regional geochemical data