Variable selection methods for model-based clustering

Model-based clustering approaches concern the paradigm of exploratory data analysis relying on the finite mixture model to automatically find a latent structure governing observed data. They are one of the most popular and successful approaches in cluster analysis. The mixture density estimation is generally performed by maximizing the observed-data log-likelihood by using the expectation-maximization (EM) algorithm. However, it is well-known that the EM algorithm initialization is crucial. In addition, the standard EM algorithm requires the number of clusters to be known a priori. Some solutions have been provided in [31], [12] for model-based clustering with Gaussian mixture models for multivariate data. In this paper we focus on model-based curve clustering approaches, when the data are curves rather than vectorial data, based on regression mixtures. We propose a new robust EM algorithm for clustering curves. We extend the model-based clustering approach presented in [31] for Gaussian mixture models, to the case of curve clustering by regression mixtures, including polynomial regression mixtures as well as spline or B-spline regressions mixtures. Our approach both handles the problem of initialization and the one of choosing the optimal number of clusters as the EM learning proceeds, rather than in a twofold scheme. This is achieved by optimizing a penalized log-likelihood criterion. A simulation study confirms the potential benefit of the proposed algorithm in terms of robustness regarding initialization and funding the actual number of clusters.

BackgroundClustering is an important analysis performed on microarray gene expression data since it groups genes which have similar expression patterns and enables the exploration of unknown gene functions. Microarray experiments are associated with many sources of experimental and biological variation and the resulting gene expression data are therefore very noisy. Many heuristic and model-based clustering approaches have been developed to cluster this noisy data. However, few of them include consideration of probe-level measurement error which provides rich information about technical variability.ResultsWe augment a standard model-based clustering method to incorporate probe-level measurement error. Using probe-level measurements from a recently developed Affymetrix probe-level model, multi-mgMOS, we include the probe-level measurement error directly into the standard Gaussian mixture model. Our augmented model is shown to provide improved clustering performance on simulated datasets and a real mouse time-course dataset.ConclusionThe performance of model-based clustering of gene expression data is improved by including probe-level measurement error and more biologically meaningful clustering results are obtained.

Several methods for variable selection have been proposed in model-based clustering and classification. These make use of backward or forward procedures to define the roles of the variables. Unfortunately, such stepwise procedures are slow and the resulting algorithms inefficient when analyzing large data sets with many variables. In this paper, we propose an alternative regularization approach for variable selection in model-based clustering and classification. In our approach the variables are first ranked using a lasso-like procedure in order to avoid slow stepwise algorithms. Thus, the variable selection methodology of Maugis et al. (Comput Stat Data Anal 53:3872–3882, 2009b) can be efficiently applied to high-dimensional data sets.

Numerous variable selection techniques have been developed for complete high-dimensional data but very few of them for censored data. The techniques for complete data must be modified if censoring is present. In this paper, we consider the variable selection technique for accelerated failure time (AFT) models by extending the ranking-based variable selection (RBVS) algorithm and its iterative procedure as proposed in the work of Baranowski et al. through the Stute’s weighted least square technique. Simulation studies are conducted to demonstrate the performance of the proposed methods. We further illustrate the performance of this method with a mantle cell lymphoma microarray example. When there is no correlation among the covariates, the proposed method outperforms the iterative sure independence screening and stability selection methods in terms of overall performance for high-dimensional data. Real data analysis also suggests that the proposed method can be chosen for high-dimensional censored data analysis in parallel to other methods in the literature.

Over the last decade, a large variety of clustering algorithms have been developed to detect coregulatory relationships among genes from microarray gene expression data. Model-based clustering approaches have emerged as statistically well-grounded methods, but the properties of these algorithms when applied to large-scale data sets are not always well understood. An in-depth analysis can reveal important insights about the performance of the algorithm, the expected quality of the output clusters, and the possibilities for extracting more relevant information out of a particular data set. We have extended an existing algorithm for model-based clustering of genes to simultaneously cluster genes and conditions, and used three large compendia of gene expression data for Saccharomyces cerevisiae to analyze its properties. The algorithm uses a Bayesian approach and a Gibbs sampling procedure to iteratively update the cluster assignment of each gene and condition. For large-scale data sets, the posterior distribution is strongly peaked on a limited number of equiprobable clusterings. A GO annotation analysis shows that these local maxima are all biologically equally significant, and that simultaneously clustering genes and conditions performs better than only clustering genes and assuming independent conditions. A collection of distinct equivalent clusterings can be summarized as a weighted graph on the set of genes, from which we extract fuzzy, overlapping clusters using a graph spectral method. The cores of these fuzzy clusters contain tight sets of strongly coexpressed genes, while the overlaps exhibit relations between genes showing only partial coexpression. GaneSh, a Java package for coclustering, is available under the terms of the GNU General Public License from our website at http://bioinformatics.psb.ugent.be/software

Evaluation of the effect of chance correlations on variable selection using Partial Least Squares-Discriminant Analysis

Clustering is a useful exploratory technique for the analysis of gene expression data. Many different heuristic clustering algorithms have been proposed in this context. Clustering algorithms based on probability models offer a principled alternative to heuristic algorithms. In particular, model-based clustering assumes that the data is generated by a finite mixture of underlying probability distributions such as multivariate normal distributions. The issues of selecting a 'good' clustering method and determining the 'correct' number of clusters are reduced to model selection problems in the probability framework. Gaussian mixture models have been shown to be a powerful tool for clustering in many applications. We benchmarked the performance of model-based clustering on several synthetic and real gene expression data sets for which external evaluation criteria were available. The model-based approach has superior performance on our synthetic data sets, consistently selecting the correct model and the number of clusters. On real expression data, the model-based approach produced clusters of quality comparable to a leading heuristic clustering algorithm, but with the key advantage of suggesting the number of clusters and an appropriate model. We also explored the validity of the Gaussian mixture assumption on different transformations of real data. We also assessed the degree to which these real gene expression data sets fit multivariate Gaussian distributions both before and after subjecting them to commonly used data transformations. Suitably chosen transformations seem to result in reasonable fits. MCLUST is available at http://www.stat.washington.edu/fraley/mclust. The software for the diagonal model is under development. kayee@cs.washington.edu. http://www.cs.washington.edu/homes/kayee/model.

An often overlooked pitfall of model-based clustering is that it typically results in the same number of clusters per margin, an assumption that may not be natural in practice. We develop a clustering method that takes advantage of the sturdiness of model-based clustering, while attempting to mitigate this issue. The proposed approach allows each margin to have a varying number of clusters and employs a strategy game-inspired algorithm, named ‘Reign-and-Conquer’, to cluster the data. Since the proposed clustering approach only specifies a model for the margins, but leaves the joint unspecified, it has the advantage of being partially parallelizable; hence, the proposed approach is computationally appealing as well as more tractable for moderate to high dimensions than a ‘full’ (joint) model-based clustering approach. A battery of numerical experiments on simulated data indicates an overall good performance of the proposed methods in a variety of scenarios, and real datasets are used to showcase their usefulness in practice.

Smart meter data analysis is essential for balancing energy consumption and minimizing power outages. However, high-resolution smart meter readings pose challenges to data analysis due to their high volume and dimensions. We propose Online-FDA, an online functional load demand analysis and forecasting framework that incorporates real-time smart meter readings with adaptive clustering to identify daily patterns in functional load consumption and predict daily load demands. This framework utilizes a model-based functional clustering approach assisted by the intra-day load consumption attributes to analyze real-time smart meter data. Moreover, the Online-FDA augments the clusters with a state-of-the-art functional deep neural network that utilizes the training-testing-updating strategy to adaptively learns from real-time smart meter data. Experimental results with real-world smart meter data showed that the proposed Online-FDA is superior to other benchmark algorithms for capturing time-varying variations in load demand, which are essential to the real-time control of electricity grids and the planning of power systems.

An efficient swarm intelligence approach to feature selection based on invasive weed optimization: Application to multivariate calibration and classification using spectroscopic data

High‐dimensional data analysis emerges in many different areas of scientific research and social practice. Variable selection is an important task in statistical analysis for high‐dimensional data. The traditional all‐subset‐selection method can be ideal for variable selection, but it is computationally intractable in the high‐dimensional setting. In both regression and classification contexts, sparsity penalties are commonly used for the purpose of defining the complexity of the resulting model so as to achieve variable selection. The current article reviews a few important milestones and some recent works in the area of variable selection, especially those methods which useLqnorm and their variants as the penalty term. In particular, we review theLqpenalty, nonconvex penalties, among others. In ultrahigh‐dimensional data analysis, independence learning is often used for the purpose of dimension reduction. Theoretical results, methodological developments and computational innovations in regard to these methods are discussed. These variable selection techniques can be easily extended to problems beyond linear regression, such as classification, quantile regression, etc. Lastly, an interesting and promising research trend is the combination of multiple methods and we review several successful methods which fall into this category.This article is categorized under:Statistical and Graphical Methods of Data Analysis > Modeling Methods and AlgorithmsStatistical Models > Model SelectionStatistical Models > Linear Models

Variable (wavelength) selection is critical in the multivariate calibration of spectra that improves prediction performance and provides clearer interpretation. Most variable selection methods use Y information (chemical reference of interest) to evaluating the importance of variables. In this work, a novel variable selection is proposed that requires no reference information. Replicate spectra of a sample under various acquisition conditions were used with an index to evaluate the importance of variables. The results show that the proposed method achieves better results than the original partial least squares model with a few available calibration samples. When a large number of calibration samples are available, this method provided comparable results to the state-of-the-art variable selection methods while reduced the risk of model over-fitting.

Coupling fast variable selection methods to neural network-based classifiers: Application to multisensor systems

Background:One substantial part of microarray studies is to predict patients’ survival based on their gene expression profile. Variable selection techniques are powerful tools to handle high dimensionality in analysis of microarray data. However, these techniques have not been investigated in competing risks setting. This study aimed to investigate the performance of four sparse variable selection methods in estimating the survival time.Methods:The data included 1381 gene expression measurements and clinical information from 301 patients with bladder cancer operated in the years 1987 to 2000 in hospitals in Denmark, Sweden, Spain, France, and England. Four methods of the least absolute shrinkage and selection operator, smoothly clipped absolute deviation, the smooth integration of counting and absolute deviation and elastic net were utilized for simultaneous variable selection and estimation under an additive hazards model. The criteria of area under ROC curve, Brier score and c-index were used to compare the methods.Results:The median follow-up time for all patients was 47 months. The elastic net approach was indicated to outperform other methods. The elastic net had the lowest integrated Brier score (0.137±0.07) and the greatest median of the over-time AUC and C-index (0.803±0.06 and 0.779±0.13, respectively). Five out of 19 selected genes by the elastic net were significant (P<0.05) under an additive hazards model. It was indicated that the expression of RTN4, SON, IGF1R and CDC20 decrease the survival time, while the expression of SMARCAD1 increase it.Conclusion:The elastic net had higher capability than the other methods for the prediction of survival time in patients with bladder cancer in the presence of competing risks base on additive hazards model.

We introduce a fast approach to classification and clustering applicable to high-dimensional continuous data, based on Bayesian mixture models for which explicit computations are available. This permits us to treat classification and clustering in a single framework, and allows calculation of unobserved class probability. The new classifier is robust to adding noise variables as a drawback of the built-in spike-and-slab structure of the proposed Bayesian model. The usefulness of classification using our method is shown on metabololomic example, and on the Iris data with and without noise variables. Agglomerative hierarchical clustering is used to construct a dendrogram based on the posterior probabilities of particular partitions, to provide a dendrogram with a probabilistic interpretation. An extension to variable selection is proposed which summarises the importance of variables for classification or clustering and has probabilistic interpretation. Having a simple model provides estimation of the model parameters using maximum likelihood and therefore yields a fully automatic algorithm. The new clustering method is applied to metabolomic, microarray, and image data and is studied using simulated data motivated by real datasets. The computational difficulties of the new approach are discussed, solutions for algorithm acceleration are proposed, and the written computer code is briefly analysed. Simulations shows that the quality of the estimated model parameters depends on the parametric distribution assumed for effects, but after fixing the model parameters to reasonable values, the distribution of the effects influences clustering very little. Simulations confirms that the clustering algorithm and the proposed variable selection method is reliable when the model assumptions are wrong. The new approach is compared with the popular Bayesian clustering alternative, MCLUST, fitted on the principal components using two loss functions in which our proposed approach is found to be more efficient in almost every situation.