An Exploratory Study of the Inputs for Ensemble Clustering Technique as a Subset Selection Problem

Observations from hyperspectral imaging sensors lead to high dimensional data sets from hundreds of images taken at closely spaced narrow spectral bands. High storage and transmission requirements, computational complexity, and statistical modeling problems combined with physical insight motivate the idea of hyperspectral dimensionality reduction using band subset selection. Many algorithms are described in the literature to solve supervised and unsupervised band subset selection problems. This paper explores the use of unsupervised band subset selection methods using column subset selection (CSS). Column subset selection is the problem (CSSP) of selecting the most independent columns of a matrix. A recent variant of this problem is the positive column subset selection problem (pCSSP) which restricts column subset selection to only consider positive linear combinations. Many algorithms have been proposed in the literature for the solution of the CSSP. However, the pCSSP is less studied. This paper will present a comparison of different algorithms to solve the CSSP and the pCSSP for band subset selection. The performance of classifiers using the algorithms as a dimensionality reduction stage will be used to evaluate the usefulness of these algorithms in hyperspectral image exploitation.

Ensemble Clustering has been developed to provide an alternative way of obtaining more stable and accurate clustering results. It aims to avoid the biases of individual clustering algorithms. However, it is still a challenge to develop an efficient and robust method for Ensemble Clustering. Based on an existing ensemble clustering method, Consensus Clustering (CC), this paper introduces an advanced Consensus Clustering algorithm called Multi-Optimisation Consensus Clustering (MOCC), which utilises an optimised Agreement Separation criterion and a Multi-Optimisation framework to improve the performance of CC. Fifteen different data sets are used for evaluating the performance of MOCC. The results reveal that MOCC can generate more accurate clustering results than the original CC algorithm.KeywordsConsensus ClusteringSimulated AnnealingMulti-optimisation

Evolutionary algorithms have been widely used to solve multi-objective optimization problems. Usually, the final population of an evolutionary algorithm is used as the output of multi-objective optimization. However, a current new trend is to select a pre-specified number of solutions from an unbounded external archive (UEA) as the final output of multi-objective optimization. Some subset selection methods have been proposed in the literature such as hypervolume-based and IGD-based selection. Recently, a distance-based subset selection (DSS) method was proposed for efficient subset selection from a large external archive. Whereas DSS efficiently finds a set of uniformly distributed solutions, it has some difficulties in the handling of solutions in the UEA as we demonstrate in this paper. To improve the performance of the DSS method, we propose a modified DSS method based on the IGD+ distance instead of the Euclidean distance. Experimental results on various benchmark problems show that the modified DSS method performs better than or equal to the original DSS method on most test problems.

Consensus clustering methods are motivated by the success of combining multiple classifiers in many areas. In this paper, graph-based consensus clustering is used to improve the quality of chemical compound clustering by enhancing the robustness, novelty, consistency and stability of individual clusterings. For this purpose, Hyper-Graph Partitioning Algorithm (HGPA) [1], was applied. The clustering is evaluated based on the ability to separate actives from inactives molecules in each cluster and the results were compared with the Ward's clustering method. The chemical dataset MDL Drug Data Report (MDDR) database has been used for experiments. The MDL Drug Data Report (MDDR) database consists of 102516 molecules. For the experiments, the dataset DS1 was chosen from the MDDR database. This dataset has been used for many virtual screening experiments [2-4]. The dataset DS1contains 10 heterogeneous activity classes (8568 molecules). For the clustering experiments, two 2D fingerprint descriptors will be used which are developed by Scitegic's Pipeline Pilot [5]. These are 120-bit ALOGP and 1024-bit extended connectivity fingerprints (ECFP_4). The results were evaluated based on the effectiveness of the methods to separate actives from non-actives molecules using QPI- (for quality partition index) measure, which was devised by Varin et al. [6]. As defined by [7], an active cluster as a non-singleton cluster for which the percentage of active molecules in the cluster is greater than the percentage of active molecules in the dataset as a whole. Let p be the number of actives in active clusters, q the number of inactives in active clusters, r the number of actives in inactive clusters (i.e., clusters that are not active clusters) and s the number of singleton actives. The high value occurs when the actives are clustered tightly together and separated from the inactive molecules. Then the quality partition index, QPI, is defined to be: QPI=pp+q+r+s (1) Then, the results will be compared with Ward's individual clustering method, the standard clustering method for chemoinformatics applications. The generation process has been done by multiple run of K-means algorithms, each with random initialization of cluster centroids. The number of partitions generated in this step is ranged between n = 5 to n = 50, with 5-times step. Then, all the generated partitions were combined using HGPA to obtain the consensus partition. This process is done for each fingerprint (ALOGP and ECFP_4). The mean of QPI values are averaged over the ten activity classes of the datasets. Tables ​Tables1,1, ​,22 show the effectiveness of MDDR dataset clustering using ALOGP and ECFP_4 fingerprints. The best PQI value of consensus clustering methods for each column has been bold-faced for ease of reference. Table 1 Effectivenss of clustering of high diverse MDDR dataset: ALOGP Fingerprint. Table 2 Effectivenss of clustering of high diverse MDDR dataset: ECFP_4 Fingerprint. Visual inspection of the results enables comparisons to be made between the effectiveness of clustering of MDDR datasets and Ward's method, the best of choice clustering method for chemoinformatics applications. In addition, ten times of consensus clustering, for each fingerprint were observed in order to study the effectiveness of consensus clustering with different ensemble sizes. The results show that HGPA consensus clustering gives robust and novel result when K-means algorithm is run 20-50 times using ALOGP. The performance of consensus clustering outperforms the Wards' method. For consensus of dataset which represented by ECFP_4 fingerprint, the best QPI values of consensus clustering are obtained from ensembles of size n = 20-50. The performance of consensus clustering gives robust results which are better than overall performance of individual clusterings. The values of QPI in both datasets for consensus clustering are close to the Wards method. The consensus clustering, HGPA, provide stable clusters by decreasing the sensitivity to noise and outliers. The average percentages of singleton clusters of individual clusterings compared with consensus clustering for both fingerprints. The results show that consensus clustering partition the datasets with average percentage of singleton equal to zero, which is much better than individual clusterings and Wards' method. For example, 16.72% of molecules of DS1 are clustered as singletons when Wards method is applied on ALOGP fingerprint with number of clusters equal to 1000 clusters. Finally we conclude that graph-based consensus clustering can improve the effectiveness of chemical compounds clustering. The performance of consensus clustering is more robust, novel, stable, consistent, and out-perform Wards' method in case of using ALOGP fingerprint. By using ECFP_4 fingerprint, consensus clustering methods provide more robust, stable, and consistent clustering and close to the Wards clustering results. The experiments reported here suggest that graph-based consensus clustering can improve the quality of individual clustering by using the efficient algorithm, K-means algorithm, to generate the ensemble with size (20-50) for both structurally diverse chemical datasets.

ObjectiveCluster analysis of spatio-temporal event-related potential (ERP) data is a promising tool for exploring the measurement time window of ERPs. However, even after preprocessing, the remaining noise can result in uncertain cluster maps followed by unreliable time windows while clustering via conventional clustering methods. MethodsWe designed an ensemble deep clustering pipeline to determine a reliable time window for the ERP of interest from temporal concatenated grand average ERP data. The proposed pipeline includes semi-supervised deep clustering methods initialized by consensus clustering and unsupervised deep clustering methods with end-to-end architectures. Ensemble clustering from those deep clusterings was used by the designed adaptive time window determination to estimate the time window. ResultsAfter applying simulated and real ERP data, our method successfully obtained the time window for identifying the P3 components (as the interest of both ERP studies) while additional noise (e.g., adding 20 dB to −5 dB white Gaussian noise) was added to the prepared data. ConclusionCompared to the state-of-the-art clustering methods, a superior clustering performance was yielded from both ERP data. Furthermore, more stable and precise time windows were elicited as the noise increased. SignificanceOur study provides a complementary understanding of identifying the cognitive process using deep clustering analysis to the existing studies. Our finding suggests that deep clustering can be used to identify the ERP of interest when the data is imperfect after preprocessing.

SummaryCluster ensemble combines multiple base clustering results in a suitable way to improve the accuracy of the clustering result. In the conventional cluster ensemble frameworks, pairwise constraints and constraint projections have not been used together, and spectral clustering algorithm is rarely adopted to serve as the consensus function. In this paper, we design a constraint projections for semi‐supervised spectral clustering ensemble (CPSSSCE) model. It takes advantages of spectral clustering algorithm and executes semi‐supervised learning twice. Compared to traditional cluster ensemble approaches, CPSSSCE is characterized by several properties. First, the original data are transformed to lower‐dimensional representations by constraint projection before base clustering. Second, a similarity matrix is constructed using the base clustering results and modified using pairwise constraints. Third, the spectral clustering algorithm is applied to process the similarity matrix to obtain a consensus cluster result. Extensive experiments on standard University of California Irvine Machine Learning Repository (UCI) and Microsoft datasets demonstrated that the CPSSSCE is superior to other cluster ensemble algorithms including a semi‐supervised spectral clustering ensemble.

Given m d -dimensional responsors and n d -dimensional predictors, sparse regression finds at most k predictors for each responsor for linear approximation, 1 ≤ k ≤ d-1 . The key problem in sparse regression is subset selection, which usually suffers from high computational cost. In recent years, many improved approximate methods of subset selection have been published. However, less attention has been paid to the nonapproximate method of subset selection, which is very necessary for many questions in data analysis. Here, we consider sparse regression from the view of correlation and propose the formula of conditional uncorrelation. Then, an efficient nonapproximate method of subset selection is proposed in which we do not need to calculate any coefficients in the regression equation for candidate predictors. By the proposed method, the computational complexity is reduced from O([1/6]k3+(m+1)k2+mkd) to O([1/6]k3+[1/2](m+1)k2) for each candidate subset in sparse regression. Because the dimension d is generally the number of observations or experiments and large enough, the proposed method can greatly improve the efficiency of nonapproximate subset selection. We also apply the proposed method in real scenarios of dental age assessment and sparse coding to validate the efficiency of the proposed method.

Prostate cancer is the commonly diagnosed cancer worldwide, and there were 1,276 thousand new prostate cancer cases and 359 thousand deaths in 2018. Prostate-specific antigen (PSA) blood level is often elevated in men with prostate cancer, so PSA testing can detect prostate tumours when they are small, low-grade, and localized. The PSA testing is hard to apply on the less developed and poor areas without sufficient medical funds, so the early accurate PSA level prediction by statistical machine learning models is significant to avoid later stages of prostate cancer that spread outside the Prostate. In this literature, we compare three linear model selection and regularization methods (shrinkage, subset selection, dimension reduction) and nine candidate models (OLS regression, Ridge regression, Lasso regression, Elastic net, best subset selection, forward subset selection, backward subset selection, PCR, PLS) based on leave-one-out-cross-validation (LOOCV) prediction error. As the selection criteria leave-one-out cross-validation is sensitive to outliers, Mahalanobis distance is used for outlier detection and deletion before running each model. The shrinkage method (only lasso and elastic net models) and subset selection method (based on adjusted <inline-formula> <tex-math notation="LaTeX">$R^{2}$ </tex-math></inline-formula>, BIC, Cp, and cross-validation prediction error) can select the variables out. The feature selection results show that prostate weight, cancer volume, amount of benign prostatic hyperplasia, and whether seminal vesicle invasion is necessary variables must include predicting PSA. Age and capsular penetration are the least important variables. The variables of Gleason score, a percent of Gleason scores 4 or 5 are essential sometimes. All the diagnostic figures and results are coded by R, open access, and published on IEEE Xplore Code Ocean.

Ensemble clustering has recently emerged as a powerful tool for aggregating multiple clustering results into a probably better and more robust clustering result. While many ensemble clustering techniques have been developed in recent years, there is still a potential drawback in most of the existing algorithms. They generally neglect the possible noise inside the ensemble of base clusterings, which may exhibit a negative influence on the robustness of the consensus result. To address this, this paper proposes a new ensemble clustering algorithm based on noise-aware graph decomposition. The co-association matrix is first constructed to accumulate information from multiple base clusterings, and then sparsified by means of K-nearest neighbors. By treating the sparsified co-association matrix as the similarity matrix, the corresponding graph is further decomposed into two parts, i.e., the corruption(noise) graph and the good(denoised) graph. In particular, the denoised graph as well as its spectral embedding can be learned simultaneously by solving a trace minimization problem. Finally, upon the learned spectral embedding, the k-means discretization is performed to obtain the final consensus clustering. Experiments on multiple real-world datasets have shown the superiority of the proposed algorithm.

Dual-granularity weighted ensemble clustering

Clustering ensemble technique has been shown to be effective in improving the accuracy and stability of single clustering algorithms. With the development of information technology, the amount of data, such as image, text and video, has increased rapidly. Efficiently clustering these large-scale datasets is a challenge. Clustering ensembles usually transform clustering results to a co-association matrix, and then to a graph-partition problem. These methods may suffer from information loss when computing the similarity among samples or base clusterings. Rich information between samples and base clusterings is ignored. Moreover, the results are not discrete. They need post-processing steps to obtain the final clustering result, which will deviate greatly from the real clustering result. To address this problem, we propose a co-clustering ensemble based on bilateral k-means (CEBKM) algorithm. Our algorithm can simultaneously cluster samples and base clusterings of a dataset, to fully exploit the potential information between the samples and the base clusterings. In addition, it can directly obtain the final clustering results without using other clustering algorithms. The proposed method, outperformed several state-of-the-art clustering ensemble methods in experiments conducted on real-world and toy datasets.

Weighted adaptively ensemble clustering method based on fuzzy Co-association matrix

In an effort to achieve improved classifier accuracy, extensive research has been conducted in classifier ensembles. Very recently, cluster ensembles have emerged. It is well known that off-the-shelf clustering methods may discover different structures in a given set of data. This is because each clustering algorithm has its own bias resulting from the optimization of different criteria. Furthermore, there is no ground truth against which the clustering result can be validated. Thus, no cross-validation technique can be carried out to tune input parameters involved in the clustering process. As a consequence, the user is not equipped with any guidelines for choosing the proper clustering method for a given dataset. Cluster ensembles offer a solution to challenges inherent to clustering arising from its ill-posed nature. Cluster ensembles can provide more robust and stable solutions by leveraging the consensus across multiple clustering results, while averaging out emergent spurious structures that arise due to the various biases to which each participating algorithm is tuned. In this chapter, we discuss the problem of combining multiple weighted clusters, discovered by a locally adaptive algorithm (Domeniconi, Papadopoulos, Gunopulos, &amp; Ma, 2004) which detects clusters in different subspaces of the input space. We believe that our approach is the first attempt to design a cluster ensemble for subspace clustering (Al-Razgan &amp; Domeniconi, 2006). Recently, several subspace clustering methods have been proposed (Parsons, Haque, &amp; Liu, 2004). They all attempt to dodge the curse of dimensionality which affects any algorithm in high dimensional spaces. In high dimensional spaces, it is highly likely that, for any given pair of points within the same cluster, there exist at least a few dimensions on which the points are far apart from each other. As a consequence, distance functions that equally use all input features may not be effective. Furthermore, several clusters may exist in different subspaces comprised of different combinations of features. In many real-world problems, some points are correlated with respect to a given set of dimensions, while others are correlated with respect to different dimensions. Each dimension could be relevant to at least one of the clusters. Global dimensionality reduction techniques are unable to capture local correlations of data. Thus, a proper feature selection procedure should operate locally in input space. Local feature selection allows one to embed different distance measures in different regions of the input space; such distance metrics reflect local correlations of data. In (Domeniconi, Papadopoulos, Gunopulos, &amp; Ma, 2004) we proposed a soft feature selection procedure (called LAC) that assigns weights to features according to the local correlations of data along each dimension. Dimensions along which data are loosely correlated receive a small weight, which has the effect of elongating distances along that dimension. Features along which data are strongly correlated receive a large weight, which has the effect of constricting distances along that dimension. Thus the learned weights perform a directional local reshaping of distances which allows a better separation of clusters, and therefore the discovery of different patterns in different subspaces of the original input space.

Recent studies show that ensemble methods enhance the stability and robustness of unsupervised learning. These approaches are successfully utilized to construct multiple clustering and combine them into a one representative consensus clustering of an improved quality. The quality of the consensus clustering is directly depended on fusion functions used in combination. In this article, the hierarchical clustering ensemble techniques are extended by introducing a new evolutionary fusion function. In the proposed method, multiple hierarchical clustering methods are generated via bagging. Thereafter, the consensus clustering is obtained using the search capability of genetic algorithm among different aggregated clustering methods made by different fusion functions. Putting some popular data sets to empirical study, the quality of the proposed method is compared with regular clustering ensembles. Experimental results demonstrate the accuracy improvement of the aggregated clustering results.

Spectral ensemble clustering with doubly stochastic co-association matrix