A survey on deep clustering: from the prior perspective

Recently, deep clustering methods have achieved remarkable results compared to traditional clustering approaches. However, its performance remains constrained by the absence of annotations. A thought-provoking observation is that there is still a significant gap between deep clustering and semi-supervised classification methods. Even with only a few labeled samples, the accuracy of semi-supervised learning is much higher than that of clustering. Given that we can annotate a small number of samples in a certain unsupervised way, the clustering task can be naturally transformed into a semi-supervised setting, thereby achieving comparable performance. Based on this intuition, we propose ClusMatch, a unified positive and negative pseudo-label learning based semi-supervised learning framework, which is pluggable and can be applied to existing deep clustering methods. Specifically, we first leverage the pre-trained deep clustering network to compute predictions for all samples, and then design specialized selection strategies to pick out a few high-quality samples as labeled samples for supervised learning. For the unselected samples, the novel unified positive and negative pseudo-label learning is introduced to provide additional supervised signals for semi-supervised fine-tuning. We also propose an adaptive positive-negative threshold learning strategy to further enhance the confidence of generated pseudo-labels. Extensive experiments on six widely-used datasets and one large-scale dataset demonstrate the superiority of our proposed ClusMatch. For example, ClusMatch achieves a significant accuracy improvement of 5.4% over the state-of-the-art method ProPos on an average of these six datasets.

ObjectiveCluster analysis of spatio-temporal event-related potential (ERP) data is a promising tool for exploring the measurement time window of ERPs. However, even after preprocessing, the remaining noise can result in uncertain cluster maps followed by unreliable time windows while clustering via conventional clustering methods. MethodsWe designed an ensemble deep clustering pipeline to determine a reliable time window for the ERP of interest from temporal concatenated grand average ERP data. The proposed pipeline includes semi-supervised deep clustering methods initialized by consensus clustering and unsupervised deep clustering methods with end-to-end architectures. Ensemble clustering from those deep clusterings was used by the designed adaptive time window determination to estimate the time window. ResultsAfter applying simulated and real ERP data, our method successfully obtained the time window for identifying the P3 components (as the interest of both ERP studies) while additional noise (e.g., adding 20 dB to −5 dB white Gaussian noise) was added to the prepared data. ConclusionCompared to the state-of-the-art clustering methods, a superior clustering performance was yielded from both ERP data. Furthermore, more stable and precise time windows were elicited as the noise increased. SignificanceOur study provides a complementary understanding of identifying the cognitive process using deep clustering analysis to the existing studies. Our finding suggests that deep clustering can be used to identify the ERP of interest when the data is imperfect after preprocessing.

Constrained clustering that integrates knowledge in the form of constraints in a clustering process has been studied for more than two decades. Popular clustering algorithms such as K-means, spectral clustering and recent deep clustering already have their constrained versions, but they usually lack of expressiveness in the form of constraints. In this paper we consider prior knowledge expressing relations between some data points and their assignments to clusters in propositional logic and we show how a deep clustering framework can be extended to integrate this knowledge. To achieve this, we define an expert loss based on the weighted models of the logical formulas; the weights depend on the soft assignment of points to clusters dynamically computed by the deep learner. This loss is integrated in the deep clustering method. We show how it can be computed efficiently using Weighted Model Counting and decomposition techniques. This method has the advantages of both integrating general knowledge and being independent of the neural architecture. Indeed, we have integrated the expert loss into two well-known deep clustering algorithms (IDEC and SCAN). Experiments have been conducted to compare our systems IDEC-LK and SCAN-LK to state-of-the-art methods for pairwise and triplet constraints in terms of computational cost, clustering quality and constraint satisfaction. We show that IDEC-LK can achieve comparable results with these systems, which are tailored for these specific constraints. To show the flexibility of our approach to learn from high-level domain constraints, we have integrated implication constraints, and a new constraint, called span-limited constraint that limits the number of clusters a set of points can belong to. Some experiments are also performed showing that constraints on some points can be extrapolated to other similar points.

Deep time-series clustering via latent representation alignment

In order to address the problem that existing methods combining spectral data and machine learning for wood species identification rely on labeled samples, this study introduces unsupervised learning into the field of wood identification. It proposes a novel wood-identification model called DCVAE (deep conditional variational autoencoder)-PLCAE (pseudo-label convolutional autoencoders). Terahertz time-domain spectra of wood samples at breast height of five broadleaf and five coniferous species were obtained (40 samples of each species of wood, 400 in total). A conditional variational autoencoder was applied to augment the terahertz spectroscopy dataset. Subsequently, a pseudo-label-guided deep clustering model was developed to extract more discriminative deep features. The model was compared with three traditional clustering algorithms and four deep clustering methods. Clustering experiments and visualization results show that the comprehensive clustering performance of DCVAE-PLCAE is better than the other comparative algorithms and that the extracted low-dimensional features has a more straightforward structure. The algorithm in this study can solve the problems of fewer labeled samples and the difficulty of extracting discriminative features by traditional clustering algorithms.

Semi-supervised clustering can be viewed as a clustering paradigm that exploits both labeled and unlabeled data to steer learning accurate data clusters and avoid local minimum solutions. Nonetheless, the attempts to refine existing semi-supervised clustering methods are relatively limited when compared to the advancements witnessed in the current benchmark methods in fully unsupervised clustering. This research introduces a novel semi-supervised method for deep clustering that leverages deep neural networks and fuzzy memberships to better capture the data partitions. In particular, the proposed Dual-Constraint-based Semi-Supervised Deep Clustering (DC-SSDEC) method utilizes two sets of pairwise soft constraints; "should-link" and "shouldNot-link", to guide the clustering process. The intended clustering task is expressed as an optimization of a newly designed objective function. Additionally, DC-SSDEC performance was evaluated through comprehensive experiments using three real-world and benchmark datasets. Moreover, a comparison with related state-of-the-art clustering techniques was conducted to showcase the DC-SSDEC outperformance. In particular, DC-SSDEC significance consists of the proposed dual-constraint formulation and its integration into a novel objective function. This contribution yielded an improvement in the resulting clustering performance compared to relevant state-of-the-art approaches. In addition, the assessment of the proposed model using real-world datasets represents another contribution of this research. In fact, increases of 3.25%, 1.44%, and 1.82% in the clustering accuracy were gained by DC-SSDEC over the best performing single-constraint-based approach, using MNIST, STL-10, and USPS datasets, respectively.

Extracting and organizing knowledge from the agricultural crop disease research literature are challenging tasks because of the heterogeneous terminologies, complicated symptom descriptions, and unstructured nature of scientific documents. In this study, we developed a multi-stage natural language processing (NLP) pipeline to automate knowledge extraction, organization, and integration from the agricultural research literature into a domain-consistent crop disease knowledge graph. The model combines transformer-based sentence embeddings with variational deep clustering to extract topics, which are further refined via facet-aware relevance scoring for sentence selection to be included in the summary. Lexicon-guided named entity recognition helps in the precise identification and normalization of terms for crops, diseases, symptoms, etc. Relation extraction based on a combination of lexical, semantic, and contextual features leads to the meaningful generation of triplets for the knowledge graph. The experimental results show that the method yielded consistently good results at each stage of the knowledge extraction process. Among the combinations of embedding and deep clustering methods, SciBERT + VaDE achieved the best clustering results. The extraction of representative sentences for disease symptoms, control/treatment, and prevention obtained high F1-scores of around 0.8. The resulting knowledge graph has high node coverage and high relation completeness, as well as high precision and recall in triplet generation. The multi-stage NLP pipeline effectively converts unstructured agricultural research texts into a coherent and semantically rich knowledge graph, providing a basis for further research in crop disease analysis, knowledge retrieval, and data-driven decision support in agricultural informatics.

Variable screening is a powerful and efficient tool for dimension reduction under ultrahigh dimensional settings. However, most existing methods overlook useful prior knowledge in specific applications. In this work, from a Bayesian modeling perspective, we develop a unified variable screening procedure for the linear regression model. We discuss different constructions of posterior mean screening (PMS) statistics to incorporate different types of prior knowledge according to specific applications. With non-informative prior specifications, PMS is equivalent to high-dimensional ordinary least-square projections (HOLP). We establish the screening consistency property for PMS with different types of prior knowledge. We show that PMS is robust to prior misspecifications; and when the prior knowledge provides correct information on summarizing the true parameter settings, PMS can substantially improve the selection accuracy compared to HOLP and other existing methods. We illustrate our method with extensive simulation studies and an analysis of neuroimaging data.

Driven by the explosive growth in available data and decreasing costs of computation, Deep Learning (DL) has found much of its fame in problems involving classification tasks which are considered supervised learning. Deep learning has also been widely used to learn richer and better data representations from big data, without relying too much on human engineered features. Even though it started mostly within the realm of supervised learning, deep learning’s success has recently inspired several deep learning-based developments in clustering algorithms which sit squarely within unsupervised learning. Most DL-based clustering approaches result in both deep representations and (either as an explicit aim or as a byproduct) clustering outputs, hence we refer to all these approaches as Deep Clustering. In this chapter, we present a simplified taxonomy of Deep Clustering methods, based mainly on the overall procedural structure or design which helps beginning readers quickly grasp how almost all approaches are designed, while allowing more advanced readers to learn how to design increasingly sophisticated deep clustering pipelines that fit their own machine learning problem-solving aims. Like DL, Deep Clustering promises to leave an impact on diverse application domains ranging from computer vision and speech recognition to recommender systems and natural language processing.

The performances of semisupervised clustering for unlabeled data are often superior to those of unsupervised learning, which indicates that semantic information attached to clusters can significantly improve feature representation capability. In a graph convolutional network (GCN), each node contains information about itself and its neighbors that is beneficial to common and unique features among samples. Combining these findings, we propose a deep clustering method based on GCN and semantic feature guidance (GFDC) in which a deep convolutional network is used as a feature generator, and a GCN with a softmax layer performs clustering assignment. First, the diversity and amount of input information are enhanced to generate highly useful representations for downstream tasks. Subsequently, the topological graph is constructed to express the spatial relationship of features. For a pair of datasets, feature correspondence constraints are used to regularize clustering loss, and clustering outputs are iteratively optimized. Three external evaluation indicators, i.e., clustering accuracy, normalized mutual information, and the adjusted Rand index, and an internal indicator, i.e., the Davidson-Bouldin index (DBI), are employed to evaluate clustering performances. Experimental results on eight public datasets show that the GFDC algorithm is significantly better than the majority of competitive clustering methods, i.e., its clustering accuracy is 20% higher than the best clustering method on the United States Postal Service dataset. The GFDC algorithm also has the highest accuracy on the smaller Amazon and Caltech datasets. Moreover, DBI indicates the dispersion of cluster distribution and compactness within the cluster.

Clustering is one of the most fundamental unsupervised learning tasks with numerous applications in various fields. Clustering methods based on neural networks, called deep clustering methods, leverage the representational power of neural networks to enhance clustering performance. ClusterGan constitutes a generative deep clustering method that exploits generative adversarial networks (GANs) to perform clustering. However, it inherits some deficiencies of GANs, such as mode collapse, vanishing gradients and training instability. In order to tackle those deficiencies, the generative approach of implicit maximum likelihood estimation (IMLE) has been recently proposed. In this paper, we present a clustering method based on generative neural networks, called neural implicit maximum likelihood clustering, which adopts ideas from both ClusterGAN and IMLE. The proposed method has been compared with ClusterGAN and other neural clustering methods on both synthetic and real datasets, demonstrating promising results.

Subspace-based parameter estimation of exponentially damped sinusoids using prior knowledge of frequency and phase

As discussed in the research of Mohamed et al. (2013), learning and studying via a different context, including the inter-cultural knowledge of a teacher, may present a challenge when trying to account for students’ prior knowledge. Enhancing their study, this paper addresses the question of whether or not an internal session comprising open discussion and brainstorming followed by asynchronous commenting and ending with self-reflection might help to assess students’ prior knowledge. Data was collected in the context of a series of academic teachings partly supported by the METIS project (The METIS Project is funded by the national German Federal Ministry of Science and Research, cp.: http://www.ihi.zittau.de/cms/de/621/Forschung.) in two separate phases. Graduate students on master level attended the phase reported here in order to study research methods as a joint activity of Technische Universitat Dresden, and Internationales Hochschulinstitut Zittau, both Germany. For the experimental group, the study found that there are no significant differences between the control and treatment group in terms of cognitive achievement and students’ attitudes. From the teacher’s perspective, however, this course is learning atmosphere challenged students’ reactions. Design is suggested to be enhanced to allow a closer look on different types of prior knowledge in the field of research training.

By cooperatively learning features and assigning clusters, deep clustering is superior to conventional clustering algorithms. Numerous deep clustering algorithms have been developed for a variety of application levels; however, the majority are still incapable of learning robust noise-resistant latent features, which limits the clustering performance. To address this open research challenge, we introduce, for the first time, a new approach called: Robust Deep Embedded Image Clustering algorithm with Separable Krawtchouk and Hahn Moments (RDEICSKHM). Our approach leverages the advantages of Krawtchouk and Hahn moments, such as local feature extraction, discrete orthogonality, and noise tolerance, to obtain a meaningful and robust image representation. Moreover, we employ LayerNormalization to further improve the latent space quality and facilitate the clustering process. We evaluate our approach on four image datasets: MNIST, MNIST-test, USPS, and Fashion-MNIST. We compare our method with several deep clustering methods based on two metrics: clustering accuracy (ACC) and normalized mutual information (NMI). The experimental results show that our method achieves superior or competitive performance on all datasets, demonstrating its effectiveness and robustness for deep image clustering.

Deep clustering is a fundamental task in machine learning and data mining that aims at learning clustering-oriented feature representations. In previous studies, most of deep clustering methods follow the idea of self-supervised representation learning by maximizing the consistency of all similar instance pairs while ignoring the effect of feature redundancy on clustering performance. In this paper, to address the above issue, we design a dual mutual information constrained clustering method named DMICC which is based on deep contrastive clustering architecture, in which the dual mutual information constraints are particularly employed with solid theoretical guarantees and experimental validations. Specifically, at the feature level, we reduce the redundancy among features by minimizing the mutual information across all the dimensionalities to encourage the neural network to extract more discriminative features. At the instance level, we maximize the mutual information of the similar instance pairs to obtain more unbiased and robust representations. The dual mutual information constraints happen simultaneously and thus complement each other to jointly optimize better features that are suitable for the clustering task. We also prove that our adopted mutual information constraints are superior in feature extraction, and the proposed dual mutual information constraints are clearly bounded and thus solvable. Extensive experiments on five benchmark datasets show that our proposed approach outperforms most other clustering algorithms. The code is available at https://github.com/Li-Hyn/DMICC.