ClusMatch: Improving Deep Clustering by Unified Positive and Negative Pseudo-Label Learning.

ObjectiveCluster analysis of spatio-temporal event-related potential (ERP) data is a promising tool for exploring the measurement time window of ERPs. However, even after preprocessing, the remaining noise can result in uncertain cluster maps followed by unreliable time windows while clustering via conventional clustering methods. MethodsWe designed an ensemble deep clustering pipeline to determine a reliable time window for the ERP of interest from temporal concatenated grand average ERP data. The proposed pipeline includes semi-supervised deep clustering methods initialized by consensus clustering and unsupervised deep clustering methods with end-to-end architectures. Ensemble clustering from those deep clusterings was used by the designed adaptive time window determination to estimate the time window. ResultsAfter applying simulated and real ERP data, our method successfully obtained the time window for identifying the P3 components (as the interest of both ERP studies) while additional noise (e.g., adding 20 dB to −5 dB white Gaussian noise) was added to the prepared data. ConclusionCompared to the state-of-the-art clustering methods, a superior clustering performance was yielded from both ERP data. Furthermore, more stable and precise time windows were elicited as the noise increased. SignificanceOur study provides a complementary understanding of identifying the cognitive process using deep clustering analysis to the existing studies. Our finding suggests that deep clustering can be used to identify the ERP of interest when the data is imperfect after preprocessing.

Facilitated by the powerful feature extraction ability of neural networks, deep clustering has achieved great success in analyzing high-dimensional and complex real-world data. The performance of deep clustering methods is affected by various factors such as network structures and learning objectives. However, as pointed out in this survey, the essence of deep clustering lies in the incorporation and utilization of prior knowledge, which is largely ignored by existing works. From pioneering deep clustering methods based on data structure assumptions to recent contrastive clustering methods based on data augmentation invariances, the development of deep clustering intrinsically corresponds to the evolution of prior knowledge. In this survey, we provide a comprehensive review of deep clustering methods by categorizing them into six types of prior knowledge. We find that in general the prior innovation follows two trends, namely, i) from mining to constructing, and ii) from internal to external. Besides, we provide a benchmark on five widely-used datasets and analyze the performance of methods with diverse priors. By providing a novel prior knowledge perspective, we hope this survey could provide some novel insights and inspire future research in the deep clustering community.

Deep time-series clustering via latent representation alignment

Constrained clustering that integrates knowledge in the form of constraints in a clustering process has been studied for more than two decades. Popular clustering algorithms such as K-means, spectral clustering and recent deep clustering already have their constrained versions, but they usually lack of expressiveness in the form of constraints. In this paper we consider prior knowledge expressing relations between some data points and their assignments to clusters in propositional logic and we show how a deep clustering framework can be extended to integrate this knowledge. To achieve this, we define an expert loss based on the weighted models of the logical formulas; the weights depend on the soft assignment of points to clusters dynamically computed by the deep learner. This loss is integrated in the deep clustering method. We show how it can be computed efficiently using Weighted Model Counting and decomposition techniques. This method has the advantages of both integrating general knowledge and being independent of the neural architecture. Indeed, we have integrated the expert loss into two well-known deep clustering algorithms (IDEC and SCAN). Experiments have been conducted to compare our systems IDEC-LK and SCAN-LK to state-of-the-art methods for pairwise and triplet constraints in terms of computational cost, clustering quality and constraint satisfaction. We show that IDEC-LK can achieve comparable results with these systems, which are tailored for these specific constraints. To show the flexibility of our approach to learn from high-level domain constraints, we have integrated implication constraints, and a new constraint, called span-limited constraint that limits the number of clusters a set of points can belong to. Some experiments are also performed showing that constraints on some points can be extrapolated to other similar points.

In order to address the problem that existing methods combining spectral data and machine learning for wood species identification rely on labeled samples, this study introduces unsupervised learning into the field of wood identification. It proposes a novel wood-identification model called DCVAE (deep conditional variational autoencoder)-PLCAE (pseudo-label convolutional autoencoders). Terahertz time-domain spectra of wood samples at breast height of five broadleaf and five coniferous species were obtained (40 samples of each species of wood, 400 in total). A conditional variational autoencoder was applied to augment the terahertz spectroscopy dataset. Subsequently, a pseudo-label-guided deep clustering model was developed to extract more discriminative deep features. The model was compared with three traditional clustering algorithms and four deep clustering methods. Clustering experiments and visualization results show that the comprehensive clustering performance of DCVAE-PLCAE is better than the other comparative algorithms and that the extracted low-dimensional features has a more straightforward structure. The algorithm in this study can solve the problems of fewer labeled samples and the difficulty of extracting discriminative features by traditional clustering algorithms.

Semi-supervised clustering can be viewed as a clustering paradigm that exploits both labeled and unlabeled data to steer learning accurate data clusters and avoid local minimum solutions. Nonetheless, the attempts to refine existing semi-supervised clustering methods are relatively limited when compared to the advancements witnessed in the current benchmark methods in fully unsupervised clustering. This research introduces a novel semi-supervised method for deep clustering that leverages deep neural networks and fuzzy memberships to better capture the data partitions. In particular, the proposed Dual-Constraint-based Semi-Supervised Deep Clustering (DC-SSDEC) method utilizes two sets of pairwise soft constraints; "should-link" and "shouldNot-link", to guide the clustering process. The intended clustering task is expressed as an optimization of a newly designed objective function. Additionally, DC-SSDEC performance was evaluated through comprehensive experiments using three real-world and benchmark datasets. Moreover, a comparison with related state-of-the-art clustering techniques was conducted to showcase the DC-SSDEC outperformance. In particular, DC-SSDEC significance consists of the proposed dual-constraint formulation and its integration into a novel objective function. This contribution yielded an improvement in the resulting clustering performance compared to relevant state-of-the-art approaches. In addition, the assessment of the proposed model using real-world datasets represents another contribution of this research. In fact, increases of 3.25%, 1.44%, and 1.82% in the clustering accuracy were gained by DC-SSDEC over the best performing single-constraint-based approach, using MNIST, STL-10, and USPS datasets, respectively.

Extracting and organizing knowledge from the agricultural crop disease research literature are challenging tasks because of the heterogeneous terminologies, complicated symptom descriptions, and unstructured nature of scientific documents. In this study, we developed a multi-stage natural language processing (NLP) pipeline to automate knowledge extraction, organization, and integration from the agricultural research literature into a domain-consistent crop disease knowledge graph. The model combines transformer-based sentence embeddings with variational deep clustering to extract topics, which are further refined via facet-aware relevance scoring for sentence selection to be included in the summary. Lexicon-guided named entity recognition helps in the precise identification and normalization of terms for crops, diseases, symptoms, etc. Relation extraction based on a combination of lexical, semantic, and contextual features leads to the meaningful generation of triplets for the knowledge graph. The experimental results show that the method yielded consistently good results at each stage of the knowledge extraction process. Among the combinations of embedding and deep clustering methods, SciBERT + VaDE achieved the best clustering results. The extraction of representative sentences for disease symptoms, control/treatment, and prevention obtained high F1-scores of around 0.8. The resulting knowledge graph has high node coverage and high relation completeness, as well as high precision and recall in triplet generation. The multi-stage NLP pipeline effectively converts unstructured agricultural research texts into a coherent and semantically rich knowledge graph, providing a basis for further research in crop disease analysis, knowledge retrieval, and data-driven decision support in agricultural informatics.

Clustering is one of the most fundamental unsupervised learning tasks with numerous applications in various fields. Clustering methods based on neural networks, called deep clustering methods, leverage the representational power of neural networks to enhance clustering performance. ClusterGan constitutes a generative deep clustering method that exploits generative adversarial networks (GANs) to perform clustering. However, it inherits some deficiencies of GANs, such as mode collapse, vanishing gradients and training instability. In order to tackle those deficiencies, the generative approach of implicit maximum likelihood estimation (IMLE) has been recently proposed. In this paper, we present a clustering method based on generative neural networks, called neural implicit maximum likelihood clustering, which adopts ideas from both ClusterGAN and IMLE. The proposed method has been compared with ClusterGAN and other neural clustering methods on both synthetic and real datasets, demonstrating promising results.

Trajectory clustering is of great significance for identifying behavioral patterns and vessel types of non-cooperative ships. However, existing trajectory clustering methods suffer from limitations in extracting cross-spatiotemporal scale features and modeling the coupling relationship between positional and motion features, which restricts clustering performance. To address this, this study proposes a deep ship trajectory clustering method based on feature embedding representation learning (ERL-DTC). The method designs a Temporal Attention-based Multi-scale feature Aggregation Network (TA-MAN) to achieve dynamic fusion of trajectory features from micro to macro scales. A Dual-feature Self-attention Fusion Encoder (DualSFE) is employed to decouple and jointly represent the spatiotemporal position and motion features of trajectories. A two-stage optimization strategy of “pre-training and joint training” is adopted, combining contrastive loss and clustering loss to jointly constrain the embedding representation learning, ensuring it preserves trajectory similarity relationships while being adapted to the clustering task. Experiments on a public vessel trajectory dataset show that for a four-class task (K = 4), ERL-DTC improves ACC by approximately 14.1% compared to the current best deep clustering method, with NMI and ARI increasing by about 28.9% and 30.2%, respectively. It achieves the highest Silhouette Coefficient (SC) and the lowest Davies-Bouldin Index (DBI), indicating a tighter and more clearly separated cluster structure. Furthermore, its inference efficiency is improved by two orders of magnitude compared to traditional point-matching-based methods, without significantly increasing runtime due to model complexity. Ablation studies and parameter sensitivity analysis further validate the necessity of each module design and the rationality of hyperparameter settings. This research provides an efficient and robust solution for feature learning and clustering of vessel trajectories across spatiotemporal scales.

Patient-Specific Quality Assurance in Radiotherapy (PSQA) demands high-resolution dosimetry to verify accurate dose delivery in personalized intensity-modulated radiotherapy (IMRT) and volumetric-modulated arc therapy (VMAT) treatments. A novel optically stimulated luminescence (OSL) film dosimeter made with BaFBr:Eu2+ phosphor, offers submm spatial resolution. However, its energy-dependent response, requires corrections. Previously, a correction was proposed for a class of prostate cancer treatments assuming similar OSL energy response within the class. This study explored other class-specific corrections using a comprehensive radiotherapy treatment dataset. New classes were formed based on the similarity of treatment parameters without the need for user-based classifications. The dataset comprised 101 IMRT and VMAT treatment plans for three different Varian linac types (2 × Halcyon, 2 × TrueBeam, and 1 × TrueBeam STx). The treatment classes are based on a K-means clustering algorithm, that utilizes twelve quantitative treatment parameters expressed in principal components. Within cluster sum square (WCSS) was used to find the optimal number of classes and prevent data-overfitting. This objective assignment to classes was compared with three independent manual classifications by experienced medical physicists and dosimetrist. Additionally, a random class assignment was conducted for comparison. The adjusted-random-index (ARI) measured the similarity between classification methods. The OSL film, produced by Agfa N.V., was calibrated using a 6 MV TrueBeam linac. It was then used to measure treatments in an MULTICube phantom (IBA). Readout was performed in a CR-15 scanner. The local dose difference distribution between the measurement and treatment was characterized using a rational function. Class-specific corrections were developed by averaging the parameters of the rational function for each class as determined by the clustering, manual, and random classification methods. Dosimetric performances were evaluated within 20% and 50% isodose lines (D20% and D50%) before and after correction. The clustering method identified eight clusters (WCSS=119 silhouette=0.6) when representing data in three principal components, that is, 75% of the data variance. No significant similarity was found between clustering results and manual classification methods (ARI<0.01). Manual classifications are subject to interoperator variability. In fact, we found moderate similarity between classes and variations in the number of classes, ranging from 9 to 16. Uncorrected global dose difference (%) had mean value 0.9%±4.1% within D20%, with 47 and 34 treatments resulting in dose difference below 3% within D20% and D50%, respectively. After class-specific correction, the clustering method had mean dose differences (%) -0.2%±2.0%. The removal of the skewness in the corrected pixel-to-pixel dose difference distribution indicated an effective reduction of the OSL over-response. 88 and 74 treatments had corrected mean dose difference below 3% within D20% and D50%, respectively. Similar average dosimetric improvements were found only for the 16 manual class-solution, which however still showed a moderate skewness (0.1) after correction. Both, automated and manual class assignment preform better than the random assignment. Eight treatment class-solutions corrected the energy-dependent response of an OSL film used for PSQA measurements. Clustering classification methods, based on quantitative treatment information, yielded better dosimetric results compared to qualitative classification techniques.

Driven by the explosive growth in available data and decreasing costs of computation, Deep Learning (DL) has found much of its fame in problems involving classification tasks which are considered supervised learning. Deep learning has also been widely used to learn richer and better data representations from big data, without relying too much on human engineered features. Even though it started mostly within the realm of supervised learning, deep learning’s success has recently inspired several deep learning-based developments in clustering algorithms which sit squarely within unsupervised learning. Most DL-based clustering approaches result in both deep representations and (either as an explicit aim or as a byproduct) clustering outputs, hence we refer to all these approaches as Deep Clustering. In this chapter, we present a simplified taxonomy of Deep Clustering methods, based mainly on the overall procedural structure or design which helps beginning readers quickly grasp how almost all approaches are designed, while allowing more advanced readers to learn how to design increasingly sophisticated deep clustering pipelines that fit their own machine learning problem-solving aims. Like DL, Deep Clustering promises to leave an impact on diverse application domains ranging from computer vision and speech recognition to recommender systems and natural language processing.

Deep clustering is a fundamental task in machine learning and data mining that aims at learning clustering-oriented feature representations. In previous studies, most of deep clustering methods follow the idea of self-supervised representation learning by maximizing the consistency of all similar instance pairs while ignoring the effect of feature redundancy on clustering performance. In this paper, to address the above issue, we design a dual mutual information constrained clustering method named DMICC which is based on deep contrastive clustering architecture, in which the dual mutual information constraints are particularly employed with solid theoretical guarantees and experimental validations. Specifically, at the feature level, we reduce the redundancy among features by minimizing the mutual information across all the dimensionalities to encourage the neural network to extract more discriminative features. At the instance level, we maximize the mutual information of the similar instance pairs to obtain more unbiased and robust representations. The dual mutual information constraints happen simultaneously and thus complement each other to jointly optimize better features that are suitable for the clustering task. We also prove that our adopted mutual information constraints are superior in feature extraction, and the proposed dual mutual information constraints are clearly bounded and thus solvable. Extensive experiments on five benchmark datasets show that our proposed approach outperforms most other clustering algorithms. The code is available at https://github.com/Li-Hyn/DMICC.

The performances of semisupervised clustering for unlabeled data are often superior to those of unsupervised learning, which indicates that semantic information attached to clusters can significantly improve feature representation capability. In a graph convolutional network (GCN), each node contains information about itself and its neighbors that is beneficial to common and unique features among samples. Combining these findings, we propose a deep clustering method based on GCN and semantic feature guidance (GFDC) in which a deep convolutional network is used as a feature generator, and a GCN with a softmax layer performs clustering assignment. First, the diversity and amount of input information are enhanced to generate highly useful representations for downstream tasks. Subsequently, the topological graph is constructed to express the spatial relationship of features. For a pair of datasets, feature correspondence constraints are used to regularize clustering loss, and clustering outputs are iteratively optimized. Three external evaluation indicators, i.e., clustering accuracy, normalized mutual information, and the adjusted Rand index, and an internal indicator, i.e., the Davidson-Bouldin index (DBI), are employed to evaluate clustering performances. Experimental results on eight public datasets show that the GFDC algorithm is significantly better than the majority of competitive clustering methods, i.e., its clustering accuracy is 20% higher than the best clustering method on the United States Postal Service dataset. The GFDC algorithm also has the highest accuracy on the smaller Amazon and Caltech datasets. Moreover, DBI indicates the dispersion of cluster distribution and compactness within the cluster.

By cooperatively learning features and assigning clusters, deep clustering is superior to conventional clustering algorithms. Numerous deep clustering algorithms have been developed for a variety of application levels; however, the majority are still incapable of learning robust noise-resistant latent features, which limits the clustering performance. To address this open research challenge, we introduce, for the first time, a new approach called: Robust Deep Embedded Image Clustering algorithm with Separable Krawtchouk and Hahn Moments (RDEICSKHM). Our approach leverages the advantages of Krawtchouk and Hahn moments, such as local feature extraction, discrete orthogonality, and noise tolerance, to obtain a meaningful and robust image representation. Moreover, we employ LayerNormalization to further improve the latent space quality and facilitate the clustering process. We evaluate our approach on four image datasets: MNIST, MNIST-test, USPS, and Fashion-MNIST. We compare our method with several deep clustering methods based on two metrics: clustering accuracy (ACC) and normalized mutual information (NMI). The experimental results show that our method achieves superior or competitive performance on all datasets, demonstrating its effectiveness and robustness for deep image clustering.

Deep clustering incorporates embedding into clustering in order to find a lower-dimensional space suitable for clustering tasks. Conventional deep clustering methods aim to obtain a single global embedding subspace (aka latent space) for all the data clusters. In contrast, in this article, we propose a deep multirepresentation learning (DML) framework for data clustering whereby each difficult-to-cluster data group is associated with its own distinct optimized latent space and all the easy-to-cluster data groups are associated with a general common latent space. Autoencoders (AEs) are employed for generating cluster-specific and general latent spaces. To specialize each AE in its associated data cluster(s), we propose a novel and effective loss function which consists of weighted reconstruction and clustering losses of the data points, where higher weights are assigned to the samples more probable to belong to the corresponding cluster(s). Experimental results on benchmark datasets demonstrate that the proposed DML framework and loss function outperform state-of-the-art clustering approaches. In addition, the results show that the DML method significantly outperforms the SOTA on imbalanced datasets as a result of assigning an individual latent space to the difficult clusters.