An Efficient Clustering Method for High-Dimensional Data Mining

For the extraction of spatially important regions from mass spectrometry imaging (MSI) data, different clustering methods have been proposed. These clustering methods are based on certain assumptions and use different criteria to assign pixels into different classes. For high-dimensional MSI data, the curse of dimensionality also limits the performance of clustering methods which are usually overcome by pre-processing the data using dimension reduction techniques. In summary, the extraction of spatial patterns from MSI data can be done using different unsupervised methods, but the robust evaluation of clustering results is what is still missing. In this study, we have performed multiple simulations on synthetic and real MSI data to validate the performance of unsupervised methods. The synthetic data were simulated mimicking important spatial and statistical properties of real MSI data. Our simulation results confirmed that K-means clustering with correlation distance and Gaussian Mixture Modeling clustering methods give optimal performance in most of the scenarios. The clustering methods give efficient results together with dimension reduction techniques. From all the dimension techniques considered here, the best results were obtained with the minimum noise fraction (MNF) transform. The results were confirmed on both synthetic and real MSI data. However, for successful implementation of MNF transform the MSI data requires to be of limited dimensions.

The technique of machinery condition monitoring has been greatly enhanced over recent years with the application of many effective classifiers. However, these classification methods suffer from the 'curse of dimensionality' when applied to high-dimensional condition monitoring data. Actually, many classification algorithms are simply intractable when the number of features in the data is sufficiently large. In order to solve the problem, engineers have to resort to complicated feature extraction methods and other statistical theories to reduce the data dimensionality. However, features extracted using these methods lose their original engineering meaning and become obscure for engineers. In this study, a novel feature selection algorithm is presented to help to identify machinery condition quickly, based only on frequency spectrum data and without considering any complicated feature extraction methods. This brings many significant benefits: it can not only help engineers out of difficulties with complicated feature extraction methods, but it also gives a clear insight into the real condition of machinery without any loss of engineering meaning from the features used. The algorithm forms a functional extension of a relevance vector machine to feature selection, with a fast step-variable sequential backward search algorithm to search relevant scale parameters within a kernel function. Two case studies on different levels of condition monitoring are conducted to demonstrate the potential of applying the algorithm to high-dimensional engineering data.

High dimensional data clustering is a difficult task due to the curse of dimensionality. Traditional clustering methods usually fail to produce meaningful results for high dimensional data. Hypergraph partition is believed to be a promising method for dealing with this challenge. In this work, a new high dimensional clustering method called Merging Dense SubGraphs (MDSG) is proposed. A graph G is first constructed from the data by defining an adjacency relationship between the data points using Shared k Nearest Neighbors (SNN). Then a hypergraph is created from the graph G by defining the hyperedges to be all the maximal cliques in the graph. After the hypergraph is produced, an improved hypergraph partitioning method is used to produce the final clustering results. The proposed MDSG method is evaluated on several real high dimensional handwritten datasets, and the experimental results show that the proposed method is superior to the traditional clustering method and other hypergraph partition methods for high dimensional handwritten data clustering.

The vector approximation file (VA-file) approach is an efficient high-dimensional indexing method using compression technique to overcome the difficulty of 'curse of dimensionality'. The VA-file method combined with tree-based index structure can improve the querying efficiency, but it still succumbs to the 'curse of dimensionality'. In this paper, a new high-dimensional indexing structure called PCR-tree for non-uniform distributed data sets was presented, which employs R-tree to manage the approximate vectors in the reduced-dimensionality space. The approximate vectors can be built in the KL transform domain, and low dimensional MBRs (minimum bounding rectangles) can be used to manage the approximations on the first few principal components. When performing k-nearest neighbor search, a lower-bound filtering algorithm is used to reject the improper nodes of PCR-tree, which can reduce the computational complexity and I/O cost without any dismissals. The experiment results on large image databases show that the new approach provides a faster search speed than other tree-structured vector approximation approaches.

In modern database applications the similarity or dissimilarity of complex objects is examined by performing distance-based queries (DBQs)on data of high dimensionality. The R-tree and its variations are commonly cited multidimensional access methods that can be used for answering such queries. Although, the related algorithms work well for low-dimensional data spaces, their performance degrades as the number of dimensions increases (dimensionality curse). In order to obtain accept- able response time in high-dimensional data spaces, algorithms that ob- tain approximate solutions can be used. Three approximation techniques (α-allowance, N-consider and M-consider) and the respective recursive branch-and-bound algorithms for DBQs are presented and studied in this paper. We investigate the performance of these algorithms for the most representative DBQs (the K-nearest neighbors query and the K-closest pairs query) in high-dimensional data spaces, where the point data sets are indexed by tree-like structures belonging to the R-tree family:R*- trees and X-trees. The searching strategy is tuned according to several parameters, in order to examine the trade-off between cost (I/O activ- ity and response time) and accuracy of the result. The outcome of the experimental evaluation is the derivation of the outperforming DBQ ap- proximate algorithm for large high-dimensional point data sets.

The Evidential K-Nearest Neighbor (EK-NN) classification rule provides a global treatment of imperfect knowledge in class labels, but still suffers from the curse of dimensionality as well as runtime and memory restrictions when performing nearest neighbors search, in particular for large and high-dimensional data. To avoid the curse of dimensionality, this article first proposes a rough evidential K-NN (REK-NN) classification rule in the framework of rough set theory. Based on a reformulated K-NN rough set model, REK-NN selects features and thus reduces complexity by minimizing a proposed neighborhood pignistic decision error rate, which considers both Bayes decision error and spatial information among samples in feature space. In contrast to existing rough set-based feature selection methods, REK-NN is a synchronized rule rather than a stepwise one, in the sense that feature selection and learning are performed simultaneously. In order to further handle data with large sample size, we derive a distributed REK-NN method and implement it in the Apache Spark. The theoretical analysis of the classifier generalization error bound is finally presented. It is shown that the distributed REK-NN achieves good performances while drastically reducing the number of features and consuming less runtime and memory. Numerical experiments conducted on real-world datasets validate our conclusions.

A topological approach for cancer subtyping from gene expression data.

The incidence of bladder cancer is on the rise, and its molecular heterogeneity presents significant challenges for personalized cancer therapy. Transcriptome data can characterize the variability among patients. Traditional machine-learning methods often struggle with high-dimensional genomic data, falling into the ’curse of dimensionality’. To address this challenge, we have developed MVMSGAT, an innovative predictive model tailored for forecasting responses to neoadjuvant therapy in bladder cancer patients. MVMSGAT significantly enhances model performance by incorporating multi-perspective biological prior knowledge. It initially utilizes the Boruta algorithm to select key genes from transcriptome data, subsequently constructing a comprehensive graph of gene co-expression and protein–protein interactions. MVMSGAT further employs a graph convolutional neural network to integrate this information within a multiview knowledge graph, amalgamating biological knowledge maps from various scales using an attention mechanism. For validation, MVMSGAT was tested using a five-fold cross-validation approach on two specific GEO datasets, GSE169455 and GSE69795, involving a total of 210 bladder cancer samples. MVMSGAT demonstrated superior performance, with the following metrics (mean ± standard deviation): AUC-ROC of 0.8724±0.0511, accuracy of 0.7789±0.068, F1 score of 0.8529±0.0338, and recall of 0.9231±0.0719. These results underscore the potential of MVMSGAT in advancing personalized treatment and precision medicine in bladder cancer.

Advances in single-cell RNA sequencing (scRNA-seq) technologies allow researchers to analyze the genome-wide transcription profile and to solve biological problems at the individual-cell resolution. However, existing clustering methods on scRNA-seq suffer from high dropout rate and curse of dimensionality in the data. Here, we propose a novel pipeline, scBKAP, the cornerstone of which is a single-cell bisecting K-means clustering method based on an autoencoder network and a dimensionality reduction model MPDR. Specially, scBKAP utilizes an autoencoder network to reconstruct gene expression values from scRNA-seq data to alleviate the dropout issue, and the MPDR model composed of the M3Drop feature selection algorithm and the PHATE dimensionality reduction algorithm to reduce the dimensions of reconstructed data. The dimensionality-reduced data are then fed into the bisecting K-means clustering algorithm to identify the clusters of cells. Comprehensive experiments demonstrate scBKAP's superior performance over nine state-of-the-art single-cell clustering methods on 21 public scRNA-seq datasets and simulated datasets. The source codes and datasets are available at https://github.com/YuBinLab-QUST/scBKAP/ and https://doi.org/10.24433/CO.4592131.v1.

Images can be represented as vectors in a high-dimensional Image space with components specifying light intensities at image pixels. To avoid the ‘curse of dimensionality’, the original high-dimensional image data are transformed into their lower-dimensional features preserving certain subject-driven data properties. These properties can include ‘information-preserving’ when using the constructed low-dimensional features instead of original high-dimensional vectors, as well preserving the distances and angles between the original high-dimensional image vectors. Under the commonly used Manifold assumption that the high-dimensional image data lie on or near a certain unknown low-dimensional Image manifold embedded in an ambient high-dimensional ‘observation’ space, a constructing of the lower-dimensional features consists in constructing an Embedding mapping from the Image manifold to Feature space, which, in turn, determines a low-dimensional parameterization of the Image manifold. We propose a new geometrically motivated Embedding method which constructs a low-dimensional parameterization of the Image manifold and provides the information-preserving property as well as the locally isometric and conformal properties.

Clustering large data sets of high has always been a challenge for clustering algorithms. Many recently developed clustering algorithms have attempted to address either handling data sets with a very large number of records and/or with a very high number of dimensions. We provide a discussion of the advantages and limitations of existing algorithms when they operate on very large multidimensional data sets. To simultaneously overcome both the curse of dimensionality and the scalability problems associated with large amounts of data, we propose a new clustering algorithm called O-Cluster. O-Cluster combines a novel active sampling technique with an axis-parallel partitioning strategy to identify continuous areas of high density in the input space. The method operates on a limited memory buffer and requires at most a single scan through the data. We demonstrate the high quality of the obtained clustering solutions, their robustness to noise, and O-Cluster's excellent scalability.

We prove that the gradient descent training of a two-layer neural network on empirical or population risk may not decrease population risk at an order faster than $t^{-4/(d-2)}$ under mean field scaling. The loss functional is mean squared error with a Lipschitz-continuous target function and data distributed uniformly on the $d$ -dimensional unit cube. Thus gradient descent training for fitting reasonably smooth, but truly high-dimensional data may be subject to the curse of dimensionality. We present numerical evidence that gradient descent training with general Lipschitz target functions becomes slower and slower as the dimension increases, but converges at approximately the same rate in all dimensions when the target function lies in the natural function space for two-layer ReLU networks. Impact Statement –Artificial neural networks perform well in many real life applications, but may suffer from the curse of dimensionality on certain problems. We provide theoretical and numerical evidence that this may be related to whether a target function lies in the hypothesis class described by infinitely wide networks. The training dynamics are considered in the fully non-linear regime and not reduced to neural tangent kernels. We believe that it will be essential to study these hypothesis classes in detail to choose an appropriate machine learning models for a given problem. The goal of the article is to illustrate this in a mathematically sound and numerically convincing fashion.

Chapter 9 - Effective dimensionality reduction model with machine learning classification for microarray gene expression data

During the last decade, the deluge of multimedia data has impacted a wide range of research areas, including multimedia retrieval, 3D tracking, database management, data mining, machine learning, social media analysis, medical imaging, and so on. Machine learning is largely involved in multimedia applications of building models for classification and regression tasks, etc., and the learning principle consists in designing the models based on the information contained in the multimedia dataset. While many paradigms exist and are widely used in the context of machine learning, most of them suffer from the `curse of dimensionality', which means that some strange phenomena appears when data are represented in a high-dimensional space. Given the high dimensionality and the high complexity of multimedia data, it is important to investigate new machine learning algorithms to facilitate multimedia data analysis. To deal with the impact of high dimensionality, an intuitive way is to reduce the dimensionality. On the other hand, some researchers devoted themselves to designing some effective learning schemes for high-dimensional data. In this survey, we cover feature transformation, feature selection and feature encoding, three approaches fighting the consequences of the curse of dimensionality. Next, we briefly introduce some recent progress of effective learning algorithms. Finally, promising future trends on multimedia learning are envisaged.

Kernel principal component analysis-based Gaussian process regression modelling for high-dimensional reliability analysis