Low‐rank approximation‐based tensor decomposition model for subspace clustering

Advances made to the traditional clustering algorithms solves the various problems such as curse of dimensionality and sparsity of data for multiple attributes. The traditional H-K clustering algorithm can solve the randomness and apriority of the initial centers of K-means clustering algorithm. But when we apply it to high dimensional data it causes the dimensional disaster problem due to high computational complexity. All the advanced clustering algorithms like subspace and ensemble clustering algorithms improve the performance for clustering high dimension dataset from different aspects in different extent. Still these algorithms will improve the performance form a single perspective. The objective of the proposed model is to improve the performance of traditional H-K clustering and overcome the limitations such as high computational complexity and poor accuracy for high dimensional data by combining the three different approaches of clustering algorithm as subspace clustering algorithm and ensemble clustering algorithm with H-K clustering algorithm.

Tensor decomposition (TD) has shown promising performance in image completion and denoising. Existing methods always aim to decompose one tensor into latent factors or core tensors by optimizing a particular cost function based on a specific tensor model. These algorithms iteratively learn the optima from random initialization given any individual tensor, resulting in slow convergence and low efficiency. In this paper, we propose an efficient TD algorithm that aims to learn a global mapping from input tensors to latent core tensors, under the assumption that the mappings of multiple tensors might be shared or highly correlated. To this end, we train a deep neural network (DNN) to model the global mapping and then apply it to decompose a newly given tensor with high efficiency. Furthermore, the initial values of DNN are learned based on meta-learning methods. By leveraging the pretrained core tensor DNN, our proposed method enables us to perform TD efficiently and accurately. Experimental results demonstrate the significant improvements of our method over other TD methods in terms of speed and accuracy.

Fuzzy weighted C-ordered means clustering algorithm

Contrastive adversarial training has shown great potential in enhancing model robustness and has been adopted in point cloud classification. There are varying spatial distributions and densities across different regions in point cloud data, which makes adversarial perturbations always exhibit non-uniform patterns of attack intensity and distribution in different regions. However, existing approaches always rely on uniform feature contrast without considering the granularity in the context of point cloud data, limiting their capacities to counter adversarial perturbations effectively. To address this issue, we propose a novel frequency-aware contrastive adversarial training framework, which considers feature contrast via a “divide-and-conquer” method. Specifically, we systematically “<italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">divide</i>” point clouds into distinct frequency components and “<italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">conquer</i>” feature contrast within each frequency band, which fosters fine-grained feature consistency learning and leads to more informative as well as robust representations. Besides, existing methods typically apply group-level contrastive learning, which emphasizes category-wise similarity but often overlooks the nuanced structural variations among instances. To remedy this, we incorporate instance-level contrastive learning to capture per-instance geometric variations. Moreover, a frequency-specific hard-masked sample generation module is designed to construct challenging sample pairs by masking keypoint features in each frequency band, thereby promoting the model to learn more robust feature representations. Extensive experiments on multiple benchmark datasets demonstrate that our proposed method significantly outperforms existing state-of-the-art approaches in adversarial robustness for point cloud classification. The code is available on DiCon-FAT.

Tensor decomposition is one of the model reduction techniques for compressing deep neural networks. Existing methods use either Tucker decomposition (TD) or Canonical Polyadic decomposition (CPD) for model compression, but none of them tried to combine those two methods, owing to the complexity of choosing a proper decomposition method for each layer. In this paper, we adopted the automatic tuning technique to design an algorithm that can mix both tensor decomposition methods, called Mixed Tensor Decomposition (MTD). The goal is to achieve better compression ratio while keeping similar accuracy as the original models. We used VGG type networks for the case study since they are relatively heavy and computationally expensive. We first studied the relation of model accuracy and compression ratio for Tucker and CPD applying to convolution neural networks (CNN). Based on the studied results, we designed a strategy to select the most suitable decomposition method for each layer, and further fine-tunes the models to recover the accuracy. We have conducted experiments using VGG11 and VGG16 with CIFAR10 dataset, and compared MTD with other tensor decomposition algorithms. The results show that MTD can achieve compression ratio 32 × and 37 × for VGG11 and VGG16 respectively with less than 1% accuracy drops, which is much better than the state-of-the-art tensor decomposition algorithms for model compression.

The separation of Electroencephalography (EEG) sources is a typical application of tensor decompositions in biomedical engineering. The objective of most approaches studied in the literature consists in providing separate spatial maps and time signatures for the identified sources. However, for some applications, a precise localization of each source is required. To achieve this, a two-step approach has been proposed. The idea of this approach is to separate the sources using the canonical polyadic decomposition in the first step and to employ the results of the tensor decomposition to estimate distributed sources in the second step, using the so-called disk algorithm. In this paper, we propose to combine the tensor decomposition and the source localization in a single step. To this end, we directly impose structural constraints, which are based on a priori information on the possible source locations, on the factor matrix of spatial characteristics. The resulting optimization problem is solved using the alternating direction method of multipliers, which is incorporated in the alternating least squares tensor decomposition algorithm. Realistic simulations with epileptic EEG data confirm that the proposed single-step source localization approach outperforms the previously developed two-step approach.

Subspace clustering based on latent low rank representation with Frobenius norm minimization

Mechanical vibration signal mapped into a high-dimensional space tends to exhibit a special distribution and movement characteristics, which can further reveal the dynamic behavior of the original time series. As the most natural representation of high-dimensional data, tensor can preserve the intrinsic structure of the data to the maximum extent. Thus, the tensor decomposition algorithm has broad application prospects in signal processing. High-dimensional tensor can be obtained from a one-dimensional vibration signal by using phase space reconstruction, which is called the tensorization of data. As a new signal decomposition method, tensor-based singular spectrum algorithm (TSSA) fully combines the advantages of phase space reconstruction and tensor decomposition. However, TSSA has some problems, mainly in estimating the rank of tensor and selecting the optimal reconstruction tensor. In this paper, the improved TSSA algorithm based on convex-optimization and permutation entropy (PE) is proposed. Firstly, aiming to accurately estimate the rank of tensor decomposition, this paper presents a convex optimization algorithm using non-convex penalty functions based on singular value decomposition (SVD). Then, PE is employed to evaluate the desired tensor and improve the denoising performance. In order to verify the effectiveness of proposed algorithm, both numerical simulation and experimental bearing failure data are analyzed.

CANDECOMP/PARAFAC (CP) tensor decomposition is a popular unsupervised machine learning method with numerous applications. This process involves modeling a high-dimensional, multi-modal array (a tensor) as the sum of several low-dimensional components. In order to decompose a tensor, one must solve an optimization problem, whose objective is often given by the sum of the squares of the tensor and decomposition model entry differences. One algorithm occasionally utilized to solve such problems is CP-OPT-DGN, a damped Gauss-Newton all-at-once optimization method for CP tensor decomposition. However, there are currently no published results that consider the decomposition of large-scale (with up to billions of non-zeros), sparse tensors using this algorithm. This work considers the decomposition of large-scale tensors using an efficiently implemented CP-OPT-DGN method. It is observed that CP-OPT-DGN significantly outperforms CP-ALS (CP-Alternating Least Squares) and CP-OPT-QNR (a quasi-Newton-Raphson all-at-once optimization method for CP tensor decomposition), two other widely used tensor decomposition algorithms, in terms of accuracy and latent behavior detection.

The development of technology generates huge amounts of non-textual information, such as images. An efficient image annotation and retrieval system is highly desired. Clustering algorithms make it possible to represent visual features of images with finite symbols. Based on this, many statistical models, which analyze correspondence between visual features and words and discover hidden semantics, have been published. These models improve the annotation and retrieval of large image databases. However, image data usually have a large number of dimensions. Traditional clustering algorithms assign equal weights to these dimensions, and become confounded in the process of dealing with these dimensions. In this paper, we propose a top-down, subspace clustering algorithm as a solution to this problem. For a given cluster, we determine relevant features based on histogram analysis and assign greater weight to relevant features as compared to less relevant features. We have implemented four different models to link visual tokens with keywords based on the clustering results of our clustering algorithm and K-means algorithm, and evaluated performance using precision, recall and correspondence accuracy using benchmark dataset. The results show that our algorithm is better than traditional ones for automatic image annotation and retrieval.

Treatment based on the syndrome differentiation is the key of traditional Chinese medicine (TCM) treating acquired immune deficiency syndrome (AIDS). Syndrome differentiation, where the patients suffering from a western medicine disease are divided into several classes based on their symptoms and signs, is an important diagnostic method and affects the effective use of TCM treatments. Some researches show that the clustering algorithms make it possible to classify the AIDS patients into several syndrome types. These algorithms improve the precision of syndrome differentiation so as to promote the TCM treatment efficacy. However, because of the complexity of AIDS disease, the AIDS clinical data usually have a large number of dimensions. The previous cluster algorithms assign equal weights to these dimensions and become confounded in the process of dealing with these dimensions. In this paper, we use a top-down subspace clustering algorithm as a solution to the syndrome differentiation. For a given cluster, we determine the relevant symptoms based on histogram analysis and assign greater weight to the relevant symptoms as compared to less relevant symptoms. Then, the symptoms with greater weight are used to differentiate the syndrome type of AIDS patients. Finally, the proposed method is compared with the traditional k-means algorithm based on the collected AIDS dataset. We evaluate their performance by the precision and the consistency. The experimental results show that the proposed algorithm is better than the traditional ones for aided TCM syndrome differentiation of AIDS patients.

Recently, many data mining techniques were revisited to cope with the new big data challenges. Nearly all of these algorithms considered the efficiency of the mining algorithm to survive the increasing size of the data. However, as the dimensionality of the data increases, not only the efficiency but also the effectiveness of traditional mining algorithms is compromised. For instance, clusters hidden in some subspaces are hard to be detected using traditional clustering algorithms, as the dimensionality of the data increases. In this paper, we consider both the huge size, and the high dimensionality of big data by providing a novel solution that presents a three-phase model for subspace stream clustering algorithms. Our novel model, overcomes the huge size of the big data in its first phase, by continuously applying a streaming concept over the huge data objects, and summarizing them into micro-clusters. Then, after each certain batch of data, or after upon a user request, the second phase is applied over the data summarized in micro-clusters, to reconstruct the current distribution of the data out of the current summaries. In the third phase, a subspace clustering algorithm is applied to overcome the high dimensionality of the data, and to find hidden clusters within some subspace. An extensive evaluation study over different scenarios that follow our model over a big data set is performed.KeywordsSubspace stream clusteringStreaming big dataReal-time streaming analysis of big data

The sparsity and the problem of the curse of dimensionality of high-dimensional data, which make the most of the traditional clustering algorithm, lose action in high-dimensional space. Therefore, clustering of data in high-dimensional space is becoming the hot research areas. By utilizing the subtractive clustering as initialized method, and combine with the revised clustering validation indices, this paper offers a subspace clustering algorithm for automatically determining the optimal number of clusters on high dimensional data. The experiment results show that the proposed clustering algorithm can get better cluster validation performance than that of conventional indices.

Efficient denoising of hyperspectral imagery (HSI) relies on an representational model that is capable of capturing the spatial and spectral correlation in HSI. Recently, an intrinsic representation (IR) approach based on the linear spectral mixture model (LSMM) was proposed for unsupervised feature extraction. The IR model constitutes a sound representational model due to its ability to account for the physical data generation process of HSI, the spatial correlation effect, and the noise variance heterogeneity effect. In this paper, we explore the potential of IR for the denoising of HSI. A noisy pixel in HSI is expressed as a nonnegative linear combination of several endmembers, plus some Gaussian noise with heterogeneous noise variances. In order to perform denoising, the IR approach is used to adaptively estimate both the endmembers and the nonnegative coefficients (i.e., the abundances), which are finally used to reconstruct the clean image. The experiments on both simulated and real hyperspectral images demonstrate that the IR approach not only can resist the influence of noise, but also can preserve the image details.

This paper presents a new algorithm for subspace clustering for high dimensional data. It is an iterative algorithm based on the minimization of an objective function. A major weakness of subspace clustering algorithms is that almost all of them are developed based on within- class information only or by employing both within-cluster and between- clusters information. The density of cluster is lost. The new function is developed by integrating the separation and compactness of clusters. The density of cluster is introduced also in the compactness term. The experimental results confirm that the proposed algorithm gives good results on different types of images by optimizing the runtime.