Scalable Varied Density Clustering Algorithm for Large Datasets

Data clustering refers to the process of dividing a set of objects into homogeneous groups or clusters such that the objects in each cluster are more similar to each other than to those of other clusters. As one of the most popular tools for exploratory data analysis, data clustering has been applied in many scientific areas. In this article, we give a review of the basics of data clustering, such as distance measures and cluster validity, and different types of clustering algorithms. We also demonstrate the applications of data clustering in insurance by using two scalable clustering algorithms, the truncated fuzzy c-means (TFCM) algorithm and the hierarchical k-means algorithm, to select representative variable annuity contracts, which are used to build predictive models. We found that the hierarchical k-means algorithm is efficient and produces high-quality representative variable annuity contracts.

The sparsity and the problem of the curse of dimensionality of high-dimensional data, which make the most of the traditional clustering algorithm, lose action in high-dimensional space. Therefore, clustering of data in high-dimensional space is becoming the hot research areas. By utilizing the subtractive clustering as initialized method, and combine with the revised clustering validation indices, this paper offers a subspace clustering algorithm for automatically determining the optimal number of clusters on high dimensional data. The experiment results show that the proposed clustering algorithm can get better cluster validation performance than that of conventional indices.

Cluster analysis is the assignment of grouping a set of observations into clusters so that observations in the same cluster are similar in some sense. One of the key features for clustering is how to define a sensible similarity measure. However, classical clustering algorithms have no ability to cluster data instances and imprecise concepts using traditional distance measures. In this paper, we proposed a (dis)similarity measure based on a new knowledge representation framework called label semantics. Based on this new measure, we can automatically cluster data instance and descriptive concepts represented by logical expressions of linguistic labels. Experimental results on a toy problem in image classification demonstrate the effectiveness of the new proposed clustering algorithm. Since the new proposed measure can be extended to measuring distance between any two granularities, the new clustering algorithms can also be extended to clustering data instance and imprecise concepts represented by other granularities.

Clustering algorithm plays important roles in many fields, such as data mining, data visualization, and so on. In a variety of clustering algorithms, the CHAMELEON algorithm has become a commonly used algorithm because of its ability to discover clusters with arbitrary shapes. This paper makes an intensive study of clustering algorithm, especially CHAMELEON algorithm. After the study of the structure and features of the proposed algorithm, this paper introduces an optimized CHAMELEON algorithm based on local features and grid structure aiming at the deficiency of CHAMELEON. This algorithm generates neighbor graph adaptively, partitions the graph and merges sub-clusters based on local features, so it can produce high quality clustering results. The proposed clustering algorithm is experimented with four data sets, and its performance is compared with CHAMELEON algorithm, DBSCAN algorithm and K-means algorithm. The experimental results show that the clustering algorithm proposed can obtain satisfied clustering effect of data sets with complex distribution.

Designing consensus function by analyzing ensemble method of clustering members and clustering, then the class center will be determined by the way of using the weighted method. Clustering ensemble, which is Based on part priority clustering algorithm, is to improve the accuracy of the algorithm. Experiments results proved that the algorithm which is optimized takes great advantages over that in scalability, stability and robustness than before.

Clustering algorithms have become one of the most critical research areas in multiple domains, especially data mining. However, with the massive growth of big data applications in the cloud world, these applications face many challenges and difficulties. Since Big Data refers to an enormous amount of data, most traditional clustering algorithms come with high computational costs. Hence, the research question is how to handle this volume of data and get accurate results at a critical time. Despite ongoing research work to develop different algorithms to facilitate complex clustering processes, there are still many difficulties that arise while dealing with a large volume of data. In this paper, we review the most relevant clustering algorithms in a categorized manner, provide a comparison of clustering methods for large-scale data and explain the overall challenges based on clustering type. The key idea of the paper is to highlight the main advantages and disadvantages of clustering algorithms for dealing with big data in a scalable approach behind the different other features.

The density based algorithms such as DBSCAN is considered as one of the most common and powerful algorithms in data clustering with the noise datasets. DBSCAN based algorithm’s is able to find out clusters with the different shape and variable size. However it is failed to detect the correct clusters, if there is density variation within the clusters. This paper presents new way to solve the problem of detecting the clusters of varying density which most of the DBSCAN based algorithms can’t deal with it correctly. Our proposed approach is depending on oscillation of clusters which is obtained by applying basic DBSCAN algorithm to conflation it in a new clusters, the proposed algorithm help to decide whether the different density regions belong to the same cluster or not. The experimental results showed that the proposed clustering algorithm gives satisfied results on different Data sets.

Recommendation systems have been widely used in various applications to solve information overload andimprove user experience. Traditional recommendation algorithms mainly used Euclidean data for calculationand abandoned the graph structure features in user and item data. Aiming at the problems in the current recommendation algorithms, this paper proposes an improved user density clustering method and extracts userfeatures through optimized graph neural network. Firstly, the improved density clustering method is used toform the clustering subgraph of users based on the influence value of users. Secondly, the user data and itemdata features of cluster subgraph are extracted by graph convolution network. Finally, the features of clustersubgraphs are processed by global graph convolution network and the recommendation results are generatedaccording to the global graph features. This model not only improves the efficiency of decomposing large graphinto small graph through the improved user density clustering algorithm, but also extracts the features of usergroups through graph convolution neural network to improve the recommendation effect. The experiment alsoproves the validity of this model.

<span>Clustering is a significant approach in data mining, which seeks to find groups or clusters of data. Both numeric and categorical features are frequently used to define the data in real-world applications. Several different clustering algorithms are proposed for the numerical and categorical datasets. In clustering algorithms, the quality of clustering results is evaluated using cluster validation. This paper proposes an efficient clustering algorithm for mixed numerical and categorical data using re-clustering and cluster validation. Initially, the mixed dataset is clustered with four traditional clustering algorithms like expectation-maximization (EM), hierarchical cluster (HC), k-means (KM), and self-organizing map (SOM). These four algorithms are validated, and the best algorithm is selected for re-clustering. It is an iterative process for improving the quality of cluster results. The incorrectly clustered data is iteratively re-clustered and evaluated based on the cluster validation. The performance of the proposed clustering method is evaluated with a real-time dataset in terms of purity, normalized mutual information, rand index, precision, and recall. The experimental results have shown that the proposed reclust algorithm achieves better performance compared to other clustering algorithms.</span>

In view of the considerable applications of data clustering techniques in various fields, such as engineering, artificial intelligence, machine learning, clinical medicine, biology, ecology, disease diagnosis, and business marketing, many data clustering algorithms and methods have been developed to deal with complicated data. These techniques include supervised learning methods and unsupervised learning methods such as density-based clustering, K-means clustering, and K-nearest neighbor clustering. This book reviews recently developed data clustering techniques and algorithms and discusses the development of data clustering, including measures of similarity or dissimilarity for data clustering, data clustering algorithms, assessment of clustering algorithms, and data clustering methods recently developed for insurance, psychology, pattern recognition, and survey data.

Spectral clustering has shown a superior performance in analyzing the cluster structure. However, the exponentially computational complexity limits its application in analyzing large-scale data. To tackle this problem, many low-rank matrix approximating algorithms are proposed, of which the Nystrom method is an approach with proved lower approximate errors. The algorithms commonly combine two powerful techniques in machine learning: spectral clustering algorithms and Nystrom methods commonly used to obtain good quality low rank approximations of large matrices. This paper proposes to analyze a scalable Nystrom-based clustering algorithm with a Minimum Sum of Squared Similarities (MSSS) sampling procedure. We provide theoretical analysis of the performance of the algorithm MSSS and demonstrate its theoretical performance in comparison to the leading spectral clustering methods that use Nystrom sampling.

In this paper, we present a scalable evolutionary algorithm for clustering large and dynamic data sets, called Scalable Evolutionary Clustering with Self Adaptive Genetic Operators (Scalable ECSAGO). The proposed evolutionary clustering algorithm can adapt its genetic operators rate while the evolution leads to the optimal centers of the clusters. The sizes of the clusters are estimated using a hybrid analytical optimization procedure. Moreover, a memorization factor is introduced in order to allow the algorithm to keep as much of the previously discovered knowledge about clusters and data summarization as desired. The proposed scalable ECSAGO algorithm is able to find accurate representations of the clusters on very large data sets of different sizes and dimensionality that might not fit in main memory, while maintaining the desirable properties of robustness to noise and automatic detection of the number of clusters. The algorithm is also useful for traking evolving cluster structures that change with the passage of time.

PFHC: A clustering algorithm based on data partitioning for unevenly distributed datasets

Clustering algorithms play a pivotal role in data mining, offering powerful tools for uncovering hidden patterns and structures within datasets. These algorithms aim to divide data points into coherent groups based on similarities or dissimilarities, making it easier to explore and understand complex data. Clustering algorithms typically rely on similarity measures to assess the likeness between data points. Consequently, selecting a suitable similarity measure is crucial for achieving satisfactory clustering outcomes. However, this decision can pose significant challenges, especially for non-experts, given the plethora of similarity measures available in the literature and their performance which is closely linked to the specific dataset, clustering algorithm, and cluster validity index employed. This difficulty is even more important when considering mixed data clustering. Mixed data refers to heterogeneous data characterized by both numerical and categorical attributes. In such a context, the same similarity measure cannot be used for both types of attributes due to their different nature. Commonly, two similarity measures are combined, one for numerical attributes and one for categorical attributes. This adds a layer of complexity to the problem since it requires the selection of two similarity measures instead of just one. This paper introduces SIMREC, a similarity measure recommendation system for mixed data clustering. The system uses meta-learning to mine the relationship between dataset characteristics and similarity measures performances for different mixed data clustering algorithms and cluster validity indices. Therefore, given a mixed dataset, a mixed data clustering algorithm, and a cluster validity index, the system can recommend suitable pairs of numerical and categorical similarity measures based on the characteristics of the dataset. We implemented the proposed system using 130 pairs of similarity measures (10 numerical and 13 categorical), 4 commonly used mixed data clustering algorithms (K-Prototypes, LSH-K-Prototypes, K-Medoids, and Hierarchical Clustering), and three cluster validity indices (Silhouette, Clustering Accuracy, and Adjusted Rand Index). Our experiments on 185 publicly available mixed datasets show that the pairs of similarity measures recommended by SIMREC outperform the baseline pairs, including classically used pairs of similarity measures in the literature.

Data clustering is an important step in data mining and machine learning. It is especially crucial to analyze the data structures for further procedures. Recently a new clustering algorithm known as ‘neutrosophic c-means’ (NCM) was proposed in order to alleviate the limitations of the popular fuzzy c-means (FCM) clustering algorithm by introducing a new objective function which contains two types of rejection. The ambiguity rejection which concerned patterns lying near the cluster boundaries, and the distance rejection was dealing with patterns that are far away from the clusters. In this paper, we extend the idea of NCM for nonlinear-shaped data clustering by incorporating the kernel function into NCM. The new clustering algorithm is called Kernel Neutrosophic c-Means (KNCM), and has been evaluated through extensive experiments. Nonlinear-shaped toy datasets, real datasets and images were used in the experiments for demonstrating the efficiency of the proposed method. A comparison between Kernel FCM (KFCM) and KNCM was also accomplished in order to visualize the performance of both methods. According to the obtained results, the proposed KNCM produced better results than KFCM.