Efficient Streaming Detection of Hidden Clusters in Big Data Using Subspace Stream Clustering

Advances made to the traditional clustering algorithms solves the various problems such as curse of dimensionality and sparsity of data for multiple attributes. The traditional H-K clustering algorithm can solve the randomness and apriority of the initial centers of K-means clustering algorithm. But when we apply it to high dimensional data it causes the dimensional disaster problem due to high computational complexity. All the advanced clustering algorithms like subspace and ensemble clustering algorithms improve the performance for clustering high dimension dataset from different aspects in different extent. Still these algorithms will improve the performance form a single perspective. The objective of the proposed model is to improve the performance of traditional H-K clustering and overcome the limitations such as high computational complexity and poor accuracy for high dimensional data by combining the three different approaches of clustering algorithm as subspace clustering algorithm and ensemble clustering algorithm with H-K clustering algorithm.

Fuzzy partition based soft subspace clustering and its applications in high dimensional data

The sparsity and the problem of the curse of dimensionality of high-dimensional data, which make the most of the traditional clustering algorithm, lose action in high-dimensional space. Therefore, clustering of data in high-dimensional space is becoming the hot research areas. By utilizing the subtractive clustering as initialized method, and combine with the revised clustering validation indices, this paper offers a subspace clustering algorithm for automatically determining the optimal number of clusters on high dimensional data. The experiment results show that the proposed clustering algorithm can get better cluster validation performance than that of conventional indices.

The text data of the social network platforms take the form of short texts, and the massive text data have high-dimensional and sparse characteristics, which does not make the traditional clustering algorithm perform well. In this paper, a new community detection method based on the sparse subspace clustering (SSC) algorithm is proposed to deal with the problem of sparsity and the high-dimensional characteristic of short texts in online social networks. The main ideal is as follows. First, the structured data including users’ attributions and user behavior and unstructured data such as user reviews are used to construct the vector space for the network. And the similarity of the feature words is calculated by the location relation of the feature words in the synonym word forest. Then, the dimensions of data are deduced based on the principal component analysis in order to improve the clustering accuracy. Further, a new community detection method of social network members based on the SSC is proposed. Finally, experiments on several data sets are performed and compared with the K-means clustering algorithm. Experimental results show that proper dimension reduction for high dimensional data can improve the clustering accuracy and efficiency of the SSC approach. The proposed method can achieve suitable community partition effect on online social network data sets.

Nowadays a lot of massive data is stored and typically, the data itself contains a lot of non-trivial but useful information. Data mining techniques can be used to discover this information which can help the companies for decision-making. However, in real life applications, data is massive and is stored over distributed sites. One of my major research topics is to protect privacy over this kind of data. Previously, the important characteristics, issues and challenges related to management of the large amount of data has been explored. Various open source data analytics frameworks that deal with large amount of Data analytics workloads have been discussed. Comparative study between the given frameworks and suitability of the same has been proposed. Digital universe is flooded with huge amount of data generated by number of users worldwide. These data are of diverse in nature, come from various sources and in many forms. To keep with the desire to store and analyze ever larger volumes of complex data, relational databases vendors have delivered specialized analytical platforms that come in many shapes and sizes from software only to analytical services that run in third party hosted environments. In addition new technologies have emerged to address exploding volumes of complex data, including web traffic, social media content and machine generated data including sensor data, global positioning system data.Big data is defined as large amount of data which requires new technologies and architectures so that it becomes possible to extract value from it by capturing and analysis process. Due to such large size of data it becomes very difficult to perform effective analysis using the existing traditional techniques. Big data has become a prominent research field, especially when it comes to decision making and data analysis. However, Big data due to its various properties like volume, velocity, variety, variability, value and complexity put forward many challenges. Since Big data is a recent upcoming technology in the market which can bring huge benefits to the business organizations, it becomes necessary that various challenges and issues associated in bringing and adapting to this technology are brought into light. Another challenge is that data collection may not have enough accuracy which will lead to a non-consistent analysis that can critically affect the decision based on this analysis. Moreover, it is clearly apparent that organizations need to employ data-driven decision making to gain competitive advantage. Processing, integrating and interacting with more data should make it better data, providing both more panoramic and more granular views to aid strategic decision making. This is made possible via Big Data exploiting affordable and usable Computational and Storage Resources. Many offerings are based on the Map-Reduce and Hadoop paradigms and most focus solely on the analytical side. Nonetheless, in many respects it remains unclear what Big Data actually is; current offerings appear as isolated silos that are difficult to integrate and/or make it difficult to better utilize existing data and systems. Since data is growing at a huge speed making it difficult to handle such large amount of data (Exabyte). The main difficulty in handling such large amount of data is because that the volume is increasing rapidly in comparison to the computing resources. The Big data term which is being used now a days is kind of misnomer as it points out only the size of the data not putting too much of attention to its other existing properties.If data is to be used to make accurate decisions in time it becomes necessary that it should be available in accurate, complete and timely manner. This makes the data management and governance process bit complex adding the necessity to make Data open and make it available to government agencies in standardized manner with standardized APIs, metadata and formats thus leading to better decision making, business intelligence and productivity improvements.This paper presents a discussion and evaluation for the most prominent techniques used in the processes of data collection and analysis in order to identify the privacy defects in them that affects the accuracy of big data. Depending on the results of this analysis, recommendations were provided for improving data collection and analysis techniques that will help to avoid if not all then most of the problems facing the use of big data in decision making. Keywords: Big Data, Big Data Challenges, Big Data Accuracy, Big Data Collection, Big Data Analytics.

Clustering algorithms have become one of the most critical research areas in multiple domains, especially data mining. However, with the massive growth of big data applications in the cloud world, these applications face many challenges and difficulties. Since Big Data refers to an enormous amount of data, most traditional clustering algorithms come with high computational costs. Hence, the research question is how to handle this volume of data and get accurate results at a critical time. Despite ongoing research work to develop different algorithms to facilitate complex clustering processes, there are still many difficulties that arise while dealing with a large volume of data. In this paper, we review the most relevant clustering algorithms in a categorized manner, provide a comparison of clustering methods for large-scale data and explain the overall challenges based on clustering type. The key idea of the paper is to highlight the main advantages and disadvantages of clustering algorithms for dealing with big data in a scalable approach behind the different other features.

Based on the traditional grid density clustering algorithm, proposing A Data Stream Clustering Algorithm Based on Density and Extended Grid(DEGDS). The algorithm combines the advantages of grid clustering algorithm and density clustering algorithm, by improving the defects of clustering parameters by artificially set, get any shape of the cluster. The algorithm uses the local density of each sample point and the distance from the other sample points, determining the number of clustering centers in the grid, and realizing the automatic determination of the clustering center, which avoids the influence of improper selection of initial centroid on clustering results. And in the process of combining the Spark parallel framework for partitioning the data to achieve its parallelization. For data points clustered outside the grid, the clustering within the grid has been effectively expanded by extending the grid, to ensure the accuracy of clustering. Introduced density estimation is connected and grid boundaries to merging grid, saving memory consumption. Using the attenuation factor to incremental update grid density, reflect the evolution of spatial data stream. The experimental results show that compared with the traditional clustering algorithm, the DEGDS algorithm has a large performance improvement in accuracy and efficiency, and can be effectively for large data clustering.

Models learned from high-dimensional spaces, where the high number of features can exceed the number of observations, are susceptible to overfit since the selection of subspaces of interest for the learning task is prone to occur by chance. In these spaces, the performance of models is commonly highly variable and dependent on the target error estimators, data regularities and model properties. High-variable performance is a common problem in the analysis of omics data, healthcare data, collaborative filtering data, and datasets composed by features extracted from unstructured data or mapped from multi-dimensional databases. In these contexts, assessing the statistical significance of the performance guarantees of models learned from these high-dimensional spaces is critical to validate and weight the increasingly available scientific statements derived from the behavior of these models. Therefore, this chapter surveys the challenges and opportunities of evaluating models learned from big data settings from the less-studied angle of big dimensionality. In particular, we propose a methodology to bound and compare the performance of multiple models. First, a set of prominent challenges is synthesized. Second, a set of principles is proposed to answer the identified challenges. These principles provide a roadmap with decisions to: i) select adequate statistical tests, loss functions and sampling schema, ii) infer performance guarantees from multiple settings, including varying data regularities and learning parameterizations, and iii) guarantee its applicability for different types of models, including classification and descriptive models. To our knowledge, this work is the first attempt to provide a robust and flexible assessment of distinct types of models sensitive to both the dimensionality and size of data. Empirical evidence supports the relevance of these principles as they offer a coherent setting to bound and compare the performance of models learned in high-dimensional spaces, and to study and refine the behavior of these models.Keywordshigh-dimensional dataperformance guaranteesstatistical significance of learning modelserror estimatorsclassificationbiclustering

In high dimensional data, general performance of the traditional clustering algorithms decreases. As some dimensions are likely to be irrelevant or contain noisy data and randomly selected initial centre of the clusters converge the clustering to local minima. In this paper, we propose a framework for clustering high dimensional data with attribute subset selection and efficient cluster centre initialization. It uses rough set theory to determine the relevant attributes (dimensions) in first phase. In second phase, maximum variance dimension is used to determine the optimal initial centres of the clusters. The k-means clustering algorithm is applied with these initial cluster centres, in phase three, to find optimal clustering of data set. It improves efficiency of the clustering process tremendously and our experiment on test data set shows that accuracy of the results has improved considerably.KeywordsFeature selectionSubspaceRough setK-MeansHigh dimensional data

Pavement distress images are typical high dimensional nonlinear data. Manifold learning algorithms can find the intrinsic characteristic hidden in the distress images, which helps to better recognize them. Unlike most of manifold learning algorithms, multilayer autoencoders have solved the data reconstructed problem through building a bi-directional mapping between the high dimensional data and the low dimensional data. An automatic pavement distress image recognition method based on multilayer autoencoders was proposed, which combined the image processing method and multilayer autoencoders. In the method, the distress images were firstly processed with the image processing method. Then the images were reduced dimensions and reconstructed with multilayer autoencoders. Lastly, the distress type was recognized through the network. Experiments showed that the recognition accuracy with the proposed method was great higher than that with the BP neural network.

针对数据流的聚类算法，近年来取得了有效的进展，出现了许多卓有成效的算法。随着信息采集技术的进步，需要处理的数据量越来越大，需要研究针对数据流的并行聚类算法。本文基于串行的数据流聚类算法D-Stream作出并行化改进，用通用的大数据处理框架Spark设计了一个基于分布式架构运行的动态数据聚类算法PDStream。实验结果表明，该算法具有更高的效率和良好的扩展性，能够实现分布式架构下的流数据动态聚类。 In the era of big data, with the rapid growth of data size, the requirements of data processing in-crease constantly. It has put forward many effective algorithms for data stream clustering these years. However, with the continuous development of social technology, single machine environ-ment has been difficult to meet the needs of data mining. Cluster environment is used more for information collection and data processing, the traditional clustering algorithm does not adapt well to the new processing requirements. This paper made some improvements from the data stream clustering algorithm D-Stream, used the big data processing framework Spark and designed a dynamic data clustering algorithm PDStream based on distributed architecture. The new algorithm is proved to be more efficient and able to perform dynamic clustering tasks under distributed architecture from the results of experiment.

Radio frequency identification (RFID) technology has been successfully applied to gather customers' shopping habits from their motion paths and other behavioral data. The customers' behavioral data can be used for marketing purposes, such as improving the store layout or optimizing targeted promotions to specific customers. Some data mining techniques, such as clustering algorithms can be used to discover customers' hidden behaviors from their shopping paths. However, shopping path data has peculiar challenges, including variable length, sequential data, and the need for a special distance measure. Due to these challenges, traditional clustering algorithms cannot be applied to shopping path data. In this paper, we analyze customer behavior from their shopping path data by using a clustering algorithm. We propose a new distance measure for shopping path data, called the Operation edit distance, to solve the aforementioned problems. The proposed distance method enables the RFID customer shopping path data to be processed effectively using clustering algorithms. We have collected a real-world shopping path data from a retail store and applied our method to the dataset. The proposed method effectively determined customers' shopping patterns from the data.

Traditional clustering algorithms are generally either keyword or index based but not semantic based. These algorithms are facing difficulties in identifying synonymies or polysemies due to high dimensionality of text data. Ontologies are identified to overcome these difficulties. In this paper, we propose a framework which automates the extraction of concepts or terms with support of: a) our proposed metric called term rank identifier (TRI), it measures the frequent terms; b) semantically enriched terms (SETs) clustering algorithm, it calculates the semantic relation between the terms with Word net; c) Ontology Building can be done automatically for the concepts extracted from SET Clustering using inference engines. The experimental results show that our proposed metric TRI and SET clustering algorithm performed significantly.

High-dimensional mixed-attribute data clustering has become an important research direction in data mining area. Because of the advantages of the information technology, data collection becomes much easier and thus result in the dimensions and complexity of the database are growing much larger. The traditional clustering algorithms are often unable to use these data to obtain good results. This paper presents mixed attributes of a high-dimensional clustering algorithm GA-KP, which has been applied to customer data segmentation of automobile industry collaborative platform.

The Bayesian network (BN) model, as a big data graph model that integrates causal inference and probabilistic representation, has received widespread attention and research in both academia and industry. However, with the advent of the big data era, traditional BN structure learning algorithms have encountered unprecedented challenges in processing high-dimensional data, mainly manifested as a sharp increase in computational complexity and difficulty in achieving ideal accuracy requirements within an acceptable time range, which greatly limits their breadth and depth in practical applications. In response to this bottleneck problem, this article innovatively proposes a new approach that combines width learning theory with BN, referred to as Broad Bayesian Neural Network (Broad-BNN). This model effectively reduces the dimensionality of the original high-dimensional data by introducing a feature mapping layer and gradually expanding it, while achieving non-linear transformation of information and effective feature extraction. The experimental results show that the model proposed in this paper has achieved significant performance improvement in high-dimensional data classification problems, not only accelerating training speed but also significantly improving classification accuracy, providing a new perspective and solution for solving the difficulties of high-dimensional data processing.