P-ROCK: A Sustainable Clustering Algorithm for Large Categorical Datasets

Clustering is one of the most important applications of data mining. It has attracted attention of researchers in statistics and machine learning. It is used in many applications like information retrieval, image processing and social network analytics etc. It helps the user to understand the similarity and dissimilarity between objects. Cluster analysis makes the users understand complex and large data sets more clearly. There are different types of clustering algorithms analyzed by various researchers. Kmeans is the most popular partitioning based algorithm as it provides good results because of accurate calculation on numerical data. But Kmeans give good results for numerical data only. Big data is combination of numerical and categorical data. Kprototype algorithm is used to deal with numerical as well as categorical data. Kprototype combines the distance calculated from numeric and categorical data. With the growth of data due to social networking websites, business transactions, scientific calculation etc., there is vast collection of structured, semi-structured and unstructured data. So, there is need of optimization of Kprototype so that these varieties of data can be analyzed efficiently.In this work, Kprototype algorithm is implemented on MapReduce in this paper. Experiments have proved that Kprototype implemented on Mapreduce gives better performance gain on multiple nodes as compared to single node. CPU execution time and speedup are used as evaluation metrics for comparison.Intellegent splitter is proposed in this paper which splits mixed big data into numerical and categorical data. Comparison with traditional algorithms proves that proposed algorithm works better for large scale of data.

Agricultural regionalization, which is largly achieved according to the similarity within a certain area and the difference between this area and other ones, is the foundation of agricultural production. Due to the fact that clustering is much like regionalization, methods for clustering are also commonly used in regionalization. However, the clustering algorithms applied are usually for only numerical attributes and large amounts of categorical data with great values cannot be handled with traditional clustering methods, which largely limits the utilization of clustering in agricultural regionalization. In this paper, we propose a new spatial clustering algorithm which combines the ROCK algorithm with fuzzy mathematics, can handle both numerical and categorical data at the same time, to satisfy the actual needs of agriculture regionalization. The effectiveness of the new algorithm is tested on the agricultural regionalization of ZengCheng, GuangZhou, China. During the test, we test both the effectiveness of the new spatial clustering algorithm and some old algorithms. The final result shows that the new algorithm performs better in agricultural regionalization, and its result is also closer to the artificial agricultural regionalization.

Extracting knowledge from hybrid data, comprising both categorical and numerical data, poses significant challenges due to the inherent difficulty in preserving information and practical meanings during the conversion process. To address this challenge, hybrid data processing methods, combining complementary rough sets, have emerged as a promising approach for handling uncertainty. However, selecting an appropriate model and effectively utilizing it in data mining requires a thorough qualitative and quantitative comparison of existing hybrid data processing models. This research aims to contribute to the analysis of hybrid data processing models based on neighborhood rough sets by investigating the inherent relationships among these models. We propose a generic neighborhood rough set-based hybrid model specifically designed for processing hybrid data, thereby enhancing the efficacy of the data mining process without resorting to discretization and avoiding information loss or practical meaning degradation in datasets. The proposed scheme dynamically adapts the threshold value for the neighborhood approximation space according to the characteristics of the given datasets, ensuring optimal performance without sacrificing accuracy. To evaluate the effectiveness of the proposed scheme, we develop a testbed tailored for Parkinson’s patients, a domain where hybrid data processing is particularly relevant. The experimental results demonstrate that the proposed scheme consistently outperforms existing schemes in adaptively handling both numerical and categorical data, achieving an impressive accuracy of 95% on the Parkinson’s dataset. Overall, this research contributes to advancing hybrid data processing techniques by providing a robust and adaptive solution that addresses the challenges associated with handling hybrid data, particularly in the context of Parkinson’s disease analysis.

Centre-based clustering is among the most applicable method for partitioning objects into homogenous groups. This paper presents two Centre-based clustering; K-Means and K-Modes algorithms to investigate and evaluate the clustering results of Y-STR data. The main goal of this paper is to compare the accuracy of clustering Y-STR results for different types of data: numerical and categorical data. The results show that the Y-STR data is more favour to categorical data. The accuracy of the Y-STR, treated as categorical data is 49%, whereas the numerical data is only a 26% chance producing a good clustering result. However, the amount of time taken by numerical data is much better compared to categorical data.

The k-modes clustering algorithm is undoubtedly one of the most widely used partitional algorithms for categorical data. Unfortunately, due to its gradient descent nature, this algorithm is highly sensitive to the initialization of clustering. Categorical initialization methods have been proposed to address this problem. In this paper, we present an overview of initialization methods of clustering for numerical data and categorical data respectively with an emphasis on their computational efficiency. We then propose a new initialization method for categorical data, which can obtain the good initial cluster centers using the new distance base on the RD, and explore the methods of density and grid. Finally, proposed method has been tested on diagnosis dataset, a real world data set from UCI Machine Learning Repository, and been analyzed the experimental results, which illustrates that the proposed method is effective and efficient for initializing categorical data.

Mathematics achievement could be influenced by cognitive and non-cognitive factors. The potential variable of cognitive factor is metacognition, whereas non-cognitive factors include Economic, Social, and Cultural Status (ESCS), resilience, life satisfaction, happiness, pride, fear, sadness, and gender. Those variables involve numerical and categorical data. For this reason, this study aims to apply the Radial Basis Function Neural Network (RBFNN) model with ensemble clustering to model the relation between cognitive and non-cognitive aspects and mathematics achievement. The RBFNN is a soft computing approach based on the neural network model and has been shown as an effective model and free of assumption. The ensemble clustering is a process in RBFNN modeling to capture the independent variables involving the numerical and categorical data. It employs K-means clustering for the numerical data and K-modes for categorical data and combines the results of those two methods. The data used in this study are published by PISA (Program for International Student Assessment) 2018. The results show that the RBFNN with ensemble clustering deliver good performance in modeling the students’ mathematics achievement based on the cognitive and non-cognitive factors in terms of prediction accuracy. Other than RBFNN model, the use of cognitive and non-cognitive factors involving in this study also contributes to the high accuracy prediction. This further emphasizes that these factors are good predictors of mathematic achievement. Additionally, we suggest the silhouette cluster validation in the clustering process, since it leads to the number of hidden neurons of the best RBFNN model.

This paper introduces a self-organizing map dedicated to clustering, analysis and visualization of categorical data. Usually, when dealing with categorical data, topological maps use an encoding stage: categorical data are changed into numerical vectors and traditional numerical algorithms (SOM) are run. In the present paper, we propose a novel probabilistic formalism of Kohonen map dedicated to categorical data where neurons are represented by probability tables. We do not need to use any coding to encode variables. We evaluate the effectiveness of our model in four examples using real data. Our experiments show that our model provides a good quality of results when dealing with categorical data.

Various applications of data analysis and their effects have been reported recently. With the remarkable progress in classification methods, one example being support vector machines, clustering as the main method of unsupervised classification has also been studied extensively. Consequently, fuzzy methods of clustering is becoming a standard technique. However, unsolved theoretical and methodological problems in fuzzy clustering remain and have to be studied more deeply. This issue collects five papers concerned with fuzzy clustering and related fields, and in all of them the main interest is methodology. Kondo and Kanzawa consider fuzzy clustering with a new objective function using q-divergence, which is a generalization of the well-known Kullback-Leibler divergence. Among different data types, they focus on categorical data. They also show the relations of different methods of fuzzy c-means. Thus, this study tends to further generalize methods of fuzzy clustering, trying to find the methodological boundaries of the capabilities of fuzzy clustering models. Kitajima, Endo, and Hamasuna propose a method of controlling cluster sizes so that the resulting clusters have an even size, which is different from the optimizing of cluster sizes dealt with in other studies. This technique enhances application fields of clustering in which cluster sizes are more important than cluster shapes. Hamasuna et al. study the validity measures of clusters for network data. Cluster validity measures are generally proposed for points in Euclidean spaces, but the authors consider the application of validity measures to network data. Several validity measures are modified and adapted to network data, and their effectiveness is examined using simple network examples. Ubukata et al. propose a new method of c-means related to rough sets, a method based on a different idea from well-known rough c-means by Lingras. Finally, Kusunoki, Wakou, and Tatsumi study the maximum margin model for the nearest prototype classifier that leads to the optimization of the difference of convex functions. All papers include methodologically important ideas that have to be further investigated and applied to real-world problems.

PurposeThe aim of the study is to propose certain agricultural policy measures for the financing and development of Greek farms, established by young farmers, based on the results of a clustering method suitable for handling socio‐economic categorical data.Design/methodology/approachThe clustering method was applied to categorical data collected from 110 randomly selected investment plans of Greek agricultural farms. The investment plans were submitted to the “Region of Central Macedonia” administrative office, in the framework of the Operational Programme “Agricultural Development – Reform of the Countryside 2000‐2006” and refer to agricultural investments by “Young Farmers”, according to the terms and conditions of Priority Axis III: “Improvement of the Age Composition of the Agricultural Population”. The input variables for the analyses were the farmers' gender, age class, education level and permanent place of residence, the farms' agricultural activity, Human Labour Units (HLU) and farms' viability level. All these variables were measured on nominal or ordinal scales. The available data were analyzed by means of a hierarchical cluster analysis method applied on the rows of an appropriate matrix of a complete disjunctive form with a dummy coding 0 or 1. The similarities were measured through the Benzécri'sχ2distance (metric), while the Ward's method was used as a criterion for cluster formation.FindingsFive clusters of farms emerged, with statistically significant diverse socio‐economic profiles. The most important impact on the formation of the groups of farms was found to be related to the number of HLU, the farmers' level of education and gender. This derived typology allows for the determination of a flexible development and funding policy for the agricultural farms, based on the socio‐economic profile of the formulated clusters.Research limitations/implicationsOne of the limitations of the current study derives from the fact that the clustering method used is suitable only for categorical, non‐metric data. Another limitation comes from the fact that a relative small number of investment plans were used in the analysis. A larger sample covering and other geographical regions is needed in order to confirm the current results and make nation‐wide comparisons and “tailor‐made” proposals for financing and development. Finally, it is interesting to contact longitudinal surveys in order to evaluate the effectiveness of the funding policy of the corresponding programme.Originality/valueThe study's results could be useful to practitioners and academics because certain agricultural policy measures for the financing and development of Greek farms established by young farmers are proposed. Additionally, the data analysis method used in this study offers an alternative way for clustering categorical data.

Some Thoughts About Data Type, Distribution, and Statistical Significance

Computational methods for identification of cell populations from polychromatic flow cytometry data are changing the paradigm of cytometry bioinformatics. Data clustering is the most common computational approach to unsupervised identification of cell populations from multidimensional cytometry data. However, interpretation of the identified data clusters is labor-intensive. Certain types of user-defined cell populations are also difficult to identify by fully automated data clustering analysis. Both are roadblocks before a cytometry lab can adopt the data clustering approach for cell population identification in routine use. We found that combining recursive data filtering and clustering with constraints converted from the user manual gating strategy can effectively address these two issues. We named this new approach DAFi: Directed Automated Filtering and Identification of cell populations. Design of DAFi preserves the data-driven characteristics of unsupervised clustering for identifying novel cell subsets, but also makes the results interpretable to experimental scientists through mapping and merging the multidimensional data clusters into the user-defined two-dimensional gating hierarchy. The recursive data filtering process in DAFi helped identify small data clusters which are otherwise difficult to resolve by a single run of the data clustering method due to the statistical interference of the irrelevant major clusters. Our experiment results showed that the proportions of the cell populations identified by DAFi, while being consistent with those by expert centralized manual gating, have smaller technical variances across samples than those from individual manual gating analysis and the nonrecursive data clustering analysis. Compared with manual gating segregation, DAFi-identified cell populations avoided the abrupt cut-offs on the boundaries. DAFi has been implemented to be used with multiple data clustering methods including K-means, FLOCK, FlowSOM, and the ClusterR package. For cell population identification, DAFi supports multiple options including clustering, bisecting, slope-based gating, and reversed filtering to meet various autogating needs from different scientific use cases. © 2018 International Society for Advancement of Cytometry.

Missing data imputation, which aims to impute the missing values in the raw datasets, is crucial for modern data-driven models like large language models (LLMs). Despite its importance, existing solutions either 1) only support numerical and categorical data or 2) show an unsatisfactory performance due to their design prioritizing text data and overlooking intrinsic characteristics of tabular data. In this paper, we propose UnIMP, a Un ified IMP utation framework that leverages LLM and high-order message passing to enhance the imputation of mixed-type data, including numerical, categorical, and text data. Specifically, we first introduce a cell-oriented hypergraph to model the table. We then propose BiHMP, an efficient Bidirectional High-order Message-Passing network to aggregate global-local and high-order information while capturing the inter-column heterogeneity and intra-column homogeneity. To align the capacity of the LLM with the information aggregated by BiHMP, we introduce Xfusion, which, together with BiHMP, acts as adapters for the LLM. We follow a pre-training and fine-tuning pipeline to train UnIMP, integrating two optimizations: chunking technique, which divides tables into smaller chunks to enhance efficiency; and progressive masking technique, which gradually adapts the model to learn more complex data patterns. Both theoretical proofs and empirical experiments on 10 real-world datasets highlight the superiority of UnIMP over existing techniques.

<span>Clustering is a significant approach in data mining, which seeks to find groups or clusters of data. Both numeric and categorical features are frequently used to define the data in real-world applications. Several different clustering algorithms are proposed for the numerical and categorical datasets. In clustering algorithms, the quality of clustering results is evaluated using cluster validation. This paper proposes an efficient clustering algorithm for mixed numerical and categorical data using re-clustering and cluster validation. Initially, the mixed dataset is clustered with four traditional clustering algorithms like expectation-maximization (EM), hierarchical cluster (HC), k-means (KM), and self-organizing map (SOM). These four algorithms are validated, and the best algorithm is selected for re-clustering. It is an iterative process for improving the quality of cluster results. The incorrectly clustered data is iteratively re-clustered and evaluated based on the cluster validation. The performance of the proposed clustering method is evaluated with a real-time dataset in terms of purity, normalized mutual information, rand index, precision, and recall. The experimental results have shown that the proposed reclust algorithm achieves better performance compared to other clustering algorithms.</span>

Estimation models in software engineering are used to predict some important attributes of future entities such as software development effort, software reliability and programmers productivity. Among these models, those estimating software effort have motivated considerable research in recent years. Estimation by analogy is one of the attractive techniques in software effort estimation field. However, the procedure used in estimation by analogy is not yet able to handle correctly categorical data. In this paper, we propose a new approach based on reasoning by analogy, fuzzy logic and linguistic quantifiers to estimate effort when the software project is described either by categorical or numerical data. Fuzzy logic-based cost estimation models are more appropriate when vague and imprecise information is to be accounted for. Fuzzy systems try to behave just like the processes of the brain with a rule base. In our proposed method the effort and the cost can be estimated using the Fuzzy logic based analogy method and the cost can be calculated. The performance of the proposed system is calculated using MARE (Mean Absolute Relative Error).

Categorical data are quantified as either nominal variables--distinguishing different groups, for example, based on socio-economic status, education, and political persuasion--or ordinal variables--distinguishing levels of interest, such as the preferred politician or the preferred type of punishment for committing burglary. This new book is a collection of up-to-date studies on modern categorical data analysis methods, emphasizing their application to relevant and interesting data sets.This volume concentrates on latent class analysis and item response theory. These methods use latent variables to explain the relationships among observed categorical variables. Latent class analysis yields the classification of a group of respondents according to their pattern of scores on the categorical variables. This provides insight into the mechanisms producing the data and allows the estimation of factor structures and regression models conditional on the latent class structure. Item response theory leads to the identification of one or more ordinal or interval scales. In psychological and educational testing these scales are used for individual measurement of abilities and personality traits.The focus of this volume is applied. After a method is explained, the potential of the method for analyzing categorical data is illustrated by means of a real data example to show how it can be used effectively for solving a real data problem. These methods are accessible to researchers not trained explicitly in applied statistics. This volume appeals to researchers and advanced students in the social and behavioral sciences, including social, developmental, organizational, clinical and health psychologists, sociologists, educational and marketing researchers, and political scientists. In addition, it is of interest to those who collect data on categorical variables and are faced with the problem of how to analyze such variables--among themselves or in relation to metric variables.