Feature Selection with β-Hill Climbing Search for Text Clustering Application

Text clustering is suitable for dividing many text documents into distinct groups. The size of the documents has an impact on the performance of text clustering, reducing its effectiveness. Text documents often include sparse and uninformative characteristics, which can negatively impact the efficiency of the text clustering technique and increase the computational time required. Feature selection is a crucial strategy in unsupervised learning that involves choosing a subset of informative text features to enhance the efficiency of text clustering and decrease computing time. This work presents a novel approach based on an improved Prairie dog algorithm to solve the feature selection problem. K-means clustering is employed to assess the efficacy of the acquired subgroups of features. The proposed algorithm is being compared to other algorithms published in the literature. The feature selection strategy ultimately promotes the clustering algorithm to get precise clusters.

Clustering consists of various applications on machine learning, image segmentation, data mining and pattern recognition. The proper selection of clustering is significant in feature selection. Therefore, in this paper, a Text Feature Selection (FS) and Clustering using Grasshopper–Chameleon Swarm Optimization with Density Peaks Clustering algorithm (TFSC-G-CSOA-DPCA) is proposed. Initially, the input features are pre-processed for converting text into numerical form. These preprocessed text features are given to Grasshopper–Chameleon Swarm Optimization Algorithm, which selects important text features. In Grasshopper–Chameleon Swarm Optimization Algorithm, the Grasshopper Optimization Algorithm selects local feature from text document and Chameleon Swarm Optimization Algorithm selects the best global feature from local feature. These important features are tested using density peaks clustering algorithm to maximize the reliability and minimize the computational time cost. The performance of Grasshopper–Chameleon Swarm Optimization Algorithm is analyzed with 20 News groups dataset. Moreover, the performance metrics, like accuracy, precision, sensitivity, specificity, execution time and memory usage are analyzed. The simulation process shows that the proposed TFSC-G-CSOA-DPCA method provides better accuracy of 97.36%, 95.14%, 94.67% and 91.91% and maximum sensitivity of 96.25%, 87.25%, 93.96% and 92.59% compared to the existing methods such as TFSC-BBA-MCL, TFSC-MVO-K-Means C, TFSC-GWO-GOA-FCM and TFSC-WM-K-Means C, respectively.

Feature selection (FS) for multi-label text classification faces issues such as high dimensionality, strong label correlations, and sparse features, which often lead to suboptimal feature subsets. Moreover, most existing methods are centralized and thus ill-suited to real-world distributed or federated settings, where text data are scattered across multiple nodes and effective FS mechanisms are lacking. To overcome these issues, this paper proposes Fed-MSMCGWO, a federated multi-label text feature selection method based on manifold-aware sparse modeling and cooperative grey wolf optimization. Under a federated learning framework, Fed-MSMCGWO integrates manifold-aware sparse modeling (MSM), and incorporates a cooperative grey wolf optimization algorithm (CGWO) to enable multi-label text FS in distributed environments. On each client, Fed-MSMCGWO employs a two-stage optimization. In Stage 1, MSM is learned by constructing sample and label graphs from text embeddings, encoding their manifolds with Laplacians, and imposing a [Formula: see text]-norm on the feature-weight matrix to induce row sparsity and compress high-dimensional features. In Stage 2, CGWO with a three-line cooperative evolution scheme further refines these weights and conducts global search for a near-optimal subset of text features. After the two-stage optimization, each client obtains a locally optimal feature subset and engages in a multi-party privacy-preserving feature aggregation strategy: clients upload only intermediate feature-weight parameters (no raw data) to the server, which aggregates them and sends the result back to guide further local updates, yielding a collaborative cross-client FS framework with preserved privacy. Experiments on several publicly available multi-label text datasets indicate that, with privacy preserved, Fed-MSMCGWO consistently surpasses standard centralized and federated FS methods on multiple evaluation metrics.

In many pattern recognition applications feature selection and instance selection can be used as two data preprocessing methods that aim at reducing the computational cost of the learning process. Moreover, in some cases, feature subset selection can improve the classification performance. Feature selection and instance selection can be interesting since the choice of features and instances greatly influence the performance of the learnt models as well as their training costs. In the past, unifying both problems was carried out by solving a global optimization problem using meta-heuristics. This paradigm not only does not exploit the manifold structure of data but can be computationally expensive. To the best of our knowledge, the joint use of sparse modeling representative and feature subset relevance have not been exploited by the joint feature and selection methods. In this paper, we target the joint feature and instance selection by adopting feature subset relevance and sparse modeling representative selection. More precisely, we propose three schemes for the joint feature and instance selection. The first is a wrapper technique while the two remaining ones are filter approaches. In the filter approaches, the search process adopts a genetic algorithm in which the evaluation is mainly given by a score that quantify the goodness of the features and instances. An efficient instance selection technique is used and integrated in the search process in order to adapt the instances to the candidate feature subset. We evaluate the performance of the proposed schemes using image classification where classifiers are the nearest neighbor classifier and support vector machine classifier. The study is conducted on five public image datasets. These experiments show the superiority of the proposed schemes over various baselines. The results confirm that the filter approaches leads to promising improvement on classification accuracy when both feature selection and instance selection are adopted.

The classification performance of previous IG algorithm may decline obviously because of the maldistribution of classes and features, due to which an improved text feature selection method UDsIG is proposed. First, we select features by classes to reduce the impact on feature selection when the classes are unevenly distributed. After that, we use feature equilibrium of distribution to decrease the interference with feature selection when features are unevenly distributed. And then we deal with class features by feature relation tree model, thus to retain strong correlation features. Finally, we use the improved information gain formula, which is based on weighed dispersion, to get the optimal feature subset. The experimental results show the proposed method has better classification performance.

In view of the advantages of deep learning model in the extraction of abstract concept, a new text clustering algorithm is designed based on Deep Boltzmann Machines. Based on Replicate Softmax Model and new Deep Boltzmann Machine, energy function of this model is proposed and the detail learning algorithm is introduced.

Random forests (RFs) have been widely used as a powerful classification method. However, with the randomization in both bagging samples and feature selection, the trees in the forest tend to select uninformative features for node splitting. This makes RFs have poor accuracy when working with high-dimensional data. Besides that, RFs have bias in the feature selection process where multivalued features are favored. Aiming at debiasing feature selection in RFs, we propose a new RF algorithm, called xRF, to select good features in learning RFs for high-dimensional data. We first remove the uninformative features using p-value assessment, and the subset of unbiased features is then selected based on some statistical measures. This feature subset is then partitioned into two subsets. A feature weighting sampling technique is used to sample features from these two subsets for building trees. This approach enables one to generate more accurate trees, while allowing one to reduce dimensionality and the amount of data needed for learning RFs. An extensive set of experiments has been conducted on 47 high-dimensional real-world datasets including image datasets. The experimental results have shown that RFs with the proposed approach outperformed the existing random forests in increasing the accuracy and the AUC measures.

Objectives: To develop a machine learning-based model to select the most important features from a high-dimensional dataset to classify the patterns at high accuracy and reduce their dimensionality. Methods: The proposed feature selection method (FSIFC) forms and combines feature clusters incrementally and produces feature subsets each time. The method uses K-means clustering and Mutual Information (MI) to refine the feature selection process iteratively. Initially, two clusters of features are formed using K-means clustering (K=2) by taking features as the basis of clustering instead of taking the patterns (a traditional way). From these two clusters, the features with the highest MI value in each cluster are kept in a feature subset. Classification accuracies (CA) of the feature subset are calculated using three classifiers namely Support Vector Machines (SVM), Random Forest (RF), and k-nearest Neighbor (knn). The process is repeated by incrementing the value of K i.e. number of clusters; until a maximum user-defined value of K is reached. The best value of CA obtained from these trials is recorded and the corresponding feature set is finally accepted. Findings: The proposed method is demonstrated using ten datasets and the results are compared with the existing published results using three classifiers to determine the method's performance. The ten datasets are classified with average CAs of 92.72%, 93.13%, and 91.5%, using the SVM, RF, and K-NN classifiers respectively. The proposed method selects a maximum of thirty features from the datasets. In terms of selecting the most effective and the smallest feature sets, the proposed method outperforms eight other feature selection methods considering CAs. Novelty: The proposed model applies feature reduction using combined feature clustering and filter methods in an incremental way. This provides an improved selection of relevant features while removing those which are irrelevant at different trials. Keywords: Feature selection, High-dimensional datasets, K-means algorithm, Mutual information, Machine learning

Feature selection entails identifying a subset of the most useful features that produces compatible results as the original entire set of features. A feature selection algorithm can be assessed in terms of both efficiency and effectiveness. While efficiency is concerned with the time required to find a subset of features, effectiveness is concerned with the quality of the subset of features. This paper proposes and experimentally evaluates a fast clustering-based feature selection algorithm, FAST, based on these criteria.  The FAST algorithm operates in two steps. Graph-theoretic clustering methods are used to partition characteristics into clusters in the initial stage. The most representative feature from each cluster that is strongly related to target classes is chosen in the second stage to construct a subset of features. Because the properties in various clusters are relatively independent, FAST's clustering-based technique is likely to produce a subset of valuable and independent features. We use the efficient Minimum-spanning tree clustering method to ensure FAST's efficiency. An empirical study is conducted to assess the efficiency and effectiveness of the FAST algorithm. FAST and several representative feature selection algorithms, such as FCBF, ReliefF, CFS, Consist, and FOCUS-SF, are compared to four types of well-known classifiers, including the probability-based Naive Bayes, the tree-based C4.5, the instance-based IB1, and the rule-based RIPPER, before and after feature selection. FAST not only provides smaller subsets of features but also improves the performances of the four types of classifiers, according to the findings, which were based on 35 publicly accessible real-world high-dimensional image, microarray, and text data.

Feature selection is one of the most significant procedures in machine learning algorithms. It is particularly to improve the performance and prediction accuracy for complex data classification. This paper discusses a hybrid feature selection technique with the decision tree-based classification algorithm. The feature selected using information gain (IG) is combined with the features selected from ReliefF which generates the resultant feature subset. Then the resultant feature subset is in turn combined with a correlation-based feature selection (CFS) method to generate the aggregated feature subset. To perform classification accuracy on the aggregated feature subset, different decision trees-based classification algorithm such as C4.5, decision stumps, naive Bayes tree, and random forest with ten-fold cross-validation have been deployed. To check the prediction accuracy of the proposed work eight different multilevel University of California, Irvine (UCI) machine learning datasets have been used with minimum to maximum numbers of features. The main objective of the hybrid feature selection is to improve the classification accuracy, prediction and to reduce the execution time using standard datasets.

Text Clustering consists of grouping objects of similar categories. The initial centroids influence operation of the system with the potential to become trapped in local optima. The second issue pertains to the impact of a huge number of features on the determination of optimal initial centroids. The problem of dimensionality may be reduced by feature selection. Therefore, Wind Driven Optimization (WDO) was employed as Feature Selection to reduce the unimportant words from the text. In addition, the current study has integrated a novel clustering optimization technique called the WDO (Wasp Swarm Optimization) to effectively determine the most suitable initial centroids. The result showed the new meta-heuristic which is WDO was employed as the multi-objective first time as unsupervised Feature Selection (WDOFS) and the second time as a Clustering algorithm (WDOC). For example, the WDOC outperformed Harmony Search and Particle Swarm in terms of F-measurement by 93.3%; in contrast, text clustering's performance improves 0.9% because of using suggested clustering on the proposed feature selection. With WDOFS more than 50 percent of features have been removed from the other examination of features. The best result got the multi-objectives with F-measurement 98.3%.

The tax gives an important role for the contributions of the economy and development of a country. The improvements to the taxation service system continuously done in order to increase the State Budget. One of consideration to know the performance of taxation particularly in Indonesia is to know the public opinion as for the object service. Text mining can be used to know public opinion about the tax system. The rapid growth of data in social media initiates this research to use the data source as big data analysis. The dataset used is derived from Facebook and Twitter as a source of data in processing tax comments. The results of opinions in the form of public sentiment in part of service, website system, and news can be used as consideration to improve the quality of tax services. In this research, text mining is done through the phases of text processing, feature selection and classification with Support Vector Machine (SVM). To reduce the problem of the number of attributes on the dataset in classifying text, Feature Selection used the Information Gain to select the relevant terms to the tax topic. Testing is used to measure the performance level of SVM with Feature Selection from two data sources. Performance measured using the parameters of precision, recall, and F-measure.

• Neural Network feature selection methodology for both numerical and categorical data. • Embedded feature selection provides model and feature subset in a single training. • Novel feature embedding for dealing with high-cardinality categorical features. • Validation of the methodology on several open and real industry dataset. • Comparison with several state-of-the-art feature selection methodologie. In an era of effortless data collection, the impact of machine learning — especially neural networks (NNs) — is undeniable. As datasets grow in size and complexity, efficiently handling mixed data types, including categorical and numerical features, becomes critical. Feature encoding and selection play a key role in improving NN performance, efficiency, interpretability, and generalisation. This paper presents GLEm-Net (Grouped Lasso with Embeddings Network), a novel NN-based approach that seamlessly integrates feature encoding and selection directly into the training process. GLEm-Net uses embedding layers to process categorical features with high cardinality, simplifying the model and improving generalisation. By extending the grouped Lasso regularisation to explicitly consider categorical features, GLEm-Net automatically identifies the most relevant features during training and returns them to the analyst. We evaluate GLEm-Net on open and proprietary industry datasets and compare it to state-of-the-art feature selection methodologies. Results show that GLEm-Net adapts to each dataset by allowing the NN to directly select subsets of most important features, offering on par performance with the best state-of-the-art feature selection methods, while eliminating the need for the external feature encoding and selection steps that are now incorporated in the NN training stage.

The spam filter based on Naive Bayes algorithm, which has good classification accuracy, but the training and learning mail sample sets takes a lot of resources, affects the overall efficiency of the system, so we should select the features of the message text in the practical application, and thus to reduce the dimension of the features vector space. TF-IDF is commonly used as a text feature selection, the method is simple, the paper improve the IDF weighting algorithm of the TF-IDF feature selection, increase the weight of the high frequency words corresponding its class, use the improved TF-IDF algorithm to select the features, and build a naive Bayesian spam filter improved TF-IDF feature weighting.

Large-scale feature selection is one of the most important fields in the big data domain that can solve real data problems, such as bioinformatics, where it is necessary to process huge amount of data. The efficiency of existing feature selection algorithms significantly downgrades, if not totally inapplicable, when data size exceeds hundreds of gigabytes, because most feature selection algorithms are designed for centralized computing architecture. For that, distributed computing techniques, such as MapReduce can be applied to handle very large data. Our approach is to scale the existing method for feature selection, Kmeans clustering and Signal to Noise Ratio (SNR) combined with optimization technique as Binary Particle Swarm Optimization (BPSO). The proposed method is divided into two stages. In the first stage, we have used parallel Kmeans on MapReduce for clustering features, and then we have applied iterative MapReduce that implement parallel SNR ranking for each cluster. After, we have selected the top ranked feature from each cluster. The top scored features from each cluster are gathered and a new feature subset is generated. In the second stage, the new feature subset is used as input to the proposed BPSO based on MapReduce which provides an optimized feature subset. The proposed method is implemented in a distributed environment, and its efficiency is illustrated through analyzing practical problems such as biomarker discovery.