Feature selection for text classification: A review

In text classification, feature selection is essential to improve the classification effectiveness. This paper provides an empirical study of a feature selection method based on genetic algorithms for different text representation methods. This feature selection algorithm can accomplish two goals: in one hand is the search of a feature subset such that the performance of classifier is best; in other hands is find a feature subset with the smallest dimensionality which achieves higher accuracy in classification. To evaluate the performance of this approach, three from the best classifiers have been selected: Naive Bayes (NB), Nearest Neighbors (KNN) and Support Vector Machines (SVMs). Our objective is to determine whether the genetic algorithms based feature selection will improve the performances in text classification with smaller size using F-measure. Experimentations were carried out on two benchmark document collections 20Newsgroups, and Reuters-21578. And the results were very interesting.

Feature selection is an important task in machine learning. It can improve classification accuracy and effectively reduce the dataset dimensionality by removing no discriminative features. Though a large body of researches were focused on feature selection for text classification, few works addressed the problem for multi-label data in big data context. Therefore, this paper proposes a distributed feature selection approach for multi-label textual big data based on the weighted chi-square method. First, a standard multi-label approach to transform the multi-label data into single-label data is applied. Then, the algorithm assigns different weights to the features based on the category term frequency and then calculates the chi-square based on the weight of each feature. The proposed method is implemented on Hadoop framework using MapReduce programming model. At last, a set of experiments were conducted on three benchmarking text datasets to evaluate the effectiveness of the proposed approach. A comparative analysis of the results with the state-of-the-art techniques proves that our method is efficient, robust and scalable.

This paper presents an improved Gini-Index algorithm to correct feature-selection bias in text classification. Gini-Index has been used as a split measure for choosing the most appropriate splitting attribute in decision tree. Recently, an improved Gini-Index algorithm for feature selection, designed for text categorization and based on Gini-Index theory, was introduced, and it has proved to be better than the other methods. However, we found that the Gini-Index still shows a feature selection bias in text classification, specifically for unbalanced datasets having a huge number of features. The feature selection bias of the Gini-Index in feature selection is shown in three ways: 1) the Gini values of low-frequency features are low (on purity measure) overall, irrespective of the distribution of features among classes, 2) for high-frequency features, the Gini values are always relatively high and 3) for specific features belonging to large classes, the Gini values are relatively lower than those belonging to small classes. Therefore, to correct that bias and improve feature selection in text classification using Gini-Index, we propose an improved Gini-Index (I-GI) algorithm with three reformulated Gini-Index expressions. In the present study, we used global dimensionality reduction (DR) and local DR to measure the goodness of features in feature selections. In experimental results for the I-GI algorithm, we obtained unbiased feature values and eliminated many irrelevant general features while retaining many specific features. Furthermore, we could improve the overall classification performances when we used the local DR method. The total averages of the classification performance were increased by 19.4%, 15.9%, 3.3%, 2.8% and 2.9% (kNN) in Micro-F1, 14%, 9.8%, 9.2%, 3.5% and 4.3% (SVM) in Micro-F1, 20%, 16.9%, 2.8%, 3.6% and 3.1% (kNN) in Macro-F1, 16.3%, 14%, 7.1%, 4.4%, 6.3% (SVM) in Macro-F1, compared with tf*idf, χ2, Information Gain, Odds Ratio and the existing Gini-Index methods according to each classifier.

In this research we apply several machine learning methods and word embedding features to process social media data, specifically comments on the Disney Plus Hotstar application. The word embedding features used include Word2Vec, GloVe, and FastText. Our aim is to evaluate the impact of these features on the classification performance of machine learning methods such as Naive Bayes (NB), K-Nearest Neighbor (KNN), and Random Forest (RF). NB is very simple and efficient and very sensitive to feature selection. Meanwhile, KNN is known for its weaknesses such as biased k values, overly complex computations, memory limitations, and ignoring irrelevant attributes. Then RF has a weakness, namely that the evaluation value can change significantly with just a slight change in the data. Feature selection in text classification is crucial for enhancing scalability, efficiency, and accuracy. Our testing results indicate that KNN achieved the highest accuracy both before and after feature selection. The FastText feature led to the highest performance for KNN, yielding balanced accuracy, precision, recall, and F1-score values.

A well representing feature set that has enough differentiating power plays an important role in classification. The existing techniques for feature set selection are mostly statistical. They only make decision between a feature inclusion or exclusion without taking into account the importance of human thinking and reasoning that surely effect the results of classification and prediction. In this paper, I propose a new method of feature selection and employed it via Fuzzy Set Theory (FST). The effectiveness of proposed method is evaluated in terms of improving the classifier's performance as compared to state of the art methods such as Gini Index, Information Gain, and Document Frequency. A case study is performed with widely known classifiers Naive Bayes, Support Vector Machine and Random Forest in the context of classification of software design patterns. The promising results indicate the applicability of proposed method in context of automated systems of certain domains.

Feature Selection (FS) is the selection of certain features by using a criterion assessment application. In the age of technological advancements and innovations, there has been a significant rise in the use of different social networking applications such as Facebook, YouTube, Twitter, and e-commerce platforms such as Amazon, Alibaba, Flipkart, and others. It has increased the use of sentiment analysis in the different real-world applications so that there is a classification of text in different preset polarities such as optimistic, pessimistic, and impartial. The current research aims to find out the selection process involved in choosing the optimal features for classification using sentimental textual data. The study presents the scope of the sentimental analysis using textual data, fundamentals of the classification process, and the significance of the feature selection schemes so that there is an implementation of effective feature selection for text classification and analysis of sentiments.

Text mining is a special case of data mining which explore unstructured or semi-structured text documents, to establish valuable patterns and rules that indicate trends and significant features about specific topics. Text mining has been in pattern recognition, predictive studies, sentiment analysis and statistical theories in many areas of research, medicine, financial analysis, social life analysis, and business intelligence. Text mining uses concept of natural language processing and machine learning. Machine learning algorithms have been used and reported to give great results, but their performance of machine learning algorithms is affected by factors such as dataset domain, number of classes, length of the corpus, and feature selection techniques used. Redundant attribute affects the performance of the classification algorithm, but this can be reduced by using different feature selection techniques and dimensionality reduction techniques. Feature selection is a data preprocessing step that chooses a subset of input variable while eliminating features with little or no predictive information. Feature selection techniques are Information gain, Term Frequency, Term Frequency-Inverse document frequency, Mutual Information, and Chi-Square, which can use a filters, wrappers, or embedded approaches. To get the most value from machine learning, pairing the best algorithms with the right tools and processes is necessary. Little research has been done on the effect of feature selection techniques on classification accuracy for pairing of these algorithms with the best feature selection techniques for optimal results. In this research, a text classification experiment was conducted using incident management dataset, where incidents were classified into their resolver groups. Support vector machine (SVM), K-Nearest Neighbors (KNN), Naïve Bayes (NB) and Decision tree (DT) machine learning algorithms were examined. Filtering approach was used on the feature selection techniques, with different ranking indices applied for optimal feature set and classification accuracy results analyzed. The classification accuracy results obtained using TF were, 88% for SVM, 70% for NB, 79% for Decision tree, and KNN had 55%, while Boolean registered 90%, 83%, 82% and 75%, for SVM, NB, DT, and KNN respectively. TF-IDF, had 91%, 83%, 76%, and 56% for SVM, NB, DT, and KNN respectively. The results showed that algorithm performance is affected by feature selection technique applied. SVM performed best, followed by DT, KNN and finally NB. In conclusion, presence of noisy data leads to poor learning performance and increases the computational time. The classifiers performed differently depending on the feature selection technique applied. For optimal results, the classifier that performed best together with the feature selection technique with the best feature subset should be applied for all types of data for accurate classification performance. Keywords: Text Classification, Supervised Machine Learning, Feature Selection DOI : 10.7176/JIEA/9-3-06 Publication date :May 31 st 2019

Text classification has been highlighted as the key process to organize online texts for better communication in the Digital Media Age. Text classification establishes classification rules based on text features, so the accuracy of feature selection is the basis of text classification. Facing fast-increasing Chinese electronic documents in the digital environment, scholars have accumulated quite a few algorithms for the feature selection for the automatic classification of Chinese texts in recent years. However, discussion about how to adapt existing feature selection algorithms for various types of Chinese texts is still inadequate. To address this, this study proposes three improved feature selection algorithms and tests their performance on different types of Chinese texts. These include an enhanced CHI square with mutual information (MI) algorithm, which simultaneously introduces word frequency and term adjustment (CHMI); a term frequency–CHI square (TF–CHI) algorithm, which enhances weight calculation; and a term frequency–inverse document frequency (TF–IDF) algorithm enhanced with the extreme gradient boosting (XGBoost) algorithm, which improves the algorithm’s ability of word filtering (TF–XGBoost). This study randomly chooses 3000 texts from six different categories of the Sogou news corpus to obtain the confusion matrix and evaluate the performance of the new algorithms with precision and the F1-score. Experimental comparisons are conducted on support vector machine (SVM) and naive Bayes (NB) classifiers. The experimental results demonstrate that the feature selection algorithms proposed in this paper improve performance across various news corpora, although the best feature selection schemes for each type of corpus are different. Further studies of the application of the improved feature selection methods in other languages and the improvement in classifiers are suggested.

The use of ensemble paradigm with classifiers is a proven approach that involves combining the outcomes of several classifiers. It has recently been extrapolated to feature selection methods to find the most relevant features. Earlier, ensemble feature selection has been used in high dimensional, low sample size datasets like bioinformatics. To one's knowledge there is no such endeavor in the text classification domain. In this work, the ensemble feature selection using data perturbation in the text classification domain has been used with an aim to enhance predictability and stability. This approach involves application of the same feature selector to different perturbed versions of training data, obtaining different ranks for a feature. Previous works focus only on one of the metrics, that is, stability or accuracy. In this work, a combined framework is adopted that assesses both the predictability and stability of the feature selection method by using feature selection ensemble. This approach has been explored on univariate and multivariate feature selectors, using two rank aggregators.

Weighting formula and feature selection are key preprocessing in text classifying and mining. We analyze the drawbacks of weighting formula based on inverse document frequency and present a novel feature weighting and selecting method based on variable precision rough set model. Inverse document frequency (IDF) doesn't take the classification information into account and the criterion based on IDF is not monotonous with the contribution that a feature makes to classification, which decreases the classifier's performance. The measure of classification quality based on variable rough set model can deal with complex classification. It measures the contribution a feature makes to classification. It is introduced as a criterion for feature selecting and weighting in text classification. We name it as TFACQ. The experimental results show that the weighting formula and feature selection based on TFACQ have greatly improved the performance.

This project focuses on using machine learning to predict a patient’s risk of developing diabetes based on a test. This database, compiled by the National Institute of Diabetes and Digestive and Kidney Diseases, contains health indicators for Pima Indian patients. The project involves several key steps: initial data, selecting a feature, selecting a model, updating the hyperparameter, and deploying it via the Flask web application. In the data preprocessing phase, feature scaling and normalization are used to standardize the dataset, while missing values and outliers are handled to ensure data integrity. Feature selection uses correlation matrix and recursive feature elimination (RFE) to reduce dimensionality and improve model efficiency. To ensure the model is optimized for latent data, the dataset is split into two parts: 66% for training and 34% for testing. Various machine learning algorithms are evaluated, including logistic regression, naive Bayes, K-nearest neighbours, decision trees, and support vector classifiers. Logistic regression was selected as the final model due to its accuracy on the test data (80.53%). The model uses grid search for hyperparameter tuning to improve its performance. The training model is embedded in the Flask web application, allowing users to access health metrics and get real-time estimates of blood pressure. The system is designed to be user-friendly and scalable, providing a practical tool for early diagnosis of diabetes. All methods ensure that the model is accurate, reliable, and capable of making real-world predictions. Keywords: prediction, diabetes, glucoses, insulin, machine learning, logistic regression, naive bayes, k- nearest neighbours, decision tree, support vector classifier.

Text classification is a classical problem of natural language processing where given texts are classified into relevant classes. Text classification techniques are extensively used in finding news categories, search engine optimization, automated textual response and many more. With the exponential growth of digital data, it is inevitable that an automatic classification of documents is needed for efficient information retrieval and document archival. Over the years, text classification problems are approached by using classical machine learning techniques like naive bayes(NB), support vector machine(SVM), logistic regression(LR), random forest(RF) and others. In this article, an Attention based deep learning (DL) model is proposed and applied to the news corpus and then the results are compared with the results of classical machine learning models. A comparative analysis of classification accuracy of the DL model with prominent models like LR, RF and NB are presented in this work.

Due to the existence of text data, multi-label (ML) text classification is an essential task in machine learning. Feature selection is an essential and effective preprocess to enhance the learning process. Choosing a Multi-Label Feature Selection (MLFS) algorithm is the most basic, critical, and sensitive choice in ML classification operations. If this choice is based on a criterion, it cannot be attributed to always being sound. Choosing the best algorithm must be evaluated using several different criteria to be examined from different aspects. In this article, we turn the issue into an election and use the Weighted Borda Count method for voting. We do the voting in three stages continuously so that a subset of different features does the voting. In the second stage, voting of different methods is done with six criteria, and each criterion selects the methods in order of priority from the beginning to the end. Voting steps 1 and 2 are performed on eighteen text datasets used. Finally, in the final voting stage, the methods are evaluated and voted on by different text datasets. The final result of the voting in the third stage shows the desired MLFS methods based on their performance from beginning to end. According to the experiments performed and the results obtained, it can be seen that the selection of the algorithm based on several different criteria and considering the overall performance of the algorithm will be better than the selection based on one criterion.

Research on feature selection in text classification is usually limited to propose various techniques to select a set of features with highest scores based on different metrics. The selected features are usually determined by using a separate validation dataset with a fixed threshold. Obviously, it may not generalize well to new data as the best number for selected features is various on different datasets. In this paper, we first conduct a deep analysis, and find that simply extracting the features based on the score calculated by a metric may not always be the best strategy as it may turn many documents into zero length, which make them not suitable for training. We then model the feature selection process as a multiple objectives optimization problem to gain the best number of selected features rationally and automatically. In addition, as the optimization process costs a lot of resources, we design a parallel algorithm to improve the running time using dynamic programming. Extensive experiments are performed on several popular datasets, and the results indicate that our proposed approach is effective and feasible.

In this paper, a simple and efficient symbolic text classification is presented. We propose a new method of representing documents based on clustering of term frequency vectors. For each class of documents we propose to create multiple clusters to preserve the intraclass variations. Term frequency vectors of each cluster are used to form a symbolic representation by the use of interval valued features. Subsequently, a new feature selection method based on a new dissimilarity measure is also presented. The new feature selection method reduces the features in the representation phase for effective text classification. It keeps the best features for effective representation and simultaneously reduces the time taken to classify a given document. To corroborate the efficacy of the proposed model we conducted experimentation on various datasets. Experimental results reveal that the proposed method gives better results when compared to the state of the art techniques. In addition, as the method is based on a simple matching scheme, it requires a negligible time.