Research on the feature selection techniques used in text classification

Text mining is a special case of data mining which explore unstructured or semi-structured text documents, to establish valuable patterns and rules that indicate trends and significant features about specific topics. Text mining has been in pattern recognition, predictive studies, sentiment analysis and statistical theories in many areas of research, medicine, financial analysis, social life analysis, and business intelligence. Text mining uses concept of natural language processing and machine learning. Machine learning algorithms have been used and reported to give great results, but their performance of machine learning algorithms is affected by factors such as dataset domain, number of classes, length of the corpus, and feature selection techniques used. Redundant attribute affects the performance of the classification algorithm, but this can be reduced by using different feature selection techniques and dimensionality reduction techniques. Feature selection is a data preprocessing step that chooses a subset of input variable while eliminating features with little or no predictive information. Feature selection techniques are Information gain, Term Frequency, Term Frequency-Inverse document frequency, Mutual Information, and Chi-Square, which can use a filters, wrappers, or embedded approaches. To get the most value from machine learning, pairing the best algorithms with the right tools and processes is necessary. Little research has been done on the effect of feature selection techniques on classification accuracy for pairing of these algorithms with the best feature selection techniques for optimal results. In this research, a text classification experiment was conducted using incident management dataset, where incidents were classified into their resolver groups. Support vector machine (SVM), K-Nearest Neighbors (KNN), Naïve Bayes (NB) and Decision tree (DT) machine learning algorithms were examined. Filtering approach was used on the feature selection techniques, with different ranking indices applied for optimal feature set and classification accuracy results analyzed. The classification accuracy results obtained using TF were, 88% for SVM, 70% for NB, 79% for Decision tree, and KNN had 55%, while Boolean registered 90%, 83%, 82% and 75%, for SVM, NB, DT, and KNN respectively. TF-IDF, had 91%, 83%, 76%, and 56% for SVM, NB, DT, and KNN respectively. The results showed that algorithm performance is affected by feature selection technique applied. SVM performed best, followed by DT, KNN and finally NB. In conclusion, presence of noisy data leads to poor learning performance and increases the computational time. The classifiers performed differently depending on the feature selection technique applied. For optimal results, the classifier that performed best together with the feature selection technique with the best feature subset should be applied for all types of data for accurate classification performance. Keywords: Text Classification, Supervised Machine Learning, Feature Selection DOI : 10.7176/JIEA/9-3-06 Publication date :May 31 st 2019

Text classification has been highlighted as the key process to organize online texts for better communication in the Digital Media Age. Text classification establishes classification rules based on text features, so the accuracy of feature selection is the basis of text classification. Facing fast-increasing Chinese electronic documents in the digital environment, scholars have accumulated quite a few algorithms for the feature selection for the automatic classification of Chinese texts in recent years. However, discussion about how to adapt existing feature selection algorithms for various types of Chinese texts is still inadequate. To address this, this study proposes three improved feature selection algorithms and tests their performance on different types of Chinese texts. These include an enhanced CHI square with mutual information (MI) algorithm, which simultaneously introduces word frequency and term adjustment (CHMI); a term frequency–CHI square (TF–CHI) algorithm, which enhances weight calculation; and a term frequency–inverse document frequency (TF–IDF) algorithm enhanced with the extreme gradient boosting (XGBoost) algorithm, which improves the algorithm’s ability of word filtering (TF–XGBoost). This study randomly chooses 3000 texts from six different categories of the Sogou news corpus to obtain the confusion matrix and evaluate the performance of the new algorithms with precision and the F1-score. Experimental comparisons are conducted on support vector machine (SVM) and naive Bayes (NB) classifiers. The experimental results demonstrate that the feature selection algorithms proposed in this paper improve performance across various news corpora, although the best feature selection schemes for each type of corpus are different. Further studies of the application of the improved feature selection methods in other languages and the improvement in classifiers are suggested.

With the growth of Internet, there has been a tremendous increases in the number of attacks and therefore Intrusion Detection Systems (IDS’s) has become a main stream of information security. The purpose of IDS is to help the computer systems to deal with attacks. This anomaly detection system creates a database of normal behaviour and deviations from the normal behaviour to trigger during the occurrence of intrusions. Based on the source of data, IDS is classified into Host based IDS and Network based IDS. In network based IDS, the individual packets flowing through the network are analyzed where as in host based IDS the activities on the single computer or host are analyzed. The feature selection used in IDS helps to reduce the classification time. In this paper, the IDS for detecting the attacks effectively has been proposed and implemented. For this purpose, a new feature selection algorithm called Optimal Feature Selection algorithm based on Information Gain Ratio has been proposed and implemented. This feature selection algorithm selects optimal number of features from KDD Cup dataset. In addition, two classification techniques namely Support Vector Machine and Rule Based Classification have been used for effective classification of the data set. This system is very efficient in detecting DoS attacks and effectively reduces the false alarm rate. The proposed feature selection and classification algorithms enhance the performance of the IDS in detecting the attacks.

A new feature selection based on comprehensive measurement both in inter-category and intra-category for text categorization

Daily physical activity monitoring and recognition have become a big health caring challenge in modern times. Fast recognition of physical activity from wearable sensors dataset with acceptable accuracy has got great research attention. In this paper, we have presented an optimization framework with feature selection techniques. The Bayesian Optimization algorithm has been employed to optimize hyper-parameters of Support Vector Machine(SVM), Random Forest (RF), Extreme Gradient Boosting(XGBoost). Two feature selection algorithms like Fast Correlation Based Filter (FCBF) and Maximum Redundancy Maximum Relevance(mRMR) feature selection have been applied to reduce the size of the extracted feature vector. Classification performances of the two feature selection techniques are compared in terms of accuracy and F1 score. For reduced feature vector the highest acceptable accuracy and F1 score of 95.4% and 94.76% respectively have been achieved by optimized XGBoost classifier with mRMR feature selection. In addition to this, it is shown that reduced computation time with considerable classification performance can be achieved by an efficient feature selection algorithm which is useful for designing a simplified system.

New hybrid data mining model for credit scoring based on feature selection algorithm and ensemble classifiers

Nondestructive Testing (NDT) assessment of materials’ health condition is useful for classifying healthy from unhealthy structures or detecting flaws in metallic or dielectric structures. Performing structural health testing for coated/uncoated metallic or dielectric materials with the same testing equipment requires a testing method that can work on metallics and dielectrics such as microwave testing. Reducing complexity and expenses associated with current diagnostic practices of microwave NDT of structural health requires an effective and intelligent approach based on feature selection and classification techniques of machine learning. Current microwave NDT methods in general based on measuring variation in the S-matrix over the entire operating frequency ranges of the sensors. For instance, assessing the health of metallic structures using a microwave sensor depends on the reflection or/and transmission coefficient measurements as a function of the sweeping frequencies of the operating band. The aim of this work is reducing sweeping frequencies using machine learning feature selection techniques. By treating sweeping frequencies as features, the number of top important features can be identified, then only the most influential features (frequencies) are considered when building the microwave NDT equipment. The proposed method of reducing sweeping frequencies was validated experimentally using a waveguide sensor and a metallic plate with different cracks. Among the investigated feature selection techniques are information gain, gain ratio, relief, chi-squared. The effectiveness of the selected features were validated through performance evaluations of various classification models; namely, Nearest Neighbor, Neural Networks, Random Forest, and Support Vector Machine. Results showed good crack classification accuracy rates after employing feature selection algorithms.

In text classification, feature selection is essential to improve the classification effectiveness. This paper provides an empirical study of a feature selection method based on genetic algorithms for different text representation methods. This feature selection algorithm can accomplish two goals: in one hand is the search of a feature subset such that the performance of classifier is best; in other hands is find a feature subset with the smallest dimensionality which achieves higher accuracy in classification. To evaluate the performance of this approach, three from the best classifiers have been selected: Naive Bayes (NB), Nearest Neighbors (KNN) and Support Vector Machines (SVMs). Our objective is to determine whether the genetic algorithms based feature selection will improve the performances in text classification with smaller size using F-measure. Experimentations were carried out on two benchmark document collections 20Newsgroups, and Reuters-21578. And the results were very interesting.

This paper aims to improve recognition accuracies of single sources of partial discharge (PD) by employing image-oriented feature extraction and selection algorithms. 419 diversified samples of PD data acquired from typically artificial defect models, where the defect size, applied voltage and insulation aging are taken into account, are employed for algorithms validation. In the proposed method, the PD gray images are decomposed into various vectors by two-dimensional principal component analysis (2DPCA) on horizontal and vertical directions, respectively, in which 9 representative parameters are extracted from each image vector. Based on the 419 PD samples, the proposed image features are confirmed to achieve better PD recognition performances than typical image compression methods relevant to 2DPCA. To further improve the recognition performance and reduce the feature dimension, the feature selection technique based on non-dominated sorting genetic algorithm II (NSGA-II) is adopted to search the optimal 2DPCA features. By using the proposed image-oriented feature extraction and selection algorithm, fuzzy k-nearest neighbor classifier (FkNNC), back-propagation neural network (BPNN) and support vector machine (SVM) with 16-dimensional 2DPCA features can now achieve 96.4%, 94.4% and 97.5% accuracies, respectively. The increment of average recognition accuracies of 5%-7% are obtained by different classifiers compared with phase-resolved partial discharge (PRPD) statistical features in previous studies. The investigations on the recognition results relevant to image distortion, cavity discharge change with time and PD contaminated by random noises with different level confirm the robustness of the 16-dimensional 2DPCA features, which signify a great potential in applying the proposed method to field PD pattern recognition in the future.

Text documents are unstructured and high dimensional. Effective feature selection is required to select the most important and significant feature from the sparse feature space. Thus, this paper proposed an embedded feature selection technique based on Term Frequency-Inverse Document Frequency (TF-IDF) and Support Vector Machine-Recursive Feature Elimination (SVM-RFE) for unstructured and high dimensional text classificationhis technique has the ability to measure the feature’s importance in a high-dimensional text document. In addition, it aims to increase the efficiency of the feature selection. Hence, obtaining a promising text classification accuracy. TF-IDF act as a filter approach which measures features importance of the text documents at the first stage. SVM-RFE utilized a backward feature elimination scheme to recursively remove insignificant features from the filtered feature subsets at the second stage. This research executes sets of experiments using a text document retrieved from a benchmark repository comprising a collection of Twitter posts. Pre-processing processes are applied to extract relevant features. After that, the pre-processed features are divided into training and testing datasets. Next, feature selection is implemented on the training dataset by calculating the TF-IDF score for each feature. SVM-RFE is applied for feature ranking as the next feature selection step. Only top-rank features will be selected for text classification using the SVM classifier. Based on the experiments, it shows that the proposed technique able to achieve 98% accuracy that outperformed other existing techniques. In conclusion, the proposed technique able to select the significant features in the unstructured and high dimensional text document.

One of the most significant, intricate, and little-discussed aspects of pressurized irrigation is cost estimation. This study attempts to model the early-stage cost of the drip irrigation system using a database of 515 projects divided into four sections the cost of the pumping station and central control system (TCP), the cost of on-farm equipment (TCF), the cost of installation and operation on-farm and pumping station (TCI), and the total cost (TCT). First, 39 environmental and management features affecting the cost of the listed sectors were extracted for each of the 515 projects previously mentioned. A database (a matrix of 515 × 43) was created, and the costs of all projects were updated for the baseline year of 2022. Then, several feature selection algorithms, such as WCC, LCA, GA, PSO, ACO, ICA, LA, HTS, FOA, DSOS, and CUK, were employed to choose the most significant features that had the biggest influence on the system cost. The selection of features was carried out for all features (a total of 39 features) as well as for easily available features (those features that existed before the irrigation system’s design phase, 18 features). Then, different machine learning models such as Multivariate Linear Regression, Support Vector Regression, Artificial Neural Networks, Gene Expression Programming, Genetic Algorithms, Deep Learning, and Decision Trees, were used to estimate the costs of each of the of the aforementioned sections. Support vector machine (SVM) and optimization algorithms (Wrapper) were found to be the best learner and feature selection techniques, respectively, out of all the available feature selection algorithms. The two LCA and FOA algorithms produced the best estimation, according to the evaluation criteria results. Their RMSE for all features was 0.0020 and 0.0018, respectively, and their R2 was 0.94 and 0.94. For readily available features, these criteria were 0.0006 and 0.95 for both algorithms. In the part of the overall feature, the early-stage cost modeling with selected features revealed that the SVM model (with RBF Kernel) is the best model among the four cost sections discussed. Its evaluation criteria in the training stage are R2 = 0.923, RMSE = 0.008, and VE = 0.082; in the testing stage, they are R2 = 0.893, RMSE = 0.009, and VE = 0.102. The ANN model (MLP) was found to be the best model for a subset of features in the easily available feature part, with R2 = 0.912, RMSE = 0.008, and VE = 0.083 in the training stage and R2 = 0.882, RMSE = 0.009, and VE = 0.103 in the testing stage. The findings of this study can be utilized to highly accurately estimate the cost of local irrigation systems based on the recognized environmental and management parameters and by employing particular models.

Today, video data, as a powerful multimedia component, is accompanied by some problems with increasing usage in communication, health, education, and social media in particular. Classification and detection of concepts in video data by automatic methods are some of these challenging problems. In this study, we propose a video classification system, which incorporates deep convolutional neural networks (CNNs) by leveraging feature selection and data fusion techniques to improve the accuracy of the classification. Principal Component Analysis (PCA) as a feature selection method and Discriminant Correlation Analysis (DCA) technique, which incorporates class associations into the correlation analysis of feature sets for data fusion, are applied to the problem at the feature level. Support Vector Machines (SVMs) have been trained with new feature vectors obtained from different deep convolutional neural networks by feature selection and data fusion methods. The proposed method is tested for 38 concepts on TRECVID 2013 SIN video task dataset and the results are evaluated. Our results show that the classification accuracy is improved by 4% with an accuracy of 50.27% when the proposed data fusion and feature selection techniques are used.

In the present state of affairs, the motive behind every educational organization is to uplift the academic achievement of students. Educational data mining (EDM) is an upward field of research, and it is very helpful for academic institutions to predict the academic performance of the students. Educational datasets are the basis of various predictive models. Quality of these models can be improved by using feature selection (FS). To get the required benefits from the available data, there must be some tools for analysis and prediction. In lieu of the above, machine learning/data mining are most suitable. In educational data mining, for better accuracy of prediction models’ and quality of various educational datasets, feature selection (FS) plays a vital role. Feature selection (FS) algorithms abolish inappropriate information from the repositories of educational background so that performance of classifier in terms of accuracy could be increased and the same could be used for better decision. In lieu of the above, a best feature selection algorithm must be selected. In this paper, two filter selection approaches namely correlation feature selection (CFS) and wrapper-based feature selection have been used to demonstrate the importance of selection of a feature subset for a classification problem. The present paper aims to find the detailed investigation of filter feature selection algorithms along with the classification algorithms on a given dataset. We found result with numerous numbers of features from various Feature selection algorithms and classifiers which will help the researcher to discover the most excellent mixture of filter feature selection algorithms and its associated classifiers. The result indicates that SMO and J48 have the highest accuracy measures with the correlation feature selection algorithms, while Naïve Bayes has the highest accuracy measures with the wrapper subset feature selection algorithms for predicting high, medium and low grade for the students.

One of the important practical applications of computer vision techniques is computer-aided diagnosis (CAD) in medical imaging. It has been shown that CAD can improve the accuracy of breast cancer detection and characterization by radiologists on mammograms. In this chapter, we discuss an important step — feature selection — in classifier design for CAD algorithms. Feature selection reduces the dimensionality of an available feature space and is therefore often used to prevent over-parameterization of a classifier. Many feature selection techniques have been proposed in the literature. We will illustrate the usefulness of genetic algorithms (GAs) for feature selection by comparing GA with a commonly used sequential selection method. A brief introduction to the GA is given and several examples using GA feature selection for the characterization of mammographic lesions are discussed. The examples illustrate the design of a fitness function for optimizing classification accuracy in terms of the receiver operating characteristics of the classifier, the dependence of GA performance on its evolution parameters, and the design of a fitness function tailored to a specific classification task.

Our research developed a non signature based approach, employing feature selection methods such as Categorical Proportional Distance (CPD), Weight of Evidence of Text (WET), Term Frequency - Inverse Document Frequency (TF-IDF), Term Frequency - Inverse Document Frequency - Class Frequency (TF-IDF-CF), Galavotti-Sebastiani-Simi Coefficient (GSS) and Term Significance (TS). Classification model is developed by considering bi--gram features ranked with these feature selection techniques. The proposed feature selection approaches detect unseen malware samples with accuracy in the range of 99% to 100%. Relevance of a feature ranking methods on variable feature length is ascertained using McNemar test.