Individual Optimal Feature Selection Based on Comprehensive Evaluation Indexs

Identification of voice disorders has been a vital role in our life nowadays. Acoustic analysis can be useful tool to diagnose voice disorders as a complementary technique to other medicine methods such as Laryngoscopy and Stroboscopy. In this paper, we scrutinized feature reduction techniques such as principal component analysis (PCA) and linear discriminant analysis (LDA) as feature subset extraction methods and individual feature selection (IFS), forward feature selection (FFS), backward feature selection (BFS) and branch and bound feature selection (BBFS) as feature subset selection procedures. Performance of each method is evaluated by different classifiers. Between feature selection methods, individual feature selection followed by SVM classifier shows the best recognition rate of 91.55% and AUC of 95.80% among these methods. The experimental results demonstrated that highest performance could be achieved by recognition rate of 94.26% and AUC of 97.94% using linear discriminant analysis along with support vector machine as a classifier. Also this mixture has lowest order of computational complexity in comparison with other architectures.

On the benefit of feature selection and ensemble feature selection for fuzzy k-nearest neighbor classification

In network security, intrusion detection plays an important role. Feature subsets obtained by different feature selection methods will lead to different accuracy of intrusion detection. Using individual feature selection method can be unstable in different intrusion detection scenarios. In this paper, the idea of ensemble is applied to feature selection to adjust feature subsets. Feature selection is converted into a two-category problem, and odd number of feature selection methods is used for voting method to decide whether a feature is required or discarded. In actual operation, mean decrease impurity, random forest classifier, stability selection, recursive feature elimination and chi-square test are used. Feature subsets obtained from them will be adjusted by our proposed method to get ensemble feature subsets. To test the performance, support vector machine, decision tree, knn and multi-layer perception are used to observe and compare the classification accuracy with ensemble feature subsets. Three intrusion detection data sets, including kddcup99, cidds-001 and unsw_nb15 are used in our experiments. The best result is reflected on cidds-001 with a 99.40% classification accuracy. The investigation shows that our method has a certain improvement on classification accuracy of intrusion detection.

The vast volume of redundant and irrelevant network traffic data poses significant hurdles for intrusion detection. Effective feature selection is crucial for eliminating irrelevant information. Presently, most filtering and embedded methods rely on fixed thresholds or ratios, necessitating prior knowledge. Conversely, wrapper methods are computationally intensive, and individual feature selection methods may introduce biases in evaluation. To address these challenges, this study introduces Adaptive Neighborhood based Feature Selection (AN-SFS), a dynamic feature selection approach that adapts to local statistical properties of the data. Unlike traditional methods, AN-SFS adjusts its threshold based on the characteristics of the current feature subset and incorporates statistical measures of neighboring features, capturing subtle relationships and dependencies. This adaptability enables AN-SFS to achieve robust and effective feature selection outcomes. Using NSL-KDD and UNSW-NB15 datasets, our model demonstrates superiority over conventional ML classifiers in detection rate, precision, and recall, achieving outstanding accuracy rates of 99.3% on NSL-KDD and 97.5% on UNSW-NB15, significantly outperforming contemporary methods. To further demonstrate the effectiveness of our feature selection approach, we conducted a series of comparisons with other statistical methods, including ANOVA, Pearson correlation, and the chi-square test. In each comparison, our approach consistently outperformed these alternatives, underscoring its superior efficacy.

The paper proposes a hybrid approach to reduce dimension in text classification problems, to overcome the issue of Curse of Dimensionality. This hybrid approach is a combination of Feature Selection (FS) and Feature Extraction (FE) methods, considering different aspects of feature relevance, to effectively reduce the dimension in large text datasets. It prevents feature selection biased in favor of a particular FS method. Many FS methods like Term Variance, Document Frequency, Information Gain, Shannons Entropy measure, Mean-Median and Mean Absolute Difference, were implemented and a comparative study was made on their performance when implemented in a hybrid system. The features selected by the individual FS methods are merged using three approaches, namely, Union, Intersection and Modified Union. The sub lists of features further undergo Feature Extraction by PCA, and the reduced feature sub list is clustered with k-means. Finally, the sentiment-score of the individual clusters are calculated using SentiWordNet database which gives the polarity of the data. The experiments were conducted on the benchmark datasets namely Reuters-21,578 and Classic4. The performance evaluation of the system made using the measures like precision, recall, f-score and accuracy shows that the proposed method has improved performance compared to its competitive methods.

Feature engineering is treated as a crucial step in any intelligent system pipeline and practitioners often spend 70–80% of their time in this phase before modeling. Especially, in machine learning (ML), the performance of the classification systems deteriorates as irrelevant features are added, even when the features presented contain enough information about the problem. ML researchers often use feature selection and feature extraction techniques individually to filter out the redundant information from the data, but would not get desired results or performance due to choosing a single method only. In this article, we take an attempt to combine both feature selection and feature extraction methods that give an improvement in the prediction performance than using a single feature selection or feature extraction method. In the application of the proposed method, we consider the Indian liver disease patient dataset to classify liver disease using machine learning classifiers. In addition, ensemble classifiers also take into account reproducing and robustifying classification accuracy. The performance is evaluated and compared with different metrics such as accuracy, precision, recall, and F1-score. Using the proposed method, the KNN classifier can diagnosis 76.03% with a precision of 76.69%, recall 96.23%, and 85.36% F1-score which is much better than the individual feature selection and feature extraction approach. The proposed method can also be extended to medical datasets other than liver disease so that these types of deadly diseases can be correctly diagnosed for the betterment of humanity.

In credit risk evaluation the accuracy of a classifier is very significant for classifying the high-risk loan applicants correctly. Feature selection is one way of improving the accuracy of a classifier. It provides the classifier with important and relevant features for model development. This study uses the ensemble of multiple feature ranking techniques for feature selection of credit data. It uses five individual rank based feature selection methods. It proposes a novel rank aggregation algorithm for combining the ranks of the individual feature selection methods of the ensemble. This algorithm uses the rank order along with the rank score of the features in the ranked list of each feature selection method for rank aggregation. The ensemble of multiple feature selection techniques uses the novel rank aggregation algorithm and selects the relevant features using the 80%, 60%, 40% and 20% thresholds from the top of the aggregated ranked list for building the C4.5, MLP, C4.5 based Bagging and MLP based Bagging models. It was observed that the performance of models using the ensemble of multiple feature selection techniques is better than the performance of 5 individual rank based feature selection methods. The average performance of all the models was observed as best for the ensemble of feature selection techniques at 60% threshold. Also, the bagging based models outperformed the individual models most significantly for the 60% threshold. This increase in performance is more significant from the fact that the number of features were reduced by 40% for building the highest performing models. This reduces the data dimensions and hence the overall data size phenomenally for model building. The use of the ensemble of feature selection techniques using the novel aggregation algorithm provided more accurate models which are simpler, faster and easy to interpret.

Ensemble feature selection algorithms aggregate the results of multiple feature selection methods in order to select an effective subset of features. However, typically, ensemble algorithms treat each feature selection method equally and do not consider performance differences. Consequently, features selected by a relatively smaller number of methods may not be included. To address this problem, we propose an interactive feature selection method that can more effectively aggregate the results of multiple feature selection methods and iteratively improve the selected features by integrating expert knowledge. The proposed method includes a learning-from-crowds-based ensemble feature selection algorithm and a visual analysis system. The algorithm models the performance of multiple feature selection methods, calculates their reliabilities, and aggregates results. To integrate expert knowledge, the visual analysis system provides a set of ranking schemes to assist experts in understanding the results of an individual feature selection method and the roles played by the features in classification tasks. A numerical experiment conducted on four real-world datasets shows that the proposed algorithm can improve classification accuracy by 0.63%–2.85% compared to state-of-the-art ensemble feature selection algorithms. In addition, we conducted case studies on text and image data to demonstrate that the proposed visual analysis system can further improve classification accuracy by 0.28%–5.24%.

Stock market prediction is a very noisy problem and the use of any additional information to increase accuracy is necessary. In this paper, for the stock daily return prediction problem, the set of features is expanded to include indicators not only for the stock to be predicted itself but also a set of other stocks and currencies. Afterwards, different feature selection and classification methods are utilized for prediction. The daily close returns of the 3 most traded stocks (GARAN, THYAO, and ISCTR) in Borsa Istanbul (BIST) are predicted using indicators computed on those stocks, indicators for all the other stocks listed in the BIST100 index, and indicators on the dollar-gold prices. Twenty-five different indicators on daily stock prices are computed to form feature vectors for each trading day. These feature vectors are assigned class labels according to the daily close returns. Expanding the feature space with BIST100 stocks features results in a high dimensional feature space, with possibly noisy or irrelevant features. Therefore, feature selection methods are utilized to select the most informative features. In order to determine relevance scores of features, fast filter-based methods, gain ratio and relief, are used. Experiments are performed based on individual stock features, dollar-gold features (DG), BIST100 stock features (BIST100), and a combination of BIST100 and DG with and without feature selection. Using the gain ratio feature selection with a gradient boosting machine (GBM), the movements of GARAN stock were predicted with an accuracy of 0.599 and an F-measure of 0.614. For THYAO, the relief feature selection with the GBM gave an accuracy of 0.558, and for ISCTR, the gain ratio feature selection with logistic regression achieved an accuracy of 0.581. It was found that using BIST100 stock features boosts classification results for all stocks in terms of accuracy.

• Neural Network feature selection methodology for both numerical and categorical data. • Embedded feature selection provides model and feature subset in a single training. • Novel feature embedding for dealing with high-cardinality categorical features. • Validation of the methodology on several open and real industry dataset. • Comparison with several state-of-the-art feature selection methodologie. In an era of effortless data collection, the impact of machine learning — especially neural networks (NNs) — is undeniable. As datasets grow in size and complexity, efficiently handling mixed data types, including categorical and numerical features, becomes critical. Feature encoding and selection play a key role in improving NN performance, efficiency, interpretability, and generalisation. This paper presents GLEm-Net (Grouped Lasso with Embeddings Network), a novel NN-based approach that seamlessly integrates feature encoding and selection directly into the training process. GLEm-Net uses embedding layers to process categorical features with high cardinality, simplifying the model and improving generalisation. By extending the grouped Lasso regularisation to explicitly consider categorical features, GLEm-Net automatically identifies the most relevant features during training and returns them to the analyst. We evaluate GLEm-Net on open and proprietary industry datasets and compare it to state-of-the-art feature selection methodologies. Results show that GLEm-Net adapts to each dataset by allowing the NN to directly select subsets of most important features, offering on par performance with the best state-of-the-art feature selection methods, while eliminating the need for the external feature encoding and selection steps that are now incorporated in the NN training stage.

With the rapid development of artificial intelligence in recent years, the research on image processing, text mining, and genome informatics has gradually deepened, and the mining of large-scale databases has begun to receive more and more attention. The objects of data mining have also become more complex, and the data dimensions of mining objects have become higher and higher. Compared with the ultra-high data dimensions, the number of samples available for analysis is too small, resulting in the production of high-dimensional small sample data. High-dimensional small sample data will bring serious dimensional disasters to the mining process. Through feature selection, redundancy and noise features in high-dimensional small sample data can be effectively eliminated, avoiding dimensional disasters and improving the actual efficiency of mining algorithms. However, the existing feature selection methods emphasize the classification or clustering performance of the feature selection results and ignore the stability of the feature selection results, which will lead to unstable feature selection results, and it is difficult to obtain real and understandable features. Based on the traditional feature selection method, this paper proposes an ensemble feature selection method, Random Bits Forest Recursive Clustering Eliminate (RBF-RCE) feature selection method, combined with multiple sets of basic classifiers to carry out parallel learning and screen out the best feature classification results, optimizes the classification performance of traditional feature selection methods, and can also improve the stability of feature selection. Then, this paper analyzes the reasons for the instability of feature selection and introduces a feature selection stability measurement method, the Intersection Measurement (IM), to evaluate whether the feature selection process is stable. The effectiveness of the proposed method is verified by experiments on several groups of high-dimensional small sample data sets.

Risk stratification of 2D ultrasound-based breast lesions using hybrid feature selection in machine learning paradigm

Landslide disaster risk reduction necessitates the investigation of different geotechnical causal factors for slope failures. Machine learning (ML) techniques have been proposed to study causal factors across many application areas. However, the development of ensemble ML techniques for identifying the geotechnical causal factors for slope failures and their subsequent prediction has lacked in literature. The primary goal of this research is to develop and evaluate novel feature selection methods for identifying causal factors for slope failures and assess the potential of ensemble and individual ML techniques for slope failure prediction. Twenty-one geotechnical causal factors were obtained from 60 sites (both landslide and non-landslide) spread across a landslide-prone area in Mandi, India. Relevant causal factors were evaluated by developing a novel ensemble feature selection method that involved an average of different individual feature selection methods like correlation, information-gain, gain-ratio, OneR, and F-ratio. Furthermore, different ensemble ML techniques (Random Forest (RF), AdaBoost (AB), Bagging, Stacking, and Voting) and individual ML techniques (Bayesian network (BN), decision tree (DT), multilayer perceptron (MLP), and support vector machine (SVM)) were calibrated to 70% of the locations and tested on 30% of the sites. The ensemble feature selection method yielded six major contributing parameters to slope failures: relative compaction, porosity, saturated permeability, slope angle, angle of the internal friction, and in-situ moisture content. Furthermore, the ensemble RF and AB techniques performed the best compared to other ensemble and individual ML techniques on test data. The present study discusses the implications of different causal factors for slope failure prediction.

With recent advancements in artificial intelligence (AI) and next-generation communication technologies, the demand for Internet-based applications and intelligent digital services is increasing, leading to a significant rise in cyber-attacks such as Distributed Denial-of-Service (DDoS). AI-based DoS detection systems promise adequate identification accuracy with lower false alarms, significantly associated with the data quality used to train the model. Several works have been proposed earlier to select optimum feature subsets for better model generalization and faster learning. However, there is a lack of investigation in the existing literature to identify a common optimum feature set for three main AI methods: machine learning, deep learning, and unsupervised learning. The current works are compromised either with the variation of the feature selection (FS) method or limited to one type of AI model for performance evaluation. Therefore, in this study, we extensively investigated and evaluated the performance of 15 individual FS methods from three major categories: filter-based, wrapper-based, and embedded, and one ensemble feature selection (EnFS) technique. Furthermore, the individual feature subset's quality is evaluated using supervised and unsupervised learning methods for extracting a common best-performing feature subset. According to our experiment, the EnFS method outperforms individual FS and provides a universal best feature set for all kinds of AI models.

In this article, an EFS-LSTM, a deep recurrent learning model, is proposed for network intrusion detection systems. The EFS-LSTM model uses ensemble-based feature selection (EFS) and LSTM (Long Short Term Memory) for the classification of network intrusions. The EFS combines five feature selection mechanisms namely, information gain, gain ratio, chi-square, correlation-based feature selection, and symmetric uncertainty-based feature selection. The experiments were conducted using the benchmark NSL-KDD dataset and implemented using Tensor flow and python. The EFS-LSTM classifier is evaluated using the classification performance metrics and also compared with all the 41 features without any feature selection as well as with each individual feature selection techniques and classified using LSTM. The performance study showed that the EFS-LSTM model outperforms better with 99.8% accuracy with a higher detection and less false alarm rates.