Relevant and Redundant Feature Analysis with Ensemble Classification

With the rising demand for integrated and autonomous systems in the field of engineering, efficient frameworks for instant detection of performance anomalies are imperative for improved productivity and cost-effectiveness. This study proposes a systematic predictive maintenance framework based on the hybrid multisensor fusion technique of fuzzy rough set feature selection and stacked ensemble for the efficient classification of fault conditions characterised by uncertainties. First, a feature vector of time-domain features was extracted from 17 multiple sensor signals. Then, a comparative study of six different Fuzzy Rough Set Feature Selection (FRFS) methods was employed to select the various combinations of optimal feature subsets for various faults classification tasks. The determined optimal feature subsets then served as inputs for training the stacked ensemble (ESB(STK)). In the ESB(STK), Support Vector Machine (SVM), Multilayer Perceptron (MLP), k -Nearest Neighbour ( k -NN), C4.5 Decision Tree (C4.5 DT), Logistic Regression (LR), and Linear Discriminant Analysis (LDA) served as the base classifiers while the LR was selected to be the metaclassifier. The proposed hybrid framework (FRFS-ESB(STK)) improved the classification accuracy with the selected combinations of optimal feature subset size whiles reducing the computational cost, overfitting, training runtime, and uncertainty in modelling. Overall analyses showed that the FRFS-ESB(STK) proved to be generalisable and versatile in the classification of all conditions of four monitored hydraulic components (i.e., cooler, valve, accumulator, and internal pump leakage) when compared with the six base classifiers (standalone) and three existing ensemble classifiers (Stochastic Gradient Boosting (SGB), Ada Boost (ADB), and Bagging (BAG)). The proposed FRFS-ESB(STK) showed an average improvement of 11.28% and 0.88% test accuracies when classifying accumulator and pump conditions, respectively, whiles 100% classification rates were obtained for both cooler and valve.

New hybrid data mining model for credit scoring based on feature selection algorithm and ensemble classifiers

Heart disease is still one of the top causes of death globally, demanding early and effective prediction tools. Traditional predictive models frequently have limitations because they rely on single‐source statistics, which may not represent the entire scenario of a patient's complex health profile. Feature selection is an essential preprocessing step in machine learning that improves model efficiency by eliminating irrelevant or redundant features. This study evaluates the impact of feature selection on six classification modelsNaïve Bayes, Logistic Regression, Decision Tree, K‐Nearest Neighbors (KNN), Support Vector Machine (SVM), and Random Forestacross four datasets. Performance was assessed using accuracy, precision, recall, F1‐score, and area under the curve (AUC) before and after feature selection. The results indicate that feature selection had varying effects across different models and datasets. In Dataset‐01, Random Forest maintained its strong performance (98.70% accuracy) even after feature selection, while SVM and KNN saw noticeable drops, with accuracy decreasing from 88.96% to 79.87% and 87.01% to 81.17% respectively. For Dataset‐02, SVM showed improvement, increasing from 80.00% to 85.00%, while Naïve Bayes and KNN also improved, reaching 82.50% and 82.50% accuracy, respectively. In Dataset‐03, Logistic Regression and KNN experienced declines, with accuracy dropping from 88.76% to 83.71% and 91.57% to 87.08%, respectively, while Random Forest fell from 97.75% to 91.57%. Dataset‐04 showed mixed results, with Logistic Regression improving from 86.67% to 91.11%, but KNN and Random Forest dropping from 97.78% to 84.44% and 91.11% to 82.22%, respectively. These findings suggest that while feature selection enhances performance for some models, it can negatively impact others, particularly those reliant on a larger feature set. Future research should explore adaptive feature selection techniques to optimize model performance across diverse datasets.

With the continuous development of Internet technology, data gradually present a complicated and high-dimensional trend. These high-dimensional data have a large number of redundant features and irrelevant features, which bring great challenges to the existing machine learning algorithms. Feature selection is one of the important research topics in the fields of machine learning, pattern recognition and data mining, and it is also an important means in the data preprocessing stage. Feature selection is to look for the optimal feature subset from the original feature set, which would improve the classification accuracy and reduce the machine learning time. The traditional feature selection algorithm tends to ignore the kind of feature which has a weak distinguishing capacity as a monomer, whereas the feature group’s distinguishing capacity is strong. Therefore, a new dynamic interaction feature selection (DIFS) algorithm is proposed in this paper. Initially, under the theoretical framework of interactive information, it redefines the relevance, irrelevance and redundancy of the features. Secondly, it offers the computational formulas for calculating interactive information. Finally, under the eleven data sets of UCI and three different classifiers, namely, KNN, SVM and C4.5, the DIFS algorithm increases the classification accuracy of the FullSet by 3.2848% and averagely decreases the number of features selected by 15.137. Hence, the DIFS algorithm can not only identify the relevance feature effectively, but also identify the irrelevant and redundant features. Moreover, it can effectively improve the classification accuracy of the data sets and reduce the feature dimensions of the data sets.

Proposing a classifier ensemble framework based on classifier selection and decision tree

Feature selection is referred to the process of obtaining a subset from an original feature set according to certain feature selection criterion, which selects the relevant features of the dataset. It plays a role in compressing the data processing scale, where the redundant and irrelevant features are removed. Feature selection techniques show that more information is not always good in machine learning applications. Apply different algorithms for the data at hand and with baseline classification performance values we can select a final feature selection algorithm. In this paper, we propose a hybrid classification model, which has correlation based filter feature selection algorithm and Machine learning as classifiers. The objective of this study is to select relevant features and analyze the outperform machine learning algorithms in order to train our model, predict and compare their classification performance. In this method, features are ordered according to their Absolute correlation value with respect to the class attribute. Then top K Features are selected from ordered list of features to form a reduced dataset. This proposed classifier model is applied to our smart meter datasets. To measure the performance of these selected features; seven benchmark classifier are used; Random Forest (RF), Logistic Regression (LR), k-Nearest Neighbor (kNN), Naive Bayes (NB), Decision Tree (DT), Linear Discriminant Analysis (LDA) and Support Vector Machine (SVM). This paper then analyzes the performance of all classifiers with feature selection in term of accuracy, sensitivity, F-Measure, Specificity, Precision, and MCC. From our experiment, we found that Random Forest classifier performed higher than other used classifiers.

The world faces various perilous threats due to computer security breaches in the present era. It is proliferating at such a fast rate that it hampers the integrity and confidentiality of people as well as organizations, resulting in a substantial monetary loss. Among different threats, it has been observed that ransomware is one of its types that results in data loss and makes victims by paying huge ransoms. In this study, a research attempt has been made to detect the attack by applying various machine learning techniques with the dataset. First, the data were trained directly using different machine learning techniques such as k-nearest neighbor, support vector machine (SVM) with different kernel functions (SVM-linear, SVM-polynomial, SVM-radial basis function, SVM-sigmoid), random forest, decision tree, and multilayer perceptron without incorporating any feature selection techniques to detect whether the attack is ransomware or benign. Further to optimize the results, feature selection methods based on the filter (Chi-square test, correlation coefficient), wrapper (forward feature selection, backward feature elimination), and embedded methods (Lasso regularization (L1)) are applied to select the prominent features, and redundant features are discarded. Finally, all the results obtained from different experiments are analyzed with critical assessment. By investigating the performance measures of various classifiers, it has been observed that significant improvement in the result is being achieved by the machine learning techniques when the feature selection techniques are considered.

Background/Objectives: Recent studies emphasized on using ensemble models over single ones to solve credit scoring problems. The objective of this study is to build a heterogeneous ensemble classifier model with an improved classification accuracy. Methods: This study focuses on developing a heterogeneous ensemble classifier using Logistic Regression, K-nearest neighbor, Decision tree, Random Forest, Naïve Base and Support vector machine as base classifiers and Random Forest, Logistic Regression and Support vector machine as meta-classifiers. The proposed model is built using these six base classifiers for ensemble aggregation. A feature selection algorithm based on the random forest technique is used for selecting the best features. A stacking and voting method are used for building ensemble model. Findings: The ensemble classifier gives superior predictive performance than single classifiers SVM, DT, RF, NB, KNN and LR with an accuracy of 91.56% for Australian dataset and 84.35% for German dataset. Novelty: The proposed model uses stacking and majority voting method for ensemble classification. Initially, stacking is applied to the base classifiers. This is done in two levels. First the training dataset is split into 10 folds for cross validation. The output of each classifier is taken, and the dataset is updated with the meta-features. In the second level, three meta-classifiers (MC), namely LR, SVM and RF are used. Majority voting is applied to the output of these meta-classifiers for the prediction. Keywords: Credit scoring; ensemble model; SVM; DT; RF; NB; KNN; LR

Ensembles need their base classifiers do not always agree for any prediction (diverse base classifiers). Disturbing Neighbors ($\mathcal{DN}$) is a method for improving the diversity of the base classifiers of any ensemble algorithm. $\mathcal{DN}$ builds for each base classifier a set of extra features based on a 1-Nearest Neighbors (1-NN) output. These 1-NN are built using a small subset of randomly selected instances from the training dataset. $\mathcal{DN}$ has already been proved successfully on unstable base classifiers (i.e. decision trees). This paper presents an experimental validation on 62 UCI datasets for standard ensemble methods using Support Vector Machines (SVM) with a linear kernel as base classifiers. SVMs are very stable, so it is hard to increase their diversity when they belong to an ensemble. However, experiments will show that $\mathcal{DN}$ usually improves ensemble accuracy and base classifiers diversity.

Feature selection is a preprocessing step that plays a crucial role in the domain of machine learning and data mining. Feature selection methods have been shown to be effective in removing redundant and irrelevant features, improving the learning algorithm’s prediction performance. Among the various methods of feature selection based on redundancy, the fast correlation-based filter (FCBF) is one of the most effective. In this paper, we developed a novel extension of FCBF, called resampling FCBF (RFCBF) that combines resampling technique to improve classification accuracy. We performed comprehensive experiments to compare the RFCBF with other state-of-the-art feature selection methods using three competitive classifiers (K-nearest neighbor, support vector machine, and logistic regression) on 12 publicly available datasets. The experimental results show that the RFCBF algorithm yields significantly better results than previous state-of-the-art methods in terms of classification accuracy and runtime.

<p>Classification problems often have a large number of features, but not all of them are useful for classification. Irrelevant and redundant features may even reduce the classification accuracy. Feature selection is a process of selecting a subset of relevant features, which can decrease the dimensionality, shorten the running time, and/or improve the classification accuracy. There are two types of feature selection approaches, i.e. wrapper and filter approaches. Their main difference is that wrappers use a classification algorithm to evaluate the goodness of the features during the feature selection process while filters are independent of any classification algorithm. Feature selection is a difficult task because of feature interactions and the large search space. Existing feature selection methods suffer from different problems, such as stagnation in local optima and high computational cost. Evolutionary computation (EC) techniques are well-known global search algorithms. Particle swarm optimisation (PSO) is an EC technique that is computationally less expensive and can converge faster than other methods. PSO has been successfully applied to many areas, but its potential for feature selection has not been fully investigated. The overall goal of this thesis is to investigate and improve the capability of PSO for feature selection to select a smaller number of features and achieve similar or better classification performance than using all features. This thesis investigates the use of PSO for both wrapper and filter, and for both single objective and multi-objective feature selection, and also investigates the differences between wrappers and filters. This thesis proposes a new PSO based wrapper, single objective feature selection approach by developing new initialisation and updating mechanisms. The results show that by considering the number of features in the initialisation and updating procedures, the new algorithm can improve the classification performance, reduce the number of features and decrease computational time. This thesis develops the first PSO based wrapper multi-objective feature selection approach, which aims to maximise the classification accuracy and simultaneously minimise the number of features. The results show that the proposed multi-objective algorithm can obtain more and better feature subsets than single objective algorithms, and outperform other well-known EC based multi-objective feature selection algorithms. This thesis develops a filter, single objective feature selection approach based on PSO and information theory. Two measures are proposed to evaluate the relevance of the selected features based on each pair of features and a group of features, respectively. The results show that PSO and information based algorithms can successfully address feature selection tasks. The group based method achieves higher classification accuracies, but the pair based method is faster and selects smaller feature subsets. This thesis proposes the first PSO based multi-objective filter feature selection approach using information based measures. This work is also the first work using other two well-known multi-objective EC algorithms in filter feature selection, which are also used to compare the performance of the PSO based approach. The results show that the PSO based multiobjective filter approach can successfully address feature selection problems, outperform single objective filter algorithms and achieve better classification performance than other multi-objective algorithms. This thesis investigates the difference between wrapper and filter approaches in terms of the classification performance and computational time, and also examines the generality of wrappers. The results show that wrappers generally achieve better or similar classification performance than filters, but do not always need longer computational time than filters. The results also show that wrappers built with simple classification algorithms can be general to other classification algorithms.</p>

Emotion recognition plays a pivotal role in advancing Human-Computer Interaction (HCI) by enabling systems to understand and respond to human emotional states. Among various modalities, Speech Emotion Recognition (SER) stands out as a non-invasive, cost-effective, and temporally efficient approach for detecting emotions. This study leverages the RAVDESS dataset, which provides high-quality audio recordings for emotion classification. The proposed methodology involves preprocessing audio signals to remove noise, extracting temporal and spectral features in both time and frequency domains, and implementing machine learning models for multi-class emotion classification. The study evaluates and compares the performance of several classification models, including Random Forest (RF), Multilayer Perceptron (MLP), Support Vector Machine (SVM), Convolutional Neural Networks (CNN), and Decision Trees (DT). Experimental results demonstrate promising accuracy levels, highlighting the potential of these machine learning techniques in SER. This research contributes to the integration of SER within Brain-Computer Interfaces (BCI) and other emotion-aware applications, paving the way for enhanced interactive systems. Keywords – Speech Emotion Recognition -Machine Learning - Random Forest -Multilayer Perceptron - Support Vector Machine - Convolutional Neural Networks - Decision Tree - RAVDESS Dataset

A Comprehensive Survey on Metaheuristic Algorithm for Feature Selection Techniques

A NSGA2-LR wrapper approach for feature selection in network intrusion detection

A hybrid ensemble-filter wrapper feature selection approach for medical data classification