Feature Selection Techniques and Classification Accuracy of Supervised Machine Learning in Text Mining

Background and aimsThe birth weight of a newborn is a crucial factor that affects their overall health and future well-being. Low birth weight (LBW) is a widespread global issue, which the World Health Organization defines as weighing less than 2,500 g. LBW can have severe negative consequences on an individual’s health, including neonatal mortality and various health concerns throughout their life. To address this problem, this study has been conducted using BDHS 2017–2018 data to uncover important aspects of LBW using a variety of machine learning (ML) approaches and to determine the best feature selection technique and best predictive ML model.MethodsTo pick out the key features, the Boruta algorithm and wrapper method were used. Logistic Regression (LR) used as traditional method and several machine learning classifiers were then used, including, DT (Decision Tree), SVM (Support Vector Machine), NB (Naïve Bayes), RF (Random Forest), XGBoost (eXtreme Gradient Boosting), and AdaBoost (Adaptive Boosting), to determine the best model for predicting LBW. The model’s performance was evaluated based on the specificity, sensitivity, accuracy, F1 score and AUC value.ResultsResult shows, Boruta algorithm identifies eleven significant features including respondent’s age, highest education level, educational attainment, wealth index, age at first birth, weight, height, BMI, age at first sexual intercourse, birth order number, and whether the child is a twin. Incorporating Boruta algorithm’s significant features, the performance of traditional LR and ML methods including DT, SVM, NB, RF, XGBoost, and AB were evaluated where LR, had a specificity, sensitivity, accuracy and F1 score of 0.85, 0.5, 85.15% and 0.915. While the ML methods DT, SVM, NB, RF, XGBoost, and AB model’s respective accuracy values were 85.35%, 85.15%, 84.54%, 81.18%, and 84.41%. Based on the specificity, sensitivity, accuracy, F1 score and AUC, RF (specificity = 0.99, sensitivity = 0.58, accuracy = 85.86%, F1 score = 0.9243, AUC = 0.549) outperformed the other methods. Both the classical (LR) and machine learning (ML) models’ performance has improved dramatically when important characteristics are extracted using the wrapper method. The LR method identified five significant features with a specificity, sensitivity, accuracy and F1 score of 0.87, 0.33, 87.12% and 0.9309. The region, whether the infant is a twin, and cesarean delivery were the three key features discovered by the DT and RF models, which were implemented using the wrapper technique. All three models had the identical F1 score of 0.9318. However, “child is twin” was recognized as a significant feature by the SVM, NB, and AB models, with an F1 score of 0.9315. Ultimately, with an F1 score of 0.9315, the XGBoost model recognized “child is twin” and “age at first sex” as relevant features. Random Forest again beat the other approaches in this instance.ConclusionsThe study reveals Wrapper method as the optimal feature selection technique. The ML method outperforms traditional methods, with Random Forest (RF) being the most effective predictive model for Low-Birth-Weight prediction. The study suggests that policymakers in Bangladesh can mitigate low birth weight newborns by considering identified risk factors.

Worldwide, cancer is the most frequent cause of passing away for women. Any development in predicting and diagnosing cancer is crucial for a healthy life. As such, vital cancer accuracy in predicting patients' survival parameters and treatment aspects is necessary. Machine learning methods significantly impact breast cancer diagnosis and early diagnosis. This study aims to increase prediction accuracy using a novel statistical feature selection technique. This article examines the classification test accuracy, standard data precision, and the process performance of multiple machine learning (ML) algorithms, such as Random; using the Wisconsin Diagnostic Breast Cancer (WDBC) dataset, the following models were used: Support Vector Machine, Logistic Regression, Decision Tree (C4.5), Forest Naïve Bayes (NB), Linear Regression (LR), k-nearest-neighbors (KNN), and Multilayer Perceptron (MLP). The data set is partitioned to use the machine learning algorithm: 20% is used during the test phase, while 80% is used throughout training. Hyper-parameters that are manually assigned are utilized to modify the classifier. When applied to a subset of data, it showed that combining SVM and model in machine learning reached the maximum accuracy of up to 90%, which was noticeably superior to the other ML model.

This study presents a comparative analysis of the classification performance of facial and emotion recognition systems using Machine Learning (ML) and Deep Learning (DL) algorithms. The primary objective of this work is to evaluate the applicability of emotion recognition in fields such as psychotherapy and crime analysis, using the FER-2013 dataset.The study was conducted with ML algorithms such as Support Vector Machines (SVM), Random Forest (RF), K-Nearest Neighbor (KNN), Decision Trees, and Gradient Boosting, as well as the DL algorithm, Convolutional Neural Network (CNN). Supported by the application of feature selection, preprocessing, and feature extraction techniques, the model’s performance was measured using standard metrics such as accuracy, precision, F1-score, and AUC-ROC.The experimental results showed that the highest classification accuracy (46.61%) was achieved with the CNN model. While the ML models generally offered lower accuracy, they provided advantages in terms of computational efficiency in specific scenarios.This study aims to contribute to the literature by providing a comparative analysis of ML and DL algorithms and by highlighting the effect of data preprocessing on performance. The findings set targets for future work, such as real-time system integration and hybrid model development.

Performances of machine learning algorithms for individual thermal comfort prediction based on data from professional and practical settings

This project focuses on using machine learning to predict a patient’s risk of developing diabetes based on a test. This database, compiled by the National Institute of Diabetes and Digestive and Kidney Diseases, contains health indicators for Pima Indian patients. The project involves several key steps: initial data, selecting a feature, selecting a model, updating the hyperparameter, and deploying it via the Flask web application. In the data preprocessing phase, feature scaling and normalization are used to standardize the dataset, while missing values and outliers are handled to ensure data integrity. Feature selection uses correlation matrix and recursive feature elimination (RFE) to reduce dimensionality and improve model efficiency. To ensure the model is optimized for latent data, the dataset is split into two parts: 66% for training and 34% for testing. Various machine learning algorithms are evaluated, including logistic regression, naive Bayes, K-nearest neighbours, decision trees, and support vector classifiers. Logistic regression was selected as the final model due to its accuracy on the test data (80.53%). The model uses grid search for hyperparameter tuning to improve its performance. The training model is embedded in the Flask web application, allowing users to access health metrics and get real-time estimates of blood pressure. The system is designed to be user-friendly and scalable, providing a practical tool for early diagnosis of diabetes. All methods ensure that the model is accurate, reliable, and capable of making real-world predictions. Keywords: prediction, diabetes, glucoses, insulin, machine learning, logistic regression, naive bayes, k- nearest neighbours, decision tree, support vector classifier.

Rainfall prediction is the highest research priority in flood-prone areas across the world. This work assesses the abilities of the Decision Tree (DT), Distributed Decision Tree (DDT), Naïve Bayes (NB), Random Forest (RF), Support Vector Machine (SVM), K Nearest Neighbour (KNN), and Fuzzy Logic Decision Tree (FDTs) machine learning algorithms for the rainfall prediction across the Kashmir province of the Union Territory of Jammu & Kashmir. On application of Machine learning algorithms on geographical datasets gave performance accuracy varying from (78.61–81.53)%. Further again machine learning algorithms were reapplied on the dataset without season variable yet again performance ranged in between (77.5–81)%. Vigorous analysis has established that these machine learning models are robust and our study has established that the dataset reaches performance stagnation and thus resulting in performance capping. The stagnation is irrespective of the choice of algorithm and the performance shall not improvise beyond a specific value irrespective of the choice of the machine learning algorithm.KeywordsDecision treeGeographical dataRandom forestNaïve BayesDistribute decision tree

Depression is an extremely serious illness of humans which causes constant mood swings and feelings of sadness. Nowadays, it is considered to be a deadly disorder in the world. At present, everyone from young to old is suffering from depression but most of them do not have the right idea about their mental state. Everyone needs to have a proper idea about their mental state. We will detect depression through a machine learning-based detection approach. Talking with psychologists and depressed people, we find some factors that are related to becoming depressed, and depending on those factors, information is collected from both depressed and non-depressed people. After applying preprocessing techniques, a processed dataset was created finally. Then, feature selection techniques were used. We applied eight machine learning algorithms and two feature selection methods to our dataset. We used k-nearest neighbor (k-NN), decision tree (DT), linear discriminant analysis (LDA), adaptive boosting (AB), support vector machine (SVM), naive Bayes (NB), random forest (RF), and logistic regression (LR) classifier. In our work, the RF classifier gave the best performance based on accuracy and the accuracy of the RF classifier was 96.00%.KeywordsMachine learningFeature selectionRandom forestSelectKBestDetectionDepression

This paper considers the use of feature selection within the state detection module for an ocean turbine condition monitoring system. The goal is to reduce the quantity of data to be processed while maintaining or improving state detection capabilities. Five feature selection techniques (Chi-squared, Information Gain, Signal-To-Noise, AUC and PRC) are evaluated based on their effects on four widely used machine learning algorithms, namely Naive Bayes, k-Nearest Neighbors, Decision Tree and Logistic Regression, when each machine learner is trained on the top n features selected by each feature selection technique. Six values of n (2, 4, 6, 8, 10 and 15) were considered. Features were extracted from the raw vibration signals using a Short Time Wavelet Transform with Baselining (STWTB) technique designed to allow for reliable state detection regardless of the turbine's operating conditions, which are often reflected within its vibration readings. The condition-independent features extracted by the STWTB are then fused to combine all the data observed by all sensor sources. Models were built on data gathered at one operating condition and tested against data from a different operating condition to simulate the problem of building models which work regardless of operating condition. Results show that k-Nearest Neighbors, Naive Bayes and Logistic Regression have improved classification performance when using less than 11% of the 78 available features, with Logistic Regression needing just 2 features selected by the Signal-To-Noise technique to generate a perfect classification model. The Decision Tree performed best without feature selection.

Background: Osteoporosis is a systemic skeletal disorder marked by reduced bone density and microarchitectural deterioration, leading to increased fracture risk. While the dual-energy X-ray absorptiometry (DEXA) scan is the World Health Organization (WHO)-recommended diagnostic standard, its limitations necessitate alternative methods. Emerging magnetic resonance imaging (MRI) techniques, radiomics, and machine learning promise to enhance osteoporosis diagnosis through detailed analysis of lumbar MRI apparent diffusion coefficient (ADC) maps, potentially revolutionizing early detection and treatment strategies. Objectives: In this study, we are going to evaluate the performance of machine learning (ML) models using radiomics features of lumbar MRI ADC map for osteoporosis detection, and to identify significant features and their diagnostic thresholds. Specific performance metrics such as accuracy, sensitivity, specificity, and Area Under the receiver operating characteristic (ROC) Curve (AUC) were assessed. Patients and Methods: This retrospective study employed a cross-sectional design, with a total of 140 cases, including 21 with osteoporosis. The study's inclusion criteria consisted of concurrent lumbar MRI and DEXA within a year, while exclusion criteria included infectious or neoplastic lumbar lesions, fractures, instrumentation, significant osteodegenerative changes, cases where the first four lumbar vertebrae were not included in the imaging field, and absence of diffusion-weighted imaging. Manual segmentation of lumbar vertebrae from ADC maps was performed to create a comprehensive dataset, comprising 5,580 radiomics features per case. Subsequently, the top five features selected by fast correlation-based filter (FCBF) were used to test the performance of seven Machine Learning algorithms (k-Nearest neighbors, decision tree, random forest, logistic regression, support vector machine, naive bayes, and neural network). Statistical tests and ROC curve analysis were conducted to determine the significance and thresholds of these features. Results: The study included 140 cases, with 132 females (94.3%) and 8 males (5.7%), and a mean age of 65.32 ± 8.50 years. The mean BMI was 31.43 ± 5.53 kg/m² for females and 26 ± 3.59 kg/m² for males. In terms of demographic differences, no significant age difference was found between the osteoporotic and non-osteoporotic groups (P = 0.889). However, the osteoporotic group had significantly lower mean body weight (64.90 ± 10.13 kg vs. 74.68 ± 13.94 kg, P = 0.003) and BMI (27.40 ± 4.38 kg/m² vs. 31.77 ± 5.52 kg/m², P = 0.001) compared to the non-osteoporotic group. The median interval between DEXA and lumbar MRI was 1 month (range 0.1 - 11.87 months). The Neural Network model demonstrated the highest performance with an AUC of 0.616 and a classification accuracy of 0.764 using all features. The Naive Bayes model, using the top five features selected by FCBF, showed the highest performance with an AUC of 0.913, accuracy of 0.907, sensitivity of 0.667, and specificity of 0.95. All ML models’ performance were elevated by feature selection. Independent t-tests and Mann-Whitney U tests identified 521 and 670 significant features, respectively (P < 0.05). ROC analysis revealed 58 features with AUC values above 0.70. Conclusion: This study's findings suggest that ML models, particularly the Naive Bayes algorithm, can effectively use lumbar ADC map radiomics to diagnose osteoporosis. These findings could enhance early detection and treatment strategies, potentially improving patient outcomes and reducing the burden of osteoporotic fractures. This study also established threshold values for significant features.

While there are many factors which could contribute to the occurrence of breast cancer, it is very difficult to attribute the exact environmental and other factors contributing to it, but still it has significance in determining the occurrence of cancer. Using machine learning techniques and regular diagnosis information, we can achieve our goal of assessing the risk of occurrence of breast cancer. Cancer data sets contain many attributes of patient information, but not every feature is relevant in predicting cancer. Feature selection techniques are useful in such scenarios for retaining the relevant feature set. In this paper we are doing a comparative study of the effect of feature selection techniques on the accuracies given by existing machine learning algorithms. For this purpose we have considered the following machine learning algorithms - Logistic Regression, Naive Bayes and Random Forest. The following feature selection techniques have been considered - Sequential Forward Feature Selection, Recursive Feature Elimination, f-test and correlation.The publicly available Breast Cancer Wisconsin (Diagnostic) Data Sets from UCI Repository have been used in this work. The results show that random forest algorithm gives the highest accuracy with feature selection. Furthermore f-test gives better results for the smaller dataset and Sequential Forward Selection for the larger dataset.

Credit risk analysis forms an important criterion for the endurance of the banking sector. Understanding credit liability of customer enhances the security stability of the banker. This research is aimed at comparing the predictive accuracy of data mining techniques with feature selection and extraction techniques in the default of credit card clients and suggesting the most efficient technique for analyzing credit card default. Customer's default payments in Taiwan were used as the dataset. Machine Learning classification algorithms like K-nearest neighbor(KNN), Naive Bayesian(NB), Decision Tree(DT) and Support Vector Machines(SVM) were applied to the dataset. Feature filter methods like information gain (IG), gain ratio (GR) and chi-square (CS) and Wrapper based methods like Best First search(BFS) and Hill Climbing search (HCS) were used to identify minimum features required. Feature extraction methods such as Principal Component Analysis(PCA) and Linear Discriminant Anal-ysis(LDA) were also applied. The minimum features of feature selection algorithms along with PCA and LDA were then applied to Base classifiers and their accuracies were tabulated and compared. SVM with feature selection and extraction techniques had the best predictive accuracy.

Financial institutions increasingly rely on machine learning (ML) models to assess credit risk and make lending decisions. Accurate prediction hinges on effective feature selection, which can significantly enhance model performance. This paper investigates the efficacy of seven supervised ML algorithms in predicting credit risk: Naive Bayes, Support Vector Machine, Decision Tree, K-Nearest Neighbor, Artificial Neural Network, Random Forest, and Logistic Regression. Using a German credit dataset comprising 1000 observations with 20 explanatory variables, we evaluated model performance using accuracy, kappa statistic, and F1 score. Two data-splitting scenarios (70-30\% and 80-20\%) were employed to assess robustness. To optimize model performance, we addressed outliers through imputation methods and applied the Boruta algorithm for feature selection, which identified and eliminated six non-contributing features. Our findings consistently demonstrate the superiority of the Random Forest algorithm across both scenarios. In terms of accuracy, Random Forest achieved 77.3\% in the 70-30\% split and 80\% in the 80-20\% split, outperforming all other methods. These results underscore the potential of Random Forest as a valuable tool for credit risk assessment in financial institutions.

An intrusion detection system (IDS) is an important protection instrument for detecting complex network attacks. Various machine learning (ML) or deep learning (DL) algorithms have been proposed for implementing anomaly-based IDS (AIDS). Our review of the AIDS literature identifies some issues in related work, including the randomness of the selected algorithms, parameters, and testing criteria, the application of old datasets, or shallow analyses and validation of the results. This paper comprehensively reviews previous studies on AIDS by using a set of criteria with different datasets and types of attacks to set benchmarking outcomes that can reveal the suitable AIDS algorithms, parameters, and testing criteria. Specifically, this paper applies 10 popular supervised and unsupervised ML algorithms for identifying effective and efficient ML-AIDS of networks and computers. These supervised ML algorithms include the artificial neural network (ANN), decision tree (DT), k-nearest neighbor (k-NN), naive Bayes (NB), random forest (RF), support vector machine (SVM), and convolutional neural network (CNN) algorithms, whereas the unsupervised ML algorithms include the expectation-maximization (EM), k-means, and self-organizing maps (SOM) algorithms. Several models of these algorithms are introduced, and the turning and training parameters of each algorithm are examined to achieve an optimal classifier evaluation. Unlike previous studies, this study evaluates the performance of AIDS by measuring the true positive and negative rates, accuracy, precision, recall, and F-Score of 31 ML-AIDS models. The training and testing time for ML-AIDS models are also considered in measuring their performance efficiency given that time complexity is an important factor in AIDSs. The ML-AIDS models are tested by using a recent and highly unbalanced multiclass CICIDS2017 dataset that involves real-world network attacks. In general, the k-NN-AIDS, DT-AIDS, and NB-AIDS models obtain the best results and show a greater capability in detecting web attacks compared with other models that demonstrate irregular and inferior results.

ObjectiveEnhanced technology in computer and internet has driven scale and quality of data to be improved in various areas including healthcare sectors. Machine Learning (ML) has played a pivotal role in efficiently analyzing those big data, but a general misunderstanding of ML algorithms still exists in applying them (e.g., ML techniques can settle a problem of small sample size, or deep learning is the ML algorithm). This paper reviewed the research of diagnosing mental illness using ML algorithm and suggests how ML techniques can be employed and worked in practice.MethodsResearches about mental illness diagnostic using ML techniques were carefully reviewed. Five traditional ML algorithms-Support Vector Machines (SVM), Gradient Boosting Machine (GBM), Random Forest, Naïve Bayes, and K-Nearest Neighborhood (KNN)-frequently used for mental health area researches were systematically organized and summarized.ResultsBased on literature review, it turned out that Support Vector Machines (SVM), Gradient Boosting Machine (GBM), Random Forest, Naïve Bayes, and K-Nearest Neighborhood (KNN) were frequently employed in mental health area, but many researchers did not clarify the reason for using their ML algorithm though every ML algorithm has its own advantages. In addition, there were several studies to apply ML algorithms without fully understanding the data characteristics.ConclusionResearchers using ML algorithms should be aware of the properties of their ML algorithms and the limitation of the results they obtained under restricted data conditions. This paper provides useful information of the properties and limitation of each ML algorithm in the practice of mental health.

Feature selection is referred to the process of obtaining a subset from an original feature set according to certain feature selection criterion, which selects the relevant features of the dataset. It plays a role in compressing the data processing scale, where the redundant and irrelevant features are removed. Feature selection techniques show that more information is not always good in machine learning applications. Apply different algorithms for the data at hand and with baseline classification performance values we can select a final feature selection algorithm. In this paper, we propose a hybrid classification model, which has correlation based filter feature selection algorithm and Machine learning as classifiers. The objective of this study is to select relevant features and analyze the outperform machine learning algorithms in order to train our model, predict and compare their classification performance. In this method, features are ordered according to their Absolute correlation value with respect to the class attribute. Then top K Features are selected from ordered list of features to form a reduced dataset. This proposed classifier model is applied to our smart meter datasets. To measure the performance of these selected features; seven benchmark classifier are used; Random Forest (RF), Logistic Regression (LR), k-Nearest Neighbor (kNN), Naive Bayes (NB), Decision Tree (DT), Linear Discriminant Analysis (LDA) and Support Vector Machine (SVM). This paper then analyzes the performance of all classifiers with feature selection in term of accuracy, sensitivity, F-Measure, Specificity, Precision, and MCC. From our experiment, we found that Random Forest classifier performed higher than other used classifiers.