Interactive visual formula composition of multidimensional data classifiers

<p>Symbolic regression (SR) is a function identification process, the task of which is to identify and express the relationship between the input and output variables in mathematical models. SR is named to emphasise its ability to find the structure and coefficients of the model simultaneously. Genetic Programming (GP) is an attractive and powerful technique for SR, since it does not require any predefined model and has a flexible representation. However, GP based SR generally has a poor generalisation ability which degrades its reliability and hampers its applications to science and real-world modeling. Therefore, this thesis aims to develop new GP approaches to SR that evolve/learn models exhibiting good generalisation ability. This thesis develops a novel feature selection method in GP for high-dimensional SR. Feature selection can potentially contribute not only to improving the efficiency of learning algorithms but also to enhancing the generalisation ability. However, feature selection is seldom considered in GP for high-dimensional SR. The proposed new feature selection method utilises GP’s built-in feature selection ability and relies on permutation to detect the truly relevant features and discard irrelevant/noisy features. The results confirm the superiority of the proposed method over the other examined feature selection methods including random forests and decision trees on identifying the truly relevant features. Further analysis indicates that the models evolved by GP with the proposed feature selection method are more likely to contain only the truly relevant features and have better interpretability. To address the overfitting issue of GP when learning from a relatively small number of instances, this thesis proposes a new GP approach by incorporating structural risk minimisation (SRM), which is a framework to estimate the generalisation performance of models, into GP. The effectiveness of SRM highly depends on the accuracy of the Vapnik-Chervonenkis (VC) dimension measuring model complexity. This thesis significantly extends an experimental method (instead of theoretical estimation) to measure the VC-dimension of a mixture of linear and nonlinear regression models in GP for the first time. The experimental method has been conducted using uniform and non-uniform settings and provides reliable VC-dimension values. The results show that our methods have an impressively better generalisation gain and evolve more compact model, which have a much smaller behavioural difference from the target models than standard GP and GP with bootstrap, The proposed method using the optimised non-uniform setting further improves the one using the uniform setting. This thesis employs geometric semantic GP (GSGP) to tackle the unsatisfied generalisation performance of GP for SR when no overfitting occurs. It proposes three new angle-awareness driven geometric semantic operators (GSO) including selection, crossover and mutation to further explore the geometry of the semantic space to gain a greater generalisation improvement in GP for SR. The angle-awareness brings new geometric properties to these geometric operators, which are expected to provide a greater leverage for approximating the target semantics in each operation, and more importantly, to be resistant to overfitting. The results show that compared with two kinds of state-of-the-art GSOs, the proposed new GSOs not only drive the evolutionary process fitting the target semantics more efficiently but also significantly improve the generalisation performance. A further comparison on the evolved models shows that the new method generally produces simpler models with a much smaller size and containing important building blocks of the target models.</p>

Today, video data, as a powerful multimedia component, is accompanied by some problems with increasing usage in communication, health, education, and social media in particular. Classification and detection of concepts in video data by automatic methods are some of these challenging problems. In this study, we propose a video classification system, which incorporates deep convolutional neural networks (CNNs) by leveraging feature selection and data fusion techniques to improve the accuracy of the classification. Principal Component Analysis (PCA) as a feature selection method and Discriminant Correlation Analysis (DCA) technique, which incorporates class associations into the correlation analysis of feature sets for data fusion, are applied to the problem at the feature level. Support Vector Machines (SVMs) have been trained with new feature vectors obtained from different deep convolutional neural networks by feature selection and data fusion methods. The proposed method is tested for 38 concepts on TRECVID 2013 SIN video task dataset and the results are evaluated. Our results show that the classification accuracy is improved by 4% with an accuracy of 50.27% when the proposed data fusion and feature selection techniques are used.

Generally, there are two types of approaches available for the detection of networks attacks, namely signature based and anomaly based. In this work, we will analyze the performance of anomaly-based detection model in SDN with the help of some common machine learning algorithms and feature selection mechanisms. We construct a mechanism of machine learning model for an intrusion detection system and train the model with the NSL-KDD Data set using feature selection technique. In order to enhance the performance of the classifier, some feature selection methods have been applied as a prepossessing of the data set. We have used five feature selection methods, namely Info Gain, Gain Ratio, CFS Subset Evaluator, Symmetric Uncertainty, and Chi-square test. A full data set of 41 features and a reduced data set after applying feature selection method has been experimented. A data set with feature selection ensures the highest accuracy with Random Forest classifier using Gain Ratio feature selection Evaluator.

In credit risk evaluation the accuracy of a classifier is very significant for classifying the high-risk loan applicants correctly. Feature selection is one way of improving the accuracy of a classifier. It provides the classifier with important and relevant features for model development. This study uses the ensemble of multiple feature ranking techniques for feature selection of credit data. It uses five individual rank based feature selection methods. It proposes a novel rank aggregation algorithm for combining the ranks of the individual feature selection methods of the ensemble. This algorithm uses the rank order along with the rank score of the features in the ranked list of each feature selection method for rank aggregation. The ensemble of multiple feature selection techniques uses the novel rank aggregation algorithm and selects the relevant features using the 80%, 60%, 40% and 20% thresholds from the top of the aggregated ranked list for building the C4.5, MLP, C4.5 based Bagging and MLP based Bagging models. It was observed that the performance of models using the ensemble of multiple feature selection techniques is better than the performance of 5 individual rank based feature selection methods. The average performance of all the models was observed as best for the ensemble of feature selection techniques at 60% threshold. Also, the bagging based models outperformed the individual models most significantly for the 60% threshold. This increase in performance is more significant from the fact that the number of features were reduced by 40% for building the highest performing models. This reduces the data dimensions and hence the overall data size phenomenally for model building. The use of the ensemble of feature selection techniques using the novel aggregation algorithm provided more accurate models which are simpler, faster and easy to interpret.

With the rapid development of artificial intelligence in recent years, the research on image processing, text mining, and genome informatics has gradually deepened, and the mining of large-scale databases has begun to receive more and more attention. The objects of data mining have also become more complex, and the data dimensions of mining objects have become higher and higher. Compared with the ultra-high data dimensions, the number of samples available for analysis is too small, resulting in the production of high-dimensional small sample data. High-dimensional small sample data will bring serious dimensional disasters to the mining process. Through feature selection, redundancy and noise features in high-dimensional small sample data can be effectively eliminated, avoiding dimensional disasters and improving the actual efficiency of mining algorithms. However, the existing feature selection methods emphasize the classification or clustering performance of the feature selection results and ignore the stability of the feature selection results, which will lead to unstable feature selection results, and it is difficult to obtain real and understandable features. Based on the traditional feature selection method, this paper proposes an ensemble feature selection method, Random Bits Forest Recursive Clustering Eliminate (RBF-RCE) feature selection method, combined with multiple sets of basic classifiers to carry out parallel learning and screen out the best feature classification results, optimizes the classification performance of traditional feature selection methods, and can also improve the stability of feature selection. Then, this paper analyzes the reasons for the instability of feature selection and introduces a feature selection stability measurement method, the Intersection Measurement (IM), to evaluate whether the feature selection process is stable. The effectiveness of the proposed method is verified by experiments on several groups of high-dimensional small sample data sets.

Last century, the challenge was to develop new technologies that store large amount of data. Recently, the challenges are to effectively utilize the incredible amount of data and to obtain knowledge that benefits business, scientific, and government transactions by using subset of features rather than the whole features in the dataset. In the present paper, we have focused on feature selection techniques as a method to gain high quality attributes to enhance the mining process. Feature selection techniques touch all disciplines that require knowledge discovery from large data. In our study, we made a comparison between benchmark feature selection methods based on three well-known datasets and three well-recognized machine learning algorithms. The study found that feature selection methods are capable to improve the performance of learning algorithms. However, no single feature selection methods that best satisfy all datasets and learning algorithms. Therefore, machine learning researchers should understand the nature of datasets and learning algorithms characteristics in order to obtain better outcome. Overall, correlation based feature selection (CFS) and consistency based subset evaluation (CB) performed better than information gain, symmetrical uncertainty, Relief (RF), and principle components analysis (PC).

High-dimensional datasets can lead to overfitting and computationally expensive model building on machine learning. This study uses a dimensionality reduction technique, namely feature selection techniques, to overcome these problems. Five feature selection methods were used, i.e., Chi-Square (CS), Information Gain (IG), Genetic Algorithm (GA), Particle Swarm Optimization (PSO), and Least Absolute Shrinkage and Selection Operator (LASSO), and three classifier methods viz. Naïve Bayes, Extreme Gradient Boosting (XGB), and RF Classifier. The dataset used is the Heart Attack Analysis & Prediction Dataset. In this study, three scenarios of the best feature selection were carried out, namely: 1. selection of the best feature using a specific feature selection, 2. the intersection of selection of the best feature from the same category, 3. the intersection of selection of the best feature from the five proposed feature selection methods. The performance model is measured using accuracy, precision, recall, f1-score, AUC, and training time. This study reveals that feature selection is very effective in improving the performance of prediction models. Based on the experiment results, the best feature selection is CS and IG in the Filter Category with the XGB model. The best feature selected improved the performance of accuracy, precision, recall, f1-score, and AUC, i.e., 1.7%, 1%, 2.3%, 1.6%, and 0.2%, respectively. Meanwhile, training time requirements decreased by 23.5%. Feature selection with specific techniques performs better than feature selection by selecting the best features from the same category feature selection technique or various other feature selection methods.

A common challenge in large-scale data sets with numerous predictors is multicollinearity, which can degrade the performance and interpretability of predictive models. Although dimensionality reduction techniques like Principal Component Analysis (PCA) can mitigate multicollinearity, they transform original variables into new uncorrelated variables, often reducing the model interpretability. For this reason, feature selection techniques are generally preferred in domains where understanding of the original predictors is essential. This paper presents a novel feature selection method that combines machine learning techniques with forward feature selection to effectively address multicollinearity while preserving the accuracy of prediction. The proposed method is evaluated on three diverse data sets from the maritime, healthcare, and environmental domains. Regression models were used to assess predictive performance based on adjusted R 2 , Root Mean Square Error and Mean Absolute Error. The results demonstrate that the proposed method achieves high performance comparable to traditional feature selection techniques while preventing multicollinearity. This balance between predictive power and feature stability makes it especially suitable for high-dimensional data sets, where redundant or highly correlated variables are prevalent. The approach supports more robust and interpretable models, highlighting its potential for widespread application in complex, data-intensive domains.

Drug-Protein Interaction (DPI) identification is crucial in drug discovery. The high dimensionality of drug and protein features poses challenges for accurate interaction prediction, necessitating the use of computational techniques. Docking-based methods rely on 3D structures, while ligand-based methods have limitations such as reliance on known ligands and neglecting protein structure. Therefore, the preferred approach is the chemogenomics-based approach using machine learning, which considers both drug and protein characteristics for DPI prediction. In machine learning, feature selection plays a vital role in improving model performance, reducing overfitting, enhancing interpretability, and making the learning process more efficient. It helps extract meaningful patterns from drug and protein data while eliminating irrelevant or redundant information, resulting in more effective machine-learning models. On the other hand, classification is of great importance as it enables pattern recognition, decision-making, predictive modeling, anomaly detection, data exploration, and automation. It empowers machines to make accurate predictions and facilitates efficient decision-making in DPI prediction. For this research work, protein data was sourced from the KEGG database, while drug data was obtained from the DrugBank data machine-learning base. To address the issue of imbalanced Drug Protein Pairs (DPP), different balancing techniques like Random Over Sampling (ROS), Synthetic Minority Over-sampling Technique (SMOTE), and Adaptive SMOTE were employed. Given the large number of features associated with drugs and proteins, feature selection becomes necessary. Various feature selection methods were evaluated: Correlation, Information Gain (IG), Chi-Square (CS), and Relief. Multiple classification methods, including Support Vector Machines (SVM), Random Forest (RF), Adaboost, and Logistic Regression (LR), were used to predict DPI. Finally, this research identifies the best balancing, feature selection, and classification methods for accurate DPI prediction. This comprehensive approach aims to overcome the limitations of existing methods and provide more reliable and efficient predictions in drug-protein interaction studies.

Software defect prediction (SDP) is used to identify defects in software modules that can be a challenge in software development. This research focuses on the problems that occur in Particle Swarm Optimization (PSO), such as the problem of noisy attributes, high-dimensional data, and premature convergence. So this research focuses on improving PSO performance by using feature selection methods with hybrid techniques to overcome these problems. The feature selection techniques used are Filter and Wrapper. The methods used are Chi-Square (CS), Correlation-Based Feature Selection (CFS), and Forward Selection (FS) because feature selection methods have been proven to overcome data dimensionality problems and eliminate noisy attributes. Feature selection is often used by some researchers to overcome these problems, because these methods have an important function in the process of reducing data dimensions and eliminating uncorrelated attributes that can cause noisy. Naive Bayes algorithm is used to support the process of determining the most optimal class. Performance evaluation will use AUC with an alpha value of 0.050. This hybrid feature selection technique brings significant improvement to PSO performance with a much lower AUC value of 0.00342. Comparison of the significance of AUC with other combinations shows the value of FS PSO of 0.02535, CFS FS PSO of 0.00180, and CS FS PSO of 0.01186. The method in this study contributes to improving PSO in the SDP domain by significantly increasing the AUC value. Therefore, this study highlights the potential of feature selection with hybrid techniques to improve PSO performance in SDP.

Introduction: The early and accurate detection of COVID-19 remains a life-threatening challenge in medical analysis. Machine learning is used for predicting disease outcomes based on clinical parameters. This analysis proposes a comparative analysis of feature selection method and classification techniques to enhance COVID-19 detection accuracy using blood biomarkers. We used a pensourse dataset of 1,724 cases, including 35 features. To improve the model performance data preprocessing process included outlier handling, normalization, and transformation techniques to improve model performance. To identify the relevant features, we employed the three-feature selection methods Chi-Square test, Pearson correlation coefficient, and Random Forest. The model prediction accuracy was enhanced using a stacking ensemble classication techniques. The machine learning based classification models effectively predicted COVID-19 infectious disease using blood biomarkers with optimized feature selection techniques. Objectives: To enhance the accuracy of COVID-19 prediction using machine learning techniques by applying feature selection and classification techniques on blood biomarkers. Methods: The comparative analysis utilized a publicly available dataset containing 1,724 cases with 35 attributes. Data preprocessing involved outlier handling, normalization, and transformation techniques. Employed Chi-Square test, Pearson correlation coefficient, and Random Forest feature selection techniques. Stacking ensemble classification algorithm was utilized for the better performance of a model. Results: The classification models demonstrated efficiency in predicting COVID-19 using blood biomarkers. Optimized feature selection significantly improved predictive accuracy, highlighting the importance of selecting relevant features for model performance enhancement. Conclusions: This study showcases the potential of ML-driven approaches for COVID-19 detection, emphasizing the role of feature selection in improving classification accuracy. The findings contribute to the advancement of diagnostic tools, offering a data-driven solution for rapid and reliable COVID-19 screening.

Background: RNA-seq data from tumor samples can be used to identify novel cancer subtypes using cluster analysis. The number of features is often large compared to the number of samples and different clusters can appear in different subsets of the feature space. Feature selection techniques are therefore commonly used to reduce the dimension and remove redundant and irrelevant features before performing cluster analysis. An abundance of feature selection methods have been proposed in the literature, but it is unclear how the ability to identify novel subtypes of cancer using cluster analysis is affected by choice of feature selection method. Method: We evaluated 13 feature selection methods on four publicly available cancer data sets from The Cancer Genome Atlas. RNA-seq data and associated clinical data were retrieved from Broad institute GDAC Firehose. Overlap and characterization of the highest ranked features (i.e. genes) were studied for top 100, 1000 and 3000 genes. Performance was measured by comparing known cancer subtypes to partitions obtained using hierarchical clustering with Euclidean distance and Ward´s linkage, based on the selected features. The result was compared to both a random selection (negative control) and a supervised approach (positive control) using adjusted Rand index. Results: The relative performance of the feature selection methods varied heavily depending on number of included genes and the data set used. Based on all data, gene selection using the Dip-test statistic and the Bimodality index generated the overall highest clustering performance. The overlap of selected genes between the Bimodality index, the Dip-test and the supervised approach was relatively low. The Dip-test and Bimodality index tended to favor genes with relatively low expression values. Conclusions: The choice of feature selection method can have a huge impact on the ability to identify cancer subtypes using cluster analysis, and the relative performance is highly dependent on the data. Low overlap of selected genes between the highest ranked methods suggests that the methods identify genes that contain complementary information, and that it might be beneficial to combine two or more feature selection methods. Citation Format: Linda Vidman, David Källberg, Patrik Rydén. Evaluation of feature selection methods used for cluster analysis in identification of novel cancer subtypes [abstract]. In: Proceedings of the Annual Meeting of the American Association for Cancer Research 2020; 2020 Apr 27-28 and Jun 22-24. Philadelphia (PA): AACR; Cancer Res 2020;80(16 Suppl):Abstract nr 5475.

Feature selection (FS) plays a crucial role in medical imaging by reducing dimensionality, improving computational efficiency, and enhancing diagnostic accuracy. Traditional FS techniques, including filter, wrapper, and embedded methods, have been widely used but often struggle with high-dimensional and heterogeneous medical imaging data. Deep learning-based FS methods, particularly Convolutional Neural Networks (CNNs) and autoencoders, have demonstrated superior performance but lack interpretability. Hybrid approaches that combine classical and deep learning techniques have emerged as a promising solution, offering improved accuracy and explainability. Furthermore, integrating multi-modal imaging data (e.g., Magnetic Resonance Imaging (MRI), Computed Tomography (CT), Positron Emission Tomography (PET), and Ultrasound (US)) poses additional challenges in FS, necessitating advanced feature fusion strategies. Multi-modal feature fusion combines information from different imaging modalities to improve diagnostic accuracy. Recently, quantum computing has gained attention as a revolutionary approach for FS, providing the potential to handle high-dimensional medical data more efficiently. This systematic literature review comprehensively examines classical, Deep Learning (DL), hybrid, and quantum-based FS techniques in medical imaging. Key outcomes include a structured taxonomy of FS methods, a critical evaluation of their performance across modalities, and identification of core challenges such as computational burden, interpretability, and ethical considerations. Future research directions—such as explainable AI (XAI), federated learning, and quantum-enhanced FS—are also emphasized to bridge the current gaps. This review provides actionable insights for developing scalable, interpretable, and clinically applicable FS methods in the evolving landscape of medical imaging.

Ensemble gene (feature) selection is a promising new strategy with many benefits including more stable gene lists and improved classification results. The ensemble portion is achieved through multiple runs of feature selection which are then aggregated into a single result. The critical question is how many iterations of feature selection are appropriate. Too few iterations can make classification performance suffer. However, too many iterations will cause issues regarding computational costs. The goal is to choose the correct number of iterations to maximize classification performance without expending too much computational power. Our paper is an in-depth study on the effect of the number of iterations of feature selection on classification performance. Our work employs eleven DNA microarray datasets, on which we apply various ensemble methods, feature selection techniques, classifiers, and feature subset sizes. The results show that using 10 iterations of feature ranking during ensemble feature selection is not sufficient to optimize classification results and that a larger number of iterations is required (20 or 50). However, there is very little distinction between 20 iterations and 50 iterations, as both produce very similar classification results. Our recommendation is to use 20 iterations because while 20 iterations and 50 iterations perform similarly, 20 iterations has a much smaller computation time. To our knowledge there has not been a previous study as expansive as this one on the effects of the number of iterations of feature selection on ensemble feature selection.

Background: Feature selection techniques are important factors for improving machine learning models because they increase prediction accuracy and decrease the time to create a model. Recently, feature selection techniques have been employed on software quality prediction problems with different results and no clear indication of which techniques are frequently used.Objective: This study aims to conduct a systematic review of the application of feature selection techniques in software quality prediction and answers eight research questions.Method: The review evaluates 15 papers in 9 journals and 6 conference proceedings from 2007 to 2017 using the standard systematic literature review method.Results: The results obtained from this study reveal that the filter feature selection method was the most commonly used in the studies (60%) and RELIEF was the most employed among this method, and a limited number of studies employed an ensemble method. Several studies used public datasets available in the PROMISE software project repository (60%). Most studies focused on software defect prediction (classification problem) using area under curve (AUC) as a primary evaluation measure, whereas only two studies focused on software maintainability prediction (regression problem) using mean magnitude of relative error (MMRE) as a primary evaluation measure. All selected studies performed k-fold cross-validation to evaluate model accuracy. Individual prediction models were mostly employed and ensemble models appeared only in three studies. Naive Bayes was the most investigated among individual models, whereas Random forest was the most investigated among ensemble models.Conclusion: Feature selection techniques used by selected primary studies have a positive impact on the performance of the prediction models. Further, both ensemble feature selection method and ensemble models have the ability for increasing prediction accuracy over single methods or individual models and have reported improvement in the prediction accuracy; however, the application of these techniques in software quality prediction is still limited.