A Review of Dimensionality Reduction Techniques for Efficient Computation

Dimensionality reduction as a preprocessing step to machine learning is effective in removing irrelevant and redundant data, increasing learning accuracy, and improving result comprehensibility. However, the recent increase of dimensionality of data poses a severe challenge to many existing feature selection and feature extraction methods with respect to efficiency and effectiveness. In the field of machine learning and pattern recognition, dimensionality reduction is important area, where many approaches have been proposed. In this paper, some widely used feature selection and feature extraction techniques have analyzed with the purpose of how effectively these techniques can be used to achieve high performance of learning algorithms that ultimately improves predictive accuracy of classifier. An endeavor to analyze dimensionality reduction techniques briefly with the purpose to investigate strengths and weaknesses of some widely used dimensionality reduction methods is presented.

Credit risk analysis forms an important criterion for the endurance of the banking sector. Understanding credit liability of customer enhances the security stability of the banker. This research is aimed at comparing the predictive accuracy of data mining techniques with feature selection and extraction techniques in the default of credit card clients and suggesting the most efficient technique for analyzing credit card default. Customer's default payments in Taiwan were used as the dataset. Machine Learning classification algorithms like K-nearest neighbor(KNN), Naive Bayesian(NB), Decision Tree(DT) and Support Vector Machines(SVM) were applied to the dataset. Feature filter methods like information gain (IG), gain ratio (GR) and chi-square (CS) and Wrapper based methods like Best First search(BFS) and Hill Climbing search (HCS) were used to identify minimum features required. Feature extraction methods such as Principal Component Analysis(PCA) and Linear Discriminant Anal-ysis(LDA) were also applied. The minimum features of feature selection algorithms along with PCA and LDA were then applied to Base classifiers and their accuracies were tabulated and compared. SVM with feature selection and extraction techniques had the best predictive accuracy.

A common challenge in large-scale data sets with numerous predictors is multicollinearity, which can degrade the performance and interpretability of predictive models. Although dimensionality reduction techniques like Principal Component Analysis (PCA) can mitigate multicollinearity, they transform original variables into new uncorrelated variables, often reducing the model interpretability. For this reason, feature selection techniques are generally preferred in domains where understanding of the original predictors is essential. This paper presents a novel feature selection method that combines machine learning techniques with forward feature selection to effectively address multicollinearity while preserving the accuracy of prediction. The proposed method is evaluated on three diverse data sets from the maritime, healthcare, and environmental domains. Regression models were used to assess predictive performance based on adjusted R 2 , Root Mean Square Error and Mean Absolute Error. The results demonstrate that the proposed method achieves high performance comparable to traditional feature selection techniques while preventing multicollinearity. This balance between predictive power and feature stability makes it especially suitable for high-dimensional data sets, where redundant or highly correlated variables are prevalent. The approach supports more robust and interpretable models, highlighting its potential for widespread application in complex, data-intensive domains.

In many medical applications, feature selection is obvious; but in medical domains, selecting features and creating a feature vector may require more effort. The wavelet transform (WT) technique is used to identify the characteristic points of an electrocardiogram (ECG) signal with fairly good accuracy, even in the presence of severe high-frequency and low-frequency noise. Principal component analysis (PCA) is a suitable technique for ECG data analysis, feature extraction, and image processing — an important technique that is not based upon a probability model. The aim of the paper is to derive better diagnostic parameters for reducing the size of ECG data while preserving morphology, which can be done by PCA. In this analysis, PCA is used for decorrelation of ECG signals, noise, and artifacts from various raw ECG data sets. The aim of this paper is twofold: first, to describe an elegant algorithm that uses WT alone to identify the characteristic points of an ECG signal; and second, to use a composite WT-based PCA method for redundant data reduction and better feature extraction. PCA scatter plots can be observed as a good basis for feature selection to account for cardiac abnormalities. The study is analyzed with higher-order statistics, in contrast to the conventional methods that use only geometric characteristics of feature waves and lower-order statistics. A new algorithm — viz. PCA variance estimator — is developed for this analysis, and the results are also obtained for different combinations of leads to find correlations for feature classification and useful diagnostic information. PCA scatter plots of various chest and augmented ECG leads are obtained to examine the varying orientations of the ECG data in different quadrants, indicating the cardiac events and abnormalities. The efficacy of the PCA algorithm is tested on different leads of 12-channel ECG data; file no. 01 of the Common Standards for Electrocardiography (CSE) database is used for this study. Better feature extraction is obtained for some specific combinations of leads, and significant improvement in signal quality is achieved by identifying the noise and artifact components. The quadrant analysis discussed in this paper highlights the filtering requirements for further ECG processing after performing PCA, as a primary step for decorrelation and dimensionality reduction. The values of the parameters obtained from the results of PCA are also compared with those of wavelet methods.

<p>Soil properties could be assessed with reflectance spectroscopy (soil spectroscopy, SS) in vis-NIR region (400-2500 nm) through absorption features found in soil spectra. A high spectral resolution (up to 1 nm) drives to high dimensional and multicollinear data. This issue is usually addressed prior to modelling with feature extraction methods such as Principal Component Analysis, or embedded methods such as Partial Least Squares Regression (PLSR). Feature Selection (FS) wrapper methods are promising dimensionality reduction approaches barely used in SS. The objective of this study was two-fold: i) evaluate the performance of FS wrapper methods built from Random Forest (RF) algorithm to predict soil organic matter (SOM), clay and carbonates using laboratory spectroscopy, ii) test the performance of FS methods for dimensionality reduction in SS. The reflectance of 100 soil samples from Sierra de las Nieves National Park (Spain), was measured under laboratory conditions using an ASD FieldSpec Pro JR. A spectral preprocessing method, Continumm Removal (CR), was applied to raw spectra. The RF wrapper considered two different feature searching approaches: Sequential Forward Selection (SFS) and Sequential Flotant Forward Selection (SFFS). The performance of RF with FS (RF-FS) was compared to that of Partial Least Squares Regression (PLSR) and RF (without FS). Models were evaluated with R-squared, root mean squared error (RMSE) and ratio of prediction to deviation (RPD).</p><p>RF-FS models outperformed PLSR and RF models for the three SAP. RF-FS best models had a RPD of 2.19 for SOM, 1.64 for carbonates and 1.52 for clay, whereas PLSR models had RPD values of 1.59, 1.22 and 1.3, and RF 1.38, 1.23 and 1.23 for SOM, carbonates, and clay, respectively. Therefore, FS was useful in obtaining models with improved accuracy by reducing redundant features and avoiding multicollinearity (Hughes effect). The application of FS wrapper methods reduced the number of features in the RF-FS models to less than 1% of the starting features. Features were selected across all spectra from SOM and clay, and around 900, 1900 and 2350 nm for carbonates. This research, thus, shows an alternative to different feature extraction approaches for modelling soil properties based on FS methods and machine learning.</p>

Correct detection of plant diseases is critical for enhancing crop yield and quality. Conventional methods, such as visual inspection and microscopic analysis, are typically labor‐intensive, subjective, and vulnerable to human error, making them infeasible for extensive monitoring. In this study, we propose a novel technique to detect tomato leaf diseases effectively and efficiently through a pipeline of four stages. First, image enhancement techniques deal with problems of illumination and noise to recover the visual details as clearly and accurately as possible. Subsequently, regions of interest (ROIs), containing possible symptoms of a disease, are accurately captured. The ROIs are then fed into K‐means clustering, which can separate the leaf sections based on health and disease, allowing the diagnosis of multiple diseases. After that, a hybrid feature extraction approach taking advantage of three methods is proposed. A discrete wavelet transform (DWT) extracts hidden and abstract textures in the diseased zones by breaking down the pixel values of the images to various frequency ranges. Through spatial relation analysis of pixels, the gray level co‐occurrence matrix (GLCM) is extremely valuable in delivering texture patterns in correlation with specific ailments. Principal component analysis (PCA) is a technique for dimensionality reduction, feature selection, and redundancy elimination. We collected 9014 samples from publicly available repositories; this dataset allows us to have a diverse and representative collection of tomato leaf images. The study addresses four main diseases: curl virus, bacterial spot, late blight, and Septoria spot. To rigorously evaluate the model, the dataset is split into 70%, 10%, and 20% as training, validation, and testing subsets, respectively. The proposed technique was able to achieve a fantastic accuracy of 99.97%, higher than current approaches. The high precision achieved emphasizes the promising implications of incorporating DWT, PCA, GLCM, and ANN techniques in an automated system for plant diseases, offering a powerful solution for farmers in managing crop health efficiently.

Today, video data, as a powerful multimedia component, is accompanied by some problems with increasing usage in communication, health, education, and social media in particular. Classification and detection of concepts in video data by automatic methods are some of these challenging problems. In this study, we propose a video classification system, which incorporates deep convolutional neural networks (CNNs) by leveraging feature selection and data fusion techniques to improve the accuracy of the classification. Principal Component Analysis (PCA) as a feature selection method and Discriminant Correlation Analysis (DCA) technique, which incorporates class associations into the correlation analysis of feature sets for data fusion, are applied to the problem at the feature level. Support Vector Machines (SVMs) have been trained with new feature vectors obtained from different deep convolutional neural networks by feature selection and data fusion methods. The proposed method is tested for 38 concepts on TRECVID 2013 SIN video task dataset and the results are evaluated. Our results show that the classification accuracy is improved by 4% with an accuracy of 50.27% when the proposed data fusion and feature selection techniques are used.

An intrusion detection system (IDS) is a device or software application that monitors a network for malicious activity or policy violations. It scans a network or a system for a harmful activity or security breaching. IDS protects networks (Network-based intrusion detection system NIDS) or hosts (Host-based intrusion detection system HIDS), and work by either looking for signatures of known attacks or deviations from normal activity. Deep learning algorithms proved their effectiveness in intrusion detection compared to other machine learning methods. In this paper, we implemented deep learning solutions for detecting attacks based on Long Short-Term Memory (LSTM). PCA (principal component analysis) and Mutual information (MI) are used as dimensionality reduction and feature selection techniques. Our approach was tested on a benchmark data set, KDD99, and the experimental outcomes show that models based on PCA achieve the best accuracy for training and testing, in both binary and multiclass classification.

This paper represents handwritten digit recognition on a very well-known dataset which is MNIST dataset using the Linear Binary Pattern (LBP) and Scale-Invariant Feature Transform (SIFT) feature extraction methods. From the dataset, features have been extracted using this extracting methods. After this, to reduce the number of features or reducing the dimension we have used Principal Component Analysis (PCA) for better performance for our proposed classifier. Then it has been trained by various classifiers. Then the accuracies and errors of those classifiers have been compared and demonstrated. Also some statistical analysis has been done for better understanding of the dataset. From those comparison, it has been shown that our proposed model (SIFT+PCA+CNN) has the better accuracy and less error than other classifiers. The results are competitively compared to previous works and they provide a baseline for evaluation of future work.

Classification of human hand movements based on EMG signals using nonlinear dimensionality reduction and data fusion techniques

Financial distress identification remains an essential topic in the scientific literature due to its importance for society and the economy. The advancements in information technology and the escalating volume of stored data have led to the emergence of financial distress that transcends the realm of financial statements and its’ indicators (ratios). The feature space could be expanded by incorporating new perspectives on feature data categories such as macroeconomics, sectors, social, board, management, judicial incident, etc. However, the increased dimensionality results in sparse data and overfitted models. This study proposes a new approach for efficient financial distress classification assessment by combining dimensionality reduction and machine learning techniques. The proposed framework aims to identify a subset of features leading to the minimization of the loss function describing the financial distress in an enterprise. During the study, 15 dimensionality reduction techniques with different numbers of features and 17 machine-learning models were compared. Overall, 1,432 experiments were performed using Lithuanian enterprise data covering the period from 2015 to 2022. Results revealed that the artificial neural network (ANN) model with 30 ranked features identified using the Random Forest mean decreasing Gini (RF_MDG) feature selection technique provided the highest AUC score. Moreover, this study has introduced a novel approach for feature extraction, which could improve financial distress classification models.

This study aims to appraise the key features and components of chronic kidney disease (CKD) problems in the case of the classification task. There are different types of the classifier which presented a result with low accuracy. A new combined classifier may be needed to achieve better results. Classification techniques used include k-nearest neighbors (k-NN), naive bayes (NB), and their proposed ensemble model. A feature selection technique (FST) based on genetic algorithms (GA) is presented and compared to famous dimensionality reduction techniques (DRT), namely principal component analysis (PCA). The proposed ensemble model classifiers have higher accuracy rates on the CKD dataset than the other classification models before and after applying the FST and DRT. The results demonstrated that employing FST using GA and DRT as PCA has enhanced classification accuracy. This model can be used for the identification of CKD in the early stage.

High-dimensional datasets can lead to overfitting and computationally expensive model building on machine learning. This study uses a dimensionality reduction technique, namely feature selection techniques, to overcome these problems. Five feature selection methods were used, i.e., Chi-Square (CS), Information Gain (IG), Genetic Algorithm (GA), Particle Swarm Optimization (PSO), and Least Absolute Shrinkage and Selection Operator (LASSO), and three classifier methods viz. Naïve Bayes, Extreme Gradient Boosting (XGB), and RF Classifier. The dataset used is the Heart Attack Analysis & Prediction Dataset. In this study, three scenarios of the best feature selection were carried out, namely: 1. selection of the best feature using a specific feature selection, 2. the intersection of selection of the best feature from the same category, 3. the intersection of selection of the best feature from the five proposed feature selection methods. The performance model is measured using accuracy, precision, recall, f1-score, AUC, and training time. This study reveals that feature selection is very effective in improving the performance of prediction models. Based on the experiment results, the best feature selection is CS and IG in the Filter Category with the XGB model. The best feature selected improved the performance of accuracy, precision, recall, f1-score, and AUC, i.e., 1.7%, 1%, 2.3%, 1.6%, and 0.2%, respectively. Meanwhile, training time requirements decreased by 23.5%. Feature selection with specific techniques performs better than feature selection by selecting the best features from the same category feature selection technique or various other feature selection methods.

Identification of high-risk genes and classification of acute myocardial infarction patients utilizing deep learning in a restricted cohort.

Day by day the number of text documents in digital form is increasing. Text classification is used to organize these text documents. However, text classification has the problem of high dimensionality of feature space. This high dimensionality of feature space is solved by feature selection and feature extraction methods and improves the performance of text categorization. The feature selection and feature extraction techniques remove the irrelevant features from the text documents and reduce the dimensionality of feature space. This paper presents the various feature selection and feature extraction methods. This paper also discusses various classifiers to classify the documents.