An evolution and evaluation of dimensionality reduction techniques — A comparative study

Dimensionality reduction as a preprocessing step to machine learning is effective in removing irrelevant and redundant data, increasing learning accuracy, and improving result comprehensibility. However, the recent increase of dimensionality of data poses a severe challenge to many existing feature selection and feature extraction methods with respect to efficiency and effectiveness. In the field of machine learning and pattern recognition, dimensionality reduction is important area, where many approaches have been proposed. In this paper, some widely used feature selection and feature extraction techniques have analyzed with the purpose of how effectively these techniques can be used to achieve high performance of learning algorithms that ultimately improves predictive accuracy of classifier. An endeavor to analyze dimensionality reduction techniques briefly with the purpose to investigate strengths and weaknesses of some widely used dimensionality reduction methods is presented.

Dimensionality Reduction is one of the preferred techniques for addressing the problem of the curse of dimensionality associated with high-dimensional data. At present, various significant research works have been already carried out toward emphasizing the dimensional reduction methods with respect to projection-based, statistical-based, and dictionary-based. However, it is still an open question to explore the best technique of dimensional reduction. Hence, we present a compact summary of our investigation towards finding the contribution of existing research methods of dimensional reduction. The paper outlines most frequently adopted techniques of dimensional reduction. At the same time, this survey also emphasizes on exploring the problems addressed by the present researchers with an aid of their own techniques associated with both advantages, limitations, and addressing the issues of the curse of dimensionality. The survey also introduces the latest research progress and significant research gap associated with the existing literature.

Hyperspectral remote sensing offers a powerful tool in many different application contexts. The imbalance between the high dimensionality of the data and the limited availability of training samples calls for the need to perform dimensionality reduction in practice. Among traditional dimensionality reduction techniques, feature extraction is one of the most widely used approaches due to its flexibility to transform the original spectral information into a subspace. In turn, band selection is important when the application requires preserving the original spectral information (especially the physically meaningful information) for the interpretation of the hyperspectral scene. In the case of hyperspectral image classification, both techniques need to discard most of the original features/bands in order to perform the classification using a feature set with much lower dimensionality. However, the discriminative information that allows a classifier to provide good performance is usually classdependent and the relevant information may live in weak features/bands that are usually discarded or lost through subspace transformation or band selection. As a result, in practice, it is challenging to use either feature extraction or band selection for classification purposes. Relevant lines of attack to address this problem have focused on multiple feature selection aiming at a suitable fusion of diverse features in order to provide relevant information to the classifier. In this paper, we present a new dimensionality reduction technique, called multiple criteria-based spectral partitioning, which is embedded in an ensemble learning framework to perform advanced hyperspectral image classification. Driven by the use of a multiple band priority criteria that is derived from classic band selection techniques, we obtain multiple spectral partitions from the original hyperspectral data that correspond to several band subgroups with much lower spectral dimensionality as compared with the original band set. An ensemble learning technique is then used to fuse the information from multiple features, taking advantage of the relevant information provided by each classifier. Our experimental results with two real hyperspectral images, collected by the reflective optics system imaging spectrometer (ROSIS) over the University of Pavia in Italy and the Airborne Visible/Infrared Imaging Spectrometer (AVIRIS) over the Salinas scene, reveal that our presented method, driven by multiple band priority criteria, is able to obtain better classification results compared with classic band selection techniques. This paper also discusses several possibilities for computationally efficient implementation of the proposed technique using various high-performance computing architectures.

Classification of human hand movements based on EMG signals using nonlinear dimensionality reduction and data fusion techniques

There are a large number of high-dimensional data in the Internet of Brain Things, and the data is reduced from the high-dimensional to low-dimensional to maintain the similarity between the data, thereby effectively ensuring the operating speed of the Internet of Brain Things. In the traditional method, the distributed dimensionality reduction reconstruction algorithm of the High dimensional data has poor dimensionality reduction effect and serious data distortion after reconstruction. A method for dimensionality reconstruction of high-dimensional data in the Internet of Brain Things is proposed. By using the algorithm of linear discriminant analysis, the projection matrix of high dimensional data is constructed to solve it. According to the solution results, using improved implicit variable model to establish a high dimensional large data dimensionality reduction model for the Internet of Brain Things. The fitness value of data after dimensionality reduction is calculated by quantum immune clonal algorithm, and the optimal individual and optimal solution are determined. The data reconfiguration is realized through the optimal solution marshalling. Experimental results show that the proposed algorithm can effectively improve the dimensionality reduction of High dimensional data in the Internet of Brain Things. After reconstruction, the reconstructed data retain accurate data information, the reliability of reconstructed data is high, and the computational complexity is not high, the need for small storage space, and the advantages of strong promotion ability.

Digital medical images can be viewed as digital representations of real physical tissue properties that are convenient for handling, storing, transmitting, retrieving, and analyzing image information. In order to perform feature extraction/identification/classification from high-dimensional medical data, we need dimensionality reduction (DR), which can be achieved by identifying the statistical patterns in medical image data with highlighting their similarities and dissimilarities. Here, the dimension of medical images is the total number of pixels in the image. In DR, we try to find an useful representation of reduced dimensionality in high-dimensional data, which minimizes information loss by maximizing local data variance. Given medical image data, the key challenging issue is how can we efficiently extract a low-dimensional latent structure? Various dimension reduction techniques have been developed to process the high-dimensional data where the intrinsic dimensions are assumed to be much lower. In the very ideal case, data can be regressed linearly and DR can be performed by principal component analysis. This chapter explains the theories, principles, and practices of DR techniques.

In this paper, we discuss dimension reduction methods for 2D visualization of high dimensional clustered data. We propose a two-stage framework for visualizing such data based on dimension reduction methods. In the first stage, we obtain the reduced dimensional data by applying a supervised dimension reduction method such as linear discriminant analysis which preserves the original cluster structure in terms of its criteria. The resulting optimal reduced dimension depends on the optimization criteria and is often larger than 2. In the second stage, the dimension is further reduced to 2 for visualization purposes by another dimension reduction method such as principal component analysis. The role of the second-stage is to minimize the loss of information due to reducing the dimension all the way to 2. Using this framework, we propose several two-stage methods, and present their theoretical characteristics as well as experimental comparisons on both artificial and real-world text data sets.

The curse of dimensionality, i.e., the fact that feature spaces of increasing dimensionality with finite sample sizes tend to be empty, has given incentive to a plethora of research activities in various disciplines and diverse application fields, e.g., statistics or neural networks. Three major application fields are multivariate data classification, data analysis and data visualization. In this contribution, methods for dimensionality reduction from three decades of interdisciplinary research are browsed and their applicability in the above application domains is briefly discussed. Complementing techniques for ensuing interactive data visualization, data navigation and visual exploratory data analysis are presented, which exploit the remarkable human perceptive and associative capabilities for interactive visual exploratory data analysis and systematic recognition system design. The main focus of this paper is on the comparison of feature selection and feature extraction techniques and the potential benefit of their combination. Further, the interesting implications of dimensionality reduction for VLSI design and related area and power consumption are pointed out.

Due to the intricate relationship between different dimensions of high-dimensional data, subspace analysis is often conducted to decompose dimensions and give prominence to certain subsets of dimensions, i.e. subspaces. Exploring and comparing subspaces are important to reveal the underlying features of subspaces, as well as to portray the characteristics of individual dimensions. To date, most of the existing high-dimensional data exploration and analysis approaches rely on dimensionality reduction algorithms (e.g. principal component analysis and multi-dimensional scaling) to project high-dimensional data, or their subspaces, to two-dimensional space and employ scatterplots for visualization. However, the dimensionality reduction algorithms are sometimes difficult to fine-tune and scatterplots are not effective for comparative visualization, making subspace comparison hard to perform. In this article, we aggregate high-dimensional data or their subspaces by computing pair-wise distances between all data items and showing the distances with matrix visualizations to present the original high-dimensional data or subspaces. Our approach enables effective visual comparisons among subspaces, which allows users to further investigate the characteristics of individual dimensions by studying their behaviors in similar subspaces. Through subspace comparisons, we identify dominant, similar, and conforming dimensions in different subspace contexts of synthetic and real-world high-dimensional data sets. Additionally, we present a prototype that integrates parallel coordinates plot and matrix visualization for high-dimensional data exploration and incremental dimensionality analysis, which also allows users to further validate the dimension characterization results derived from the subspace comparisons.

In order to solve the problem of low accuracy of time dimension feature extraction and classification of high-dimensional large data streams, this paper proposes a time dimension feature extraction and classification algorithm of high-dimensional large data streams based on unsupervised learning. Analyze the trend of high-dimensional data flow changes under machine learning, and achieve dimensionality reduction of high-dimensional large traffic time dimensional data through local save projection. Analyze the spatial relationship between feature attributes and feature space, segment and fit high-dimensional big data streams and time dimensional feature data streams, further segment time dimensional sequences using sliding windows, and complete feature extraction through discrete dyadic wavelet transform. According to the clustering algorithm, cluster the time dimension feature data stream, calculate the cosine similarity of the feature data, model the time dimension feature stream of training samples, use the feature classification function to minimize the classification loss, and use unsupervised learning to achieve the final classification task. The test results show that this method can improve the temporal feature extraction and classification accuracy streams.

Soft label based Linear Discriminant Analysis for image recognition and retrieval

The early diagnosis of chronic diseases plays a vital role in the field of healthcare communities and biomedical, where it is necessary for detecting the disease at an initial phase to reduce the death rate. This paper investigates the use of feature selection, dimensionality reduction and classification techniques to predict and diagnose chronic disease. The appropriate selection of attributes plays a crucial role in improving the classification accuracy of the diagnosis systems. Additionally, dimensionality reduction techniques effectively improve the overall performance of the machine learning algorithms. On chronic disease databases, the classification techniques deliver efficient predictive results by developing intelligent, adaptive and automated system. Parallel and adaptive classification techniques are also analyzed in chronic disease diagnosis which is used to stimulate the classification procedure and to improve the computational cost and time. This survey article represents the overview of feature selection, dimensionality reduction and classification techniques and their inherent benefits and drawbacks.

Along with the rapid advancement of computer science and data engineering, the complexity and dimension of the data considered are bursting. With the increase of the data dimension, the performance of high dimensional index structure drops rapidly. And these high-dimensional data usually contains many redundant, its essence dimension often is much smaller than the original data, so the high-dimensional data processing question boils down to the related dimension reduction method to cut some less relevant data and reduce its dimension. This paper integrates the ameliorated supportive vector machine to propose the novel high-dimensional data dimension reduction algorithm. Our algorithm optimises the traditional methodologies from the three layers: 1) data cleaning and pre-processing steps are added to enhance the feasibility of the captured data; 2) SVM is modified to hold the feature of higher robustness; 3) application scenarios are built to test the performance. The experimental result proves that the proposed method outperforms compared with other state-of-theart algorithms theoretically and numerically.

A Review of Dimensionality Reduction Techniques for Efficient Computation

High-dimensional features often cause computational complexity and dimensionality curse. Feature selection and feature extraction are the two mainstream methods for dimensionality reduction. Feature selection but not feature extraction can preserve the critical information and maintain the physical meaning simultaneously. Herein, we proposed a dimensionality reduction method based on rough set theory (DRM-RST) for feature selection. We defined the hyperspectral image as a decision system, extracted the features as decision attributes, and selected the effective features based on information entropy. We used the Washington D.C. Mall dataset and New York dataset to evaluate the performance of DRM-RST on dimensionality reduction. Compared with full band classification, 184 or 185 redundant bands were removed in DRM-RST, respectively. DRM-RST achieved similar accuracy (overall accuracy >94%) by SVM classifier and reduced computing time by about 85%. We further compared the dimensionality reduction efficiency of DRM-RST against other popular methods, including ReliefF, Sequential Backward Elimination (SBE) and Information Gain (IG). The Producer’s accuracy (PA) and User’s accuracy (UA) of DRM-RST was greater than that of ReliefF and IG. DRM-RST showed greater stability of accuracy than SBE in dimensionality reduction when using for different datasets. Collectively, this study provides a new method for dimensionality reduction that can reduce computational complexity and alleviate dimensionality curse.