Discriminative analysis of dimensionality reduction methods for pattern recognition

We introduce two enhanced Fisher linear discriminant (FLD) models (EFM) in order to improve the generalization ability of the standard FLD based classifiers such as Fisherfaces. Similar to Fisherfaces, both EFM models apply first principal component analysis (PCA) for dimensionality reduction before proceeding with FLD type of analysis. EFM-1 implements the dimensionality reduction with the goal to balance between the need that the selected eigenvalues account for most of the spectral energy of the raw data and the requirement that the eigenvalues of the within-class scatter matrix in the reduced PCA subspace are not too small. EFM-2 implements the dimensionality reduction as Fisherfaces do. It proceeds with the whitening of the within-class scatter matrix in the reduced PCA subspace and then chooses a small set of features (corresponding to the eigenvectors of the within-class scatter matrix) so that the smaller trailing eigenvalues are not included in further computation of the between-class scatter matrix. Experimental data using a large set of faces-1,107 images drawn from 369 subjects and including duplicates acquired at a later time under different illumination-from the FERET database shows that the EFM models outperform the standard FLD based methods.

UAVs (unmanned aerial vehicles) have been widely used in many fields, where they need to be detected and controlled. Small-sample UAV recognition requires an effective detecting and recognition method. When identifying a UAV target using the backward propagation (BP) neural network, fully connected neurons of BP neural network and the high-dimensional input features will generate too many weights for training, induce complex network structure, and poor recognition performance. In this paper, a novel recognition method based on non-negative matrix factorization (NMF) with sparseness constraint feature dimension reduction and BP neural network is proposed for the above difficulties. The Edgeboxes are used for candidate regions and Log-Gabor features are extracted in candidate target regions. In order to avoid the complexity of the matrix operation with the high-dimensional Log-Gabor features, preprocessing for feature reduction by downsampling is adopted, which makes the NMF fast and the feature discriminative. The classifier is trained by neural network with the feature of dimension reduction. The experimental results show that the method is better than the traditional methods of dimension reduction, such as PCA (principal component analysis), FLD (Fisher linear discrimination), LPP (locality preserving projection), and KLPP (kernel locality preserving projection), and can identify the UAV target quickly and accurately.

This research features a new method for automatic face recognition robust to variations in lighting, facial expression and eyewear. The new algorithm named SKKUfaces (Sungkyunkwan University faces) employs PCA (Principal Component Analysis) and FLD (Fisher's Linear Discriminant) in series similarly to Fisherfaces. The fundamental difference is that SKKUfaces effectively eliminates, in the reduced PCA subspace, portions of the subspace that are responsible for variations in lighting and facial expression and then applies FLD to the resulting subspace. This results in superb discriminating power for pattern classification and excellent recognition accuracy. We also propose an efficient method to compute the between-class scatter and within-class scatter matrices for the FLD analysis. We have evaluated the performance of SKKUfaces using YALE and SKKU facial databases. Experimental results show that the SKKUface method is computationally efficient and achieves much better recognition accuracy than the Fisherface method [1] especially for facial images with variations in lighting and eyewear.

Recently, many dimensionality reduction algorithms, including local methods and global methods, have been presented. The representative local linear methods are locally linear embedding (LLE) and linear preserving projections (LPP), which seek to find an embedding space that preserves local information to explore the intrinsic characteristics of high dimensional data. However, both of them still fail to nicely deal with the sparsely sampled or noise contaminated datasets, where the local neighborhood structure is critically distorted. On the contrary, principal component analysis (PCA), the most frequently used global method, preserves the total variance by maximizing the trace of feature variance matrix. But PCA cannot preserve local information due to pursuing maximal variance. In order to integrate the locality and globality together and avoid the drawback in LLE and PCA, in this paper, inspired by the dimensionality reduction methods of LLE and PCA, we propose a new dimensionality reduction method for face recognition, namely, unsupervised linear difference projection (ULDP). This approach can be regarded as the integration of a local approach (LLE) and a global approach (PCA), so that it has better performance and robustness in applications. Experimental results on the ORL, YALE and AR face databases show the effectiveness of the proposed method on face recognition.

This paper carries on research surrounding the influences produced by dimensionality reduction on machine learning classification effect. Firstly, paper constructs the analysis architecture of data dimension reduction classification, combines the two different unsupervised dimension reduction methods, locally linear embedding (LLE) and principal component analysis (PCA) with the five machine learning classification methods: Gradient Boosting Decision Tree (GBDT), Random Forest, Support Vector Machine (SVM), K-Nearest Neighbor (KNN) and Logistic Regression. And then uses the handwritten digital identification dataset to analyze the classification performance of these five classification methods on different dimension datasets by different dimensionality reduction methods. The analysis shows that using the appropriate dimensionality reduction method for dimensionality reduction classification can effectively improve the classification accuracy; the dimensionality reduction classification effect of non-linear dimensionality reduction method is generally better than the linear dimensionality reduction method; different machine learning classification algorithms have significant differences in the sensitivity of dimensions.

Dimensionality reduction is the process of mapping high-dimension patterns to a lower dimension subspace. When done prior to classification, estimates obtained in the lower dimension subspace are more reliable. For some classifiers, there is also an improvement in performance due to the removal of the diluting effect of redundant information. A majority of the present approaches to dimensionality reduction are based on scatter matrices or other statistics of the data which do not directly correlate to classification accuracy. The optimality criteria of choice for the purposes of classification is the Bayes error. Usually however, Bayes error is difficult to express analytically. We propose an optimality criteria based on an approximation of the Bayes error and use it to formulate a linear and a nonlinear method of dimensionality reduction. The nonlinear method we propose, relies on using a multilayered perceptron which produces as output the lower dimensional representation. It thus differs from autoassociative like multilayered perceptrons which have been proposed and used for dimensionality reduction. Our results show that the nonlinear method is, as anticipated, superior to the linear method in that it can perform unfolding of a nonlinear manifold. In addition, the nonlinear method we propose provides substantially better lower dimension representation (for classification purposes) than Fisher's linear discriminant (FLD) and two other nonlinear methods of dimensionality reduction that are often used.

Face recognition is one of the most fundamental functions for many kinds of intelligent robots. Among many methods proposed for face recognition, linear approaches such as Eigenface or principal component analysis (PCA), Fisher’s linear discriminant (FLD), Fisherface, and common vector (CV), have attracted great attention because of their simplicity. It is known that when the number of training examples is large enough, we should use FLD; otherwise, we should use CV. This paper proposes a rough common vector (RCV) approach. The basic idea of RCV is to divide the feature space into two subspaces. One is spanned by the eigenvectors corresponding to the largest eigenvalues of the within-class scatter matrix, and another is spanned by the eigenvectors corresponding to the smallest eigenvalues. The later plays the role of the null space of the within-class scatter matrix, and is important for extracting useful discriminative features for recognition. RCV can be used regardless the within-class scatter matrix is singular or not. Experimental results on four databases show that RCV outperforms nearest neighbor classifier, Eigenface, Fisherface, and CV in most cases.

Dimensionality reduction is the process of mapping high-dimension patterns to a lower dimension subspace. When done prior to classification, estimates obtained in the lower dimension subspace are more reliable. We propose a novel method based on graphical multivariate feature fusion and use it to offer a visual representation of high dimensional data. The graphical processing method we propose, relies on using a multilayered structure of feature fusion which produces as output of the lower dimensional representation. We implement feature fusion by combining method of feature selection and feature extraction. Experiments on the data set of machine learning database indicate the novel method we propose provides better representation than Fisher's linear discriminant (FLD) and some other nonlinear methods of dimensionality reduction that are often used.

During the COVID-19 pandemic, researchers have made efforts to detect COVID-19 through various methods. In the dataset used for this study, COVID-19 patients were identified using chest computed tomography (CT) images. High dimensionality is frequently an issue in machine learning image classification. Accordingly, this study implemented three dimensionality reduction methods in combination with various machine learning algorithms for improved classification. Principal component analysis (PCA), uniform manifold approximation and projection (UMAP), and diffusion maps were applied to the dataset to extract the most important features of the chest CT images. The extracted features were given as input either to logistic regression or the extreme gradient boosting (XGBoost) algorithm to perform classification. The strongest model identified from this study was diffusion maps in combination with logistic regression. This model, evaluated against existing models from similar studies in recent years, yielded strong performance for detecting COVID-19 cases using chest CT images. Our proposed model achieved 97.35% accuracy, 92.16% sensitivity, and 98.59% specificity on the held-out test set in differentiating between COVID-19-positive cases and healthy, non-COVID-19 cases. This study aimed to detect COVID-19 without the use of viral testing. Importantly, this method could assist clinicians in making an initial diagnosis, especially when viral testing is not available or timely enough for the patient’s case. This study also provides deeper insight into various dimensionality reduction methods and how compatible they are with biomedical imaging data. Models were trained using stratified cross-validation on the training set, with final performance evaluated on a held-out test set at the patient level to prevent data leakage. Additional imbalance-aware metrics were used to assess robustness given class distribution differences.

Adaptive linear dimensionality reduction for classification

Principal Component Analysis (PCA) is a widely used dimensional reduction method that aims to find a low dimension sub space of highly correlated data for its major information to be used in further analysis. Machine learning methods based on PCA are popular in high dimensional data analysis, such as video and image processing. In video processing, the Robust PCA (RPCA), which is a modified method of the traditional PCA, has good properties in separating moving objects from the background, but it may have difficulties in separating those when light intensity of the background varies significantly in time. To overcome the difficulties, a modified PCA method, called Regression PCA (RegPCA), is proposed. The method is developed by combining the traditional PCA and regression approaches together, and it can be easily combined with RPCA for video processing. We focus the presentation of RegPCA with the combination of RPCA on video processing and find that it is more reliable than RPCA only. We use RegPCA to separate moving object from the background in a color video and get a better result than that given by RPCA. In the implementation, we first derive the explanatory variables by the background information. we then process a number of frames of the video and use those as a set of response variables. We remove the impact of the background by regressing the response against the explanatory variables by a regression model. The regression model provides a set of residuals, which can be further analyzed by RPCA. We compare the results of RegRPCA against those of RPCA only. It is evident that the moving objects can be completely removed from the background using our method but not in RPCA. Note that our result is based on a combination of RegPCA with RPCA. Our proposed method provides a new implementation of RPCA under the framework of regression approaches, which can be used to account for the impact of risk factors. This problem cannot be addressed by the application of RPCA only.

The aim of data transformation is to transform the original feature space of data into another space with better properties. This is typically combined with dimensionality reduction, so that the dimensionality of the transformed space is smaller. A widely used method for data transformation and dimensionality reduction is Principal Component Analysis (PCA). PCA finds a subspace that explains most of the data variance. While the new PCA feature space has interesting properties, such as removing linear correlation, PCA is an unsupervised method. Therefore, there is no guarantee that the PCA feature space will be the most appropriate for supervised tasks, such as classification or regression. On the other hand, 3-layer Multi Layer Perceptrons (MLP), which are supervised methods, can also be understood as a data transformation carried out by the hidden layer, followed by a classification/regression operation performed by the output layer. Given that the hidden layer is obtained after a supervised training process, it can be considered that it is performing a supervised data transformation. And if the number of hidden neurons is smaller than the input, also dimensionality reduction. Despite this kind of transformation being widely available (any neural network package that allows access to the hidden layer weights can be used), no extensive experimentation on the quality of 3-layer MLP data transformation has been carried out. The aim of this article is to carry out this research for classification problems. Results show that, overall, this transformation offers better results than the PCA unsupervised transformation method.

Fisher discriminant analysis (FDA) is a classic supervised dimensionality reduction method in statistical pattern recognition. FDA can maximize the scatter between different classes, while minimizing the scatter within each class. As it only utilizes the labeled data and ignores the unlabeled data in the analysis process of FDA, it cannot be used to solve the unsupervised learning problems. Its performance is also very poor in dealing with semi-supervised learning problems in some cases. Recently, several semi-supervised learning methods as an extension of FDA have proposed. Most of these methods solve the semi-supervised problem by using a tradeoff parameter that evaluates the ratio of the supervised and unsupervised methods. In this paper, we propose a general semi-supervised dimensionality learning idea for the partially labeled data, namely the reconstruction probability class of labeled and unlabeled data. Based on the probability class optimizes Fisher criterion function, we propose a novel Semi-Supervised Local Fisher Discriminant Analysis (S2LFDA) method. Experimental results on real-world datasets demonstrate its effectiveness compared to the existing similar correlation methods.

A kernel based nonlinear subspace projection (KNSP) method is proposed for reduction of hyperspectral image dimensionality. This method involves three steps: subspace partition of full data space, feature extraction based on kernel principal component analysis (KPCA) in subspace and feature selection based on class separability criterion. The main merit of the proposed method is that it is more suitable for feature extraction than linear principal component analysis (PCA) and segmented principal component: transformation (SPCT), in particular, when hyperspectral data have nonlinear characteristics. In order to testify the effectiveness of the KNSP method for reduction of hyperspectral image dimensionality, hyperspectral image classification is performed on AVIRIS data. Experimental results show that when the hyperspectral dimensionality is reduced to a few features, the average classification accuracy of the new method is higher than those of PCA and SPCT methods.

To generalize the Fisher Discriminant Analysis (FDA) algorithm to the case of discriminant functions belonging to a nonlinear, finite dimensional function space F (Nonlinear FDA or NFDA), it is sufficient to expand the input data by computing the output of a basis of F when applied to it [1,2,3,4]. The solution to NFDA can then be found like in the linear case by solving a generalized eigenvalue problem on the between- and within-classes covariance matrices (see e.g. [5]). The goal of NFDA is to find linear projections of the expanded data (i.e., nonlinear transformations of the original data) that minimize the variance within a class and maximize the variance between different classes. Such a representation is of course ideal to perform classification. The application of NFDA to pattern recognition is particularly appealing, because for a given input signal and a fixed function space it has no parameters and it is easy to implement and apply. Moreover, given C classes only C - 1 projections are relevant [5]. As a consequence, the feature space is very small and the algorithm has low memory requirements and high speed during recognition. Here we apply NFDA to a handwritten digit recognition problem using the MNIST database, a standard and freely available set of 70,000 handwritten digits (28 × 28 pixels large), divided into a training set (60,000 digits) and a test set (10,000 digits). Several established pattern recognition methods have been applied to this database by Le Cun et al. [6]. Their paper provides a standard reference work to benchmark new algorithms. We perform NFDA on spaces of polynomials of a given degree d, whose corresponding basis functions include all monomials up to order d in all input variables. It is clear that the problem quickly becomes intractable because of the high memory requirements. For this reason, the input dimensionality is first reduced by principal component analysis. On the preprocessed data we then apply NFDA by expanding the training patterns in the polynomial space and solving the linear FDA eigenvalue problem. As mentioned above, since we have 10 classes we only need to compute the first 9 eigenvectors. Since the within-class variance is minimized, the patterns belonging to different classes tend to cluster in the feature space when projected on the eigenvectors. For this reason we classify the digits with a simple method such as Gaussian classifiers. We perform simulations with polynomials of degree 2 to 5. With polynomials of degree 2 the explosion in the dimensionality of the expanded space with increasing number of input dimensions is relatively restricted, so that it is possible to use up to 140 dimensions. With higher order polynomials one has to rely on a smaller number of input dimensions, but since the function space gets larger and includes new nonlinearities, one obtains a remarkable improvement in performance. The best performance is achieved with polynomials of degree 3 and 35 input dimensions, with an error rate of 1.5% on test data. This error rate is comparable to but does not outperform that of the most elaborate algorithms (Table 1). The performance of NFDA is however remarkable considering the simplicity of the method and the fact that it has no a priori knowledge on the problem, in contrast for example to the LeNet-5 algorithm [6] which has been designed specifically for handwritten character recognition. In addition, for recognition, NFDA has to store and compute only 9 functions and has thus small memory requirements and a high recognition speed. It is also possible to formulate NFDA using the kernel trick, in which case one can in principle use function spaces of infinite dimensionality [1,2,3,4]. However, the limiting factor in that formulation is the number of training patterns, which makes it not realistic for this application. The performance of NFDA could be further improved using for example a more problem-specific preprocessing of the patterns (e.g., by increasing the size of the training set with new patterns generated by artificial distortion of the original one), boosting techniques, or mixture of experts with other algorithms [5,6].