Regularized linear discriminant analysis and its recursive implementation for gene subset selection

Boosting is a general method for improving the accuracy of any given learning algorithm. In this paper, we have proposed the boosting method for face recognition (FR) that improves the linear discriminant analysis (LDA)-based technique. The improvement is achieved by incorporating the regularized LDA (R-LDA) technique into the boosting framework. R-LDA is based on a new regularized Fisher’s discriminant criterion, which is particularly robust against the small sample size problem compared to the traditional one used in LDA. The AdaBoost technique is utilized within this framework to generalize a set of simple FR subproblems and their corresponding LDA solutions and combines the results from the multiple, relatively weak, LDA solutions to form a strong solution. The comparative experimental result on FERET database demonstrates that the proposed boosting method achieves more accurate results over the individual algorithms.

Fisher linear discriminant analysis (LDA) involves within-class and between-class covariance matrices. For 2-D data such as images, regularized LDA (RLDA) can improve LDA due to the regularized eigenvalues of the estimated within-class matrix. However, it fails to consider the eigenvectors and the estimated between-class matrix. To improve these two matrices simultaneously, we propose in this paper a new two-stage method for 2-D data, namely a bidirectional LDA (BLDA) in the first stage and the RLDA in the second stage, where both BLDA and RLDA are based on the Fisher criterion that tackles correlation. BLDA performs the LDA under special separable covariance constraints that incorporate the row and column correlations inherent in 2-D data. The main novelty is that we propose a simple but effective statistical test to determine the subspace dimensionality in the first stage. As a result, the first stage reduces the dimensionality substantially while keeping the significant discriminant information in the data. This enables the second stage to perform RLDA in a much lower dimensional subspace, and thus improves the two estimated matrices simultaneously. Experiments on a number of 2-D synthetic and real-world data sets show that BLDA+RLDA outperforms several closely related competitors.

A new face recognition method is proposed in this paper. The proposed method is based on fuzzy regularized linear discriminant analysis (FR-LDA) and combines the regularized linear discriminant analysis (R-LDA) and the fuzzy set theory. R-LDA is based on a new regularized Fisher's discriminant criterion, which is particularly robust against the small sample size problem compared to the traditional one used in LDA. In the proposed method, we calculate the membership degree matrix by Fuzzy K-nearest neighbor (FKNN) and then incorporate the membership degree into the definition of the between-class and within-class scatter matrices and get the fuzzy between-class and within-class scatter matrices. Experimental results obtained on the FERET database demonstrate that the proposed method improves the classification rate performance.

Background The diagnosis of chronic rhinosinusitis (CRS) is a complicated procedure. An electronic nose (eNose) is a novel method that detects disease from gas-phase mixtures, such as human breath. Aims/Objectives To determine whether an eNose based on differential mobility spectrometry (DMS) can detect chronic rhinosinusitis without nasal polyps (CRSsNP) by analyzing aspirated nasal air. Materials and methods Adult patients with CRSsNP were examined. The control group consisted of patients with septal deviation. Nasal air was aspirated into a collection bag and analyzed with DMS. The DMS data were classified using regularized linear discriminant analysis (LDA) models with 10-fold cross-validation. Results The accuracy of the DMS to distinguish CRSsNP from patients with septal deviation was 69%. Sensitivity and specificity were 67 and 70%, respectively. Bonferroni-corrected statistical differences were clearly noted. When a subgroup with more severe inflammatory disease was compared to controls, the classification accuracy increased to 82%. Conclusions The results of this feasibility study demonstrate that CRSsNP can potentially be differentiated distinguished from patients with similar nasal symptoms by analyzing the aspirated nasal air using DMS. Further research is warranted to evaluate the ability of this novel method in the differential diagnostics of CRS.

Although mostly used for pattern classification, linear discriminant analysis (LDA) can also be used in feature selection as an effective measure to evaluate the separative ability of a feature subset. When applied to feature selection on high-dimensional small-sized (HDSS) data (generally) with class-imbalance, LDA encounters four problems, including singularity of scatter matrix, overfitting, overwhelming and prohibitively computational complexity. In this study, we propose the LDA-based feature selection method minority class emphasized linear discriminant analysis (MCE-LDA) with a new regularization technique to address the first three problems. Different to giving equal or more emphasis to majority class in conventional forms of regularization, the proposed regularization emphasizes more on minority class, with the expectation of improving overall performance by alleviating overwhelming of majority class to minority class as well as overfitting in minority class. In order to reduce computational overhead, an incremental implementation of LDA-based feature selection has been introduced. Comparative studies with other forms of regularization to LDA as well as with other popular feature selection methods on five HDSS problems show that MCE-LDA can produce feature subsets with excellent performance in both classification and robustness. Further experimental results of true positive rate (TPR) and true negative rate (TNR) have also verified the effectiveness of the proposed technique in alleviating overwhelming and overfitting problems.

Biomedical spectra, such as those acquired from magnetic resonance (MR) spectrometers, often have the characteristics of high dimensionality and small sample size. These two characteristics make the classification of such spectra difficult. Hierarchical clustering produces robust clustering results, especially when working on small size high-dimensional datasets. The goal of this research is to investigate the effectiveness of hierarchical clustering for the classification of high-dimensional biomedical spectra. The classification results are benchmarked against linear discriminant analysis (LDA).

Classification in high dimensions has been highlighted for the past two decades since Fisher's linear discriminant analysis (LDA) is not optimal in a smaller sample size n comparing the number of covariates p, i.e. p>n, which is mostly due to the singularity of the sample covariance matrix. Rather than modifying how to estimate the sample covariance and sample mean vector in constructing a classifier, we build two types of high-dimensional classifiers using data splitting, i.e. single data splitting (SDS) and multiple data splitting (MDS). Moreover, we introduce a weighted version of MDS classifier that improves classification performance as illustrated in numerical studies. Each of the split data sets has a smaller size of covariates compared to the sample size so that LDA is applicable, and classification results can be combined with respect to minimizing the misclassification rate. We present theoretical justification backing up our proposed methods by comparing misclassification rates with LDA in high dimension. We also conduct a wide range of simulations and analyse four microarray data sets, which demonstrates that our proposed methods outperform some existing methods or at least yield comparable performances.

Linear discriminant analysis (LDA) is one of the important techniques for dimensionality reduction, machine learning, and pattern recognition. However, in many applications, applying the classical LDA often faces the following problems: (1) sensitivity to outliers, (2) absence of local geometric information, and (3) small sample size or matrix singularity that can result in weak robustness and efficiency. Although several researchers have attempted to address one or more of the problems, little work has been done to address all of them together to produce a more effective and efficient LDA algorithm. This article proposes 3E-LDA, an enhanced LDA algorithm, that deals with all three problems as an attempt to further improve LDA. It proposes to learn a weighted median rather than the mean of the samples to deal with (1), to embed both between-class and within-class local geometric information to deal with (2), and to calculate the projection vectors in the null space of the matrix to deal with (3). Experiments on six benchmark datasets show that these three enhancements enable 3E-LDA to markedly outperform state-of-the-art LDA baselines in both accuracy and efficiency.

A novel feature dimensionality reduction strategy based on kernel regularized relevance weighted discriminant analysis is proposed in this paper with some interesting characteristics. First, the proposed method has shown its effectiveness in dealing with a small sample size problem when using the regularized linear discriminant analysis (RLDA) technique and Kernel theory. Second, a new computation method is proposed to solve the complicated and inefficient computation procedure problem in the traditional RLDA technique while using the cross-validation method. The experimental results indicate that the proposed algorithm shows better performance than the other methods.

Complexity-reduced implementations of complete and null-space-based linear discriminant analysis

Linear Discriminant Analysis (LDA) is a well-known technique for feature extraction and dimension reduction. The performance of classical LDA however, significantly degrades on the High Dimension Low Sample Size (HDLSS) data for the ill-posed inverse problem. Existing approaches for HDLSS data classification typically assume the data in question are with Gaussian distribution and deal the HDLSS classification problem with regularization. However, these assumptions are too strict to hold in many emerging real-life applications, such as enabling personalized predictive analysis using Electronic Health Records (EHRs) data collected from an extremely limited number of patients who have been diagnosed with or without the target disease for prediction. In this paper, we revised the problem of predictive analysis of disease using personal EHR data and LDA classifier. To fill the gap, in this paper, we first studied an analytical model that understands the accuracy of LDA for classifying data with arbitrary distribution. The model gives a theoretical upper bound of LDA error rate that is controlled by two factors: (1) the statistical convergence rate of (inverse) covariance matrix estimators and (2) the divergence of the training/testing datasets to fitted distributions. To this end, we could lower the error rate by balancing the two factors for better classification performance. Hereby, we further proposed a novel LDA classifier De-Sparse that leverages De-sparsified Graphical Lasso to improve the estimation of LDA, which outperforms state-of-the-art LDA approaches developed for HDLSS data. Such advances and effectiveness are further demonstrated by both theoretical analysis and extensive experiments on EHR datasets https://www.overleaf.com/project/5d2728c718f6ff3b2bcf5991 .

Regularized least squares fisher linear discriminant with applications to image recognition

Objective. To date, most research on electroencephalography (EEG)-based mental workload detection for passive brain-computer interface (pBCI) applications has focused on identifying the overall level of cognitive resources required, such as whether the workload is high or low. We propose, however, that being able to determine the specific type of cognitive resources being used, such as visual or auditory, would also be useful. This would enable the pBCI to take more appropriate action to reduce the overall level of cognitive demand on the user. For example, if a high level of workload was detected and it is determined that the user is primarily engaged in visual information processing, then the pBCI could cause some information to be presented aurally instead. In our previous work we showed that EEG could be used to differentiate visual from auditory processing tasks when the level of processing is high, but the two modalities could not be distinguished when the level of cognitive processing demand was very low. The current study aims to build on this work and move toward the overall objective of developing a pBCI that is capable of predicting both the level and the type of cognitive resources being used. Approach. Fifteen individuals undertook carefully designed visual and auditory tasks while their EEG data was being recorded. In this study, we incorporated a more diverse range of sensory processing conditions including not only single-modality conditions (i.e. those requiring one of either visual or auditory processing) as in our previous study, but also dual-modality conditions (i.e. those requiring both visual and auditory processing) and no-task/baseline conditions (i.e. when the individual is not engaged in either visual or auditory processing). Main results. Using regularized linear discriminant analysis within a hierarchical classification algorithm, the overall cognitive demand was predicted with an accuracy of more than 86%, while the presence or absence of visual and auditory sensory processing were each predicted with an accuracy of approximately 70%. Significance. The findings support the feasibility of establishing a pBCI that can determine both the level and type of attentional resources required by the user at any given moment. This pBCI could assist in enhancing safety in hazardous jobs by triggering the most effective and efficient adaptation strategies when high workload conditions are detected.

Feature extraction is a very important preprocessing step for classification of hyperspectral images. The linear discriminant analysis (LDA) method fails to work in small sample size situations. Moreover, LDA has poor efficiency for non-Gaussian data. LDA is optimized by a global criterion. Thus, it is not sufficiently flexible to cope with the multi-modal distributed data. We propose a new feature extraction method in this paper, which uses the boundary semi-labeled samples for solving small sample size problem. The proposed method, which called hybrid feature extraction based on boundary semi-labeled samples (HFE-BSL), uses a hybrid criterion that integrates both the local and global criteria for feature extraction. Thus, it is robust and flexible. The experimental results with three real hyperspectral images show the good efficiency of HFE-BSL compared to some popular and state-of-the-art feature extraction methods.

Linear Discriminant Analysis (LDA) has been one of the popular subspace methods for face recognition. But this method suffers from the small sample size (SSS) problem, also known as 'curse of dimensionality'. Various techniques have been proposed in literature to overcome this limitation. But it is still unclear which method provides the best solution to SSS problem. In this paper, we have investigated the performance of some popular subspace methods such as principal component analysis (PCA), PCA + LDA, LDA via QR decomposition, Null-space LDA, Exponential Discriminant Analysis (EDA), PCA+EDA etc. Extensive experiments have been performed on five publically available face datasets viz. AR, CMU-PIE, PIX, Yale and YaleB. The performance is measured in terms of average classification accuracy. Experimental results show that the performance increases with the increase in the number of images per person in training set irrespective of the datasets. There is no clear winner among the subspace methods under investigation. But, the performance of PCA+LDA and PCA+EDA is consistent in tackling SSS problem irrespective of the dataset and can also handle the illumination variation in face recognition.