Saliency-Based Multilabel Linear Discriminant Analysis.

Linear discriminant analysis (LDA) is a classical method for discriminative dimensionality reduction. The original LDA may degrade in its performance for non-Gaussian data, and may be unable to extract sufficient features to satisfactorily explain the data when the number of classes is small. Two prominent extensions to address these problems are subclass discriminant analysis (SDA) and mixture subclass discriminant analysis (MSDA). They divide every class into subclasses and re-define the within-class and between-class scatter matrices on the basis of subclass. In this paper we study the issue of how to obtain subclasses more effectively in order to achieve higher class separation. We observe that there is significant overlap between models of the subclasses, which we hypothesise is undesirable. In order to reduce their overlap we propose an extension of LDA, separability oriented subclass discriminant analysis (SSDA), which employs hierarchical clustering to divide a class into subclasses using a separability oriented criterion, before applying LDA optimisation using re-defined scatter matrices. Extensive experiments have shown that SSDA has better performance than LDA, SDA and MSDA in most cases. Additional experiments have further shown that SSDA can project data into LDA space that has higher class separation than LDA, SDA and MSDA in most cases.

A new face recognition method is proposed in this paper. The proposed method is based on fuzzy regularized linear discriminant analysis (FR-LDA) and combines the regularized linear discriminant analysis (R-LDA) and the fuzzy set theory. R-LDA is based on a new regularized Fisher's discriminant criterion, which is particularly robust against the small sample size problem compared to the traditional one used in LDA. In the proposed method, we calculate the membership degree matrix by Fuzzy K-nearest neighbor (FKNN) and then incorporate the membership degree into the definition of the between-class and within-class scatter matrices and get the fuzzy between-class and within-class scatter matrices. Experimental results obtained on the FERET database demonstrate that the proposed method improves the classification rate performance.

Local Discriminant Embedding (LDE) was recently proposed to overcome some limitations of the global Linear Discriminant Analysis (LDA) method. In the case of a small training data set, however, LDE cannot directly be applied to high-dimensional data. This case is the so-called small-sample-size (SSS) problem. In this paper, we introduce an Exponential Local Discriminant Embedding (ELDE) technique to overcome the SSS problem. The advantages of ELDE are that, compared with Principal Component Analysis (PCA) + LDE, the ELDE method can extract the most discriminant information that was contained in the null space of the locality preserving between-class and within-class scatter matrices. In addition, the proposed ELDE is equivalent to transforming original data into a new space by distance diffusion mapping, and then, LDE is applied in such a new space. As a result of diffusion mapping, the margin between samples belonging to different classes is enlarged, which is helpful in improving classification accuracy. The experiments are conducted on four public face databases, YALE, PIE, Extended Yale and PF01. Experiments conducted on real data show that the performance of ELDE is better than that of LDE and many state-of-the-art discriminant analysis techniques.

Subspace selection is widely applied in data classification, clustering, and visualization. The samples projected into subspace can be processed efficiently. In this paper, we research the linear discriminant analysis (LDA) and maximum margin criterion (MMC) algorithms intensively and analyze the effects of scatters to subspace selection. Meanwhile, we point out the boundaries of scatters in LDA and MMC algorithms to illustrate the differences and similarities of subspace selection in different circumstances. Besides, the effects of outlier classes on subspace selection are also analyzed. According to the above analysis, we propose a new subspace selection method called angle linear discriminant embedding (ALDE) on the basis of angle measurement. ALDE utilizes the cosine of the angle to get new within-class and between-class scatter matrices and avoids the small sample size problem simultaneously. To deal with high-dimensional data, we extend ALDE to a two-stage ALDE (TS-ALDE). The synthetic data experiments indicate that ALDE can balance the within-class and between-class scatters and be robust to outlier classes. The experimental results based on UCI machine-learning repository and image databases show that TS-ALDE has a lower time complexity than ALDE while processing high-dimensional data.

Principal component analysis (PCA) and linear discriminant analysis (LDA) have been extended to be a group of classical methods in dimensionality reduction for unsupervised and supervised learning, respectively. However, compared with the PCA, the LDA loses several advantages because of the singularity of its between-class scatter, resulting in singular mapping and restriction of reduced dimension. In this paper, we propose a dimensionality reduction method by defining a full-rank between-class scatter, called reversible discriminant analysis (RDA). Based on the new defined between-class scatter matrix, our RDA obtains a nonsingular mapping. Thus, RDA can reduce the sample space to arbitrary dimension and the mapped sample can be recovered. RDA is also extended to kernel based dimensionality reduction. In addition, PCA and LDA are the special cases of our RDA. Experiments on the benchmark and real problems confirm the effectiveness of the proposed method.

Sequence describes the primary structure of a protein, which contains important structural, characteristic, and genetic information and thereby motivates many sequence-based computational approaches to infer protein function. Among them, feature-base approaches attract increased attention because they make prediction from a set of transformed and more biologically meaningful sequence features. However, original features extracted from sequence are usually of high dimensionality and often compromised by irrelevant patterns, therefore dimension reduction is necessary prior to classification for efficient and effective protein function prediction. A protein usually performs several different functions within an organism, which makes protein function prediction a multi-label classification problem. In machine learning, multi-label classification deals with problems where each object may belong to more than one class. As a well-known feature reduction method, linear discriminant analysis (LDA) has been successfully applied in many practical applications. It, however, by nature is designed for single-label classification, in which each object can belong to exactly one class. Because directly applying LDA in multi-label classification causes ambiguity when computing scatters matrices, we apply a new Multi-label Linear Discriminant Analysis (MLDA) approach to address this problem and meanwhile preserve powerful classification capability inherited from classical LDA. We further extend MLDA by l1-normalization to overcome the problem of over-counting data points with multiple labels. In addition, we incorporate biological network data using Laplacian embedding into our method, and assess the reliability of predicted putative functions. Extensive empirical evaluations demonstrate promising results of our methods.

Improved PCA + LDA Applies to Gastric Cancer Image Classification Process

The purpose of this study was to investigate approaches for combining information from the MLO and CC mammographic views for Computer-aided Diagnosis (CADx) algorithms. Feature level and classifier output level combinations were explored. Linear discriminant analysis (LDA) with step-wise feature selection from a set of Haralick's texture features was used to develop classifiers for distinguishing between benign and malignant mammographic lesions. The effect of correlation between features from the two views on the performance of classifiers was investigated. The single view models included: (a) an LDA model with stepwise selection based on the MLO view only (MLO-Only) and similarly (b) a CC-Only LDA model. The feature-level combination models included: (a) LDA based on concatenation of feature sets selected independently from the two views (FEAT_CON), (b) LDA based on the concatenated feature sets along with the corresponding value of each feature from the opposite view (FEAT_COR_CON) if the correlation was below a threshold, (c) LDA based on the average of the MLO and CC feature values (FEAT_AVG). The classifier output level combination models investigated included: (a) average of the outputs of the MLO-Only and CC-Only classifiers (OUTPUT_AVG), (b) maximum of the outputs of the MLO-Only and CC-Only classifiers (OUTPUT_MAX), (c) minimum of the outputs of the MLO-Only and CC-Only classifiers (OUTPUT_MIN), (d) a second level LDA classifier on the outputs of the MLO-Only and CC-Only classifiers (OUTPUT_LDA), (e) product of the output values of the two classifiers (OUTPUT_PROD). The performance of the models was assessed and compared using the ROC methodology to determine if combination models performed better than the single-view models.

Linear Discriminant Analysis (LDA) is a classical supervised dimensionality reduction algorithm. However, LDA focuses more on global structure and overly depends on reliable data labels. For data with outliers and nonlinear structures, LDA cannot effectively capture the true structure of the data. Moreover, the subspace dimension learned by LDA must be smaller than cluster number, which limits its practical applications. To address these issues, we propose a novel unsupervised LDA method that combines centerless K-means and LDA. This method eliminates the need to calculate cluster centroids and improves model robustness. By fusing centerless K-means and LDA into a unified framework and deducing the connection between K-means and manifold learning, this method captures the local manifold structure and discriminative structure. Additionally, the dimensionality of the subspace is not restricted. This method not only overcomes the limitations of traditional LDA but also improves the model’s adaptability to complex data. Extensive experiments on seven datasets demonstrate the effectiveness of the proposed method.

A new face recognition method via semi-discrete decomposition for one sample problem

Linear Discriminant Analysis (LDA) is a popular statistical method for both feature extraction and dimensionality reduction in face recognition. The major drawback of LDA is the so-called small sample size (3S) problem. This problem always occurs when the total number of training samples is smaller than the dimension of feature space. Under this situation, the within-class scatter matrix Sw becomes singular and LDA approach cannot be implemented directly. To overcome the 3S problem, this paper proposes a novel wavelet-face based subspace LDA algorithm. Wavelet-face feature extraction and dimensionality reduction are based on two-level D4-filter wavelet transform and discarding the null space of total class scatter matrix St. It is shown that our obtained projection matrix satisfies the uncorrelated constraint conditions. Hence in the sense of statistical uncorrelation, this projection matrix is optimal. The proposed method for face recognition has been evaluated with two public available databases, namely ORL and FERET databases. Comparing with existing LDA-based methods to solve the 3S problem, our method gives the best performance.

One of the main problems that face Motor Imagery-based system is addressing multi-class problem. Various approaches have been used to tackle this problem. Most of these approaches tend to divide multi-class problem into binary sub problems. This study aims to address the multi-class problem by comparing five multi-class approaches; One-vs-One (OVO), One-vs-Rest (OVR), Divide & Conquer (DC), Binary Hierarchy (BH), and Multi-class approaches. Renyi entropy was examined for feature extraction. Three linear classifiers were used to implement these five-approaches: Support Vector Machine (SVM), Multinomial Logistic Regression (MLR) and Linear Discriminant Analysis (LDA). These approaches were compared according to their performance and time consumption. The comparative results show that, Renyi entropy demonstrated its robustness not only as a feature extraction technique but also as a powerful dimension reduction technique, for multi-class problem. In addition, LDA proved to be the best classifier for almost all approaches with minimum execution time.

Linear Discriminant Analysis (LDA) is a well-known technique for feature extraction and dimension reduction. The performance of classical LDA however, significantly degrades on the High Dimension Low Sample Size (HDLSS) data for the ill-posed inverse problem. Existing approaches for HDLSS data classification typically assume the data in question are with Gaussian distribution and deal the HDLSS classification problem with regularization. However, these assumptions are too strict to hold in many emerging real-life applications, such as enabling personalized predictive analysis using Electronic Health Records (EHRs) data collected from an extremely limited number of patients who have been diagnosed with or without the target disease for prediction. In this paper, we revised the problem of predictive analysis of disease using personal EHR data and LDA classifier. To fill the gap, in this paper, we first studied an analytical model that understands the accuracy of LDA for classifying data with arbitrary distribution. The model gives a theoretical upper bound of LDA error rate that is controlled by two factors: (1) the statistical convergence rate of (inverse) covariance matrix estimators and (2) the divergence of the training/testing datasets to fitted distributions. To this end, we could lower the error rate by balancing the two factors for better classification performance. Hereby, we further proposed a novel LDA classifier De-Sparse that leverages De-sparsified Graphical Lasso to improve the estimation of LDA, which outperforms state-of-the-art LDA approaches developed for HDLSS data. Such advances and effectiveness are further demonstrated by both theoretical analysis and extensive experiments on EHR datasets https://www.overleaf.com/project/5d2728c718f6ff3b2bcf5991 .

Dimensionality reduction is one of the important preprocessing steps to handle high-dimensional data. Linear discriminant analysis (LDA) is a classical and popular approach for this purpose. LDA finds an optimal linear transformation, which maximizes the ratio of the variance in the between-class distance to the variance in the within-class distance. On the other hand, in order to overcome the limitation in LDA resulting from the assumption of equal covariance, several heteroscedastic extensions, such as heteroscedastic discriminant analysis (HDA), have been proposed. However, it is difficult to find one particular criterion suitable for any kind of data set in carrying out dimensionality reduction while preserving discriminative information. In this paper, we propose a new generalized framework which we call power linear discriminant analysis (PLDA). PLDA can describe various criteria including LDA and HDA with one parameter. Numerical results show that the PLDA is effective for various data sets.

Linear Discriminant Analysis (LDA) has been successfully applied to face recognition which is based on a linear projection from the image space to a low dimensional space by maximizing the between class scatter and minimizing the within-class scatter. LDA allows objective evaluation of the significance of visual information in different features of the face for identifying the human face. The LDA also provides us with a small set of features that carry the most relevant information for classification purposes. LDA method overcomes the limitation of Principle Component Analysis method by applying the linear discriminant criterion. This criterion tries to maximize the ratio of determinant of the between-class scatter matrix of the projected samples to the determinant of the within-class scatter matrix of the projected samples. Linear discriminant groups the images of the same class and separate images of different classes. Here to identify an input test image, the projected test image is compared to each projected training, and the test image is identified as the closest training image. The experiments in this paper we present to use LDA for face recognition. The experiments in this paper are performed with the ORL face database. The experimental results show that the correct recognition rate of this method is higher than that of previous techniques.