Robust Fisher Linear Discriminant Analysis with Generalized Correntropic Loss Function

A computer vision system for automatic recognition and classification of five varieties of plant leaves under controlled laboratory imaging conditions, comprising: 1–Cydonia oblonga (quince), 2–Eucalyptus camaldulensis dehn (river red gum), 3–Malus pumila (apple), 4–Pistacia atlantica (mt. Atlas mastic tree) and 5–Prunus armeniaca (apricot), is proposed. 516 tree leaves images were taken and 285 features computed from each object including shape features, color features, texture features based on the gray level co-occurrence matrix, texture descriptors based on histogram and moment invariants. Seven discriminant features were selected and input for classification purposes using three classifiers: hybrid artificial neural network–ant bee colony (ANN–ABC), hybrid artificial neural network–biogeography based optimization (ANN–BBO) and Fisher linear discriminant analysis (LDA). Mean correct classification rates (CCR), resulted in 94.04%, 89.23%, and 93.99%, for hybrid ANN–ABC; hybrid ANN–BBO; and LDA classifiers, respectively. Best classifier mean area under curve (AUC), mean sensitivity, and mean specificity, were computed for the five tree varieties under study, resulting in: 1–Cydonia oblonga (quince) 0.991 (ANN–ABC), 95.89% (ANN–ABC), 95.91% (ANN–ABC); 2–Eucalyptus camaldulensis dehn (river red gum) 1.00 (LDA), 100% (LDA), 100% (LDA); 3–Malus pumila (apple) 0.996 (LDA), 96.63% (LDA), 94.99% (LDA); 4–Pistacia atlantica (mt. Atlas mastic tree) 0.979 (LDA), 91.71% (LDA), 82.57% (LDA); and 5–Prunus armeniaca (apricot) 0.994 (LDA), 88.67% (LDA), 94.65% (LDA), respectively.

The microarray data classification problem is a recent complex pattern recognition problem. The most important goal in supervised classification of microarray data, is to select a small number of relevant genes from the initial data in order to obtain high predictive classification accuracy. With the framework of a hybrid filter-wrapper, we study in this paper the role of the multi-parent recombination operator. For this purpose, we introduce a Random Multi Parent crossover (RMPX) and we analyze their effects in a genetic algorithm (GA) which is combined with Fisher's Linear Discriminant Analysis (LDA). This hybrid algorithm has the major characteristic that the GA uses not only a LDA classifier in its fitness function, but also LDA's discriminant coefficients to integrate a multi-parent specialized crossover and mutation operation to improve the performance of gene selection. In the experimental results it is observed that RPMX operator work very well by achieving lower classification error rates.

Fisher linear discriminant analysis (LDA) involves within-class and between-class covariance matrices. For 2-D data such as images, regularized LDA (RLDA) can improve LDA due to the regularized eigenvalues of the estimated within-class matrix. However, it fails to consider the eigenvectors and the estimated between-class matrix. To improve these two matrices simultaneously, we propose in this paper a new two-stage method for 2-D data, namely a bidirectional LDA (BLDA) in the first stage and the RLDA in the second stage, where both BLDA and RLDA are based on the Fisher criterion that tackles correlation. BLDA performs the LDA under special separable covariance constraints that incorporate the row and column correlations inherent in 2-D data. The main novelty is that we propose a simple but effective statistical test to determine the subspace dimensionality in the first stage. As a result, the first stage reduces the dimensionality substantially while keeping the significant discriminant information in the data. This enables the second stage to perform RLDA in a much lower dimensional subspace, and thus improves the two estimated matrices simultaneously. Experiments on a number of 2-D synthetic and real-world data sets show that BLDA+RLDA outperforms several closely related competitors.

An automated wood recognition system is designed to classify tropical wood species. The wood features are extracted based on two feature extractors: Basic Grey Level Aura Matrix (BGLAM) technique and statistical properties of pores distribution (SPPD) technique. Due to the nonlinearity of the tropical wood species separation boundaries, a pre classification stage is proposed which consists of Kmeans clustering and kernel discriminant analysis (KDA). Finally, Linear Discriminant Analysis (LDA) classifier and K-Nearest Neighbour (KNN) are implemented for comparison purposes. The study involves comparison of the system with and without pre classification using KNN classifier and LDA classifier. The results show that the inclusion of the pre classification stage has improved the accuracy of both the LDA and KNN classifiers by more than 12%.

Linear Discriminant Analysis (LDA) is a well-known technique for feature extraction and dimension reduction. The performance of classical LDA however, significantly degrades on the High Dimension Low Sample Size (HDLSS) data for the ill-posed inverse problem. Existing approaches for HDLSS data classification typically assume the data in question are with Gaussian distribution and deal the HDLSS classification problem with regularization. However, these assumptions are too strict to hold in many emerging real-life applications, such as enabling personalized predictive analysis using Electronic Health Records (EHRs) data collected from an extremely limited number of patients who have been diagnosed with or without the target disease for prediction. In this paper, we revised the problem of predictive analysis of disease using personal EHR data and LDA classifier. To fill the gap, in this paper, we first studied an analytical model that understands the accuracy of LDA for classifying data with arbitrary distribution. The model gives a theoretical upper bound of LDA error rate that is controlled by two factors: (1) the statistical convergence rate of (inverse) covariance matrix estimators and (2) the divergence of the training/testing datasets to fitted distributions. To this end, we could lower the error rate by balancing the two factors for better classification performance. Hereby, we further proposed a novel LDA classifier De-Sparse that leverages De-sparsified Graphical Lasso to improve the estimation of LDA, which outperforms state-of-the-art LDA approaches developed for HDLSS data. Such advances and effectiveness are further demonstrated by both theoretical analysis and extensive experiments on EHR datasets https://www.overleaf.com/project/5d2728c718f6ff3b2bcf5991 .

In this paper, we investigate the application of Fisher’s linear discriminant analysis (FLDA) to hyperspectral remote sensing image classification. The basic idea of FLDA is to design an optimal transform so that the classes can be well separated in the low-dimensional space. The practical difficulty of applying FLDA to hyperspectral images includes the unavailability of enough training samples and unknown information for all the classes present. So the original FLDA is modified to avoid the requirements of complete class knowledge, such as the number of actual classes present. We also investigate the performance of the class of principal component analysis (PCA) techniques prior to FLDA and find that the interference and noise adjusted PCA (INAPCA) can provide the improvement in the final classification.KeywordsFisher’s Linear Discriminant AnalysisDimensionality ReductionClassificationHyperspectral Imagery

Gem-Pro is a new tool for gene mining and functional profiling of bacteria. It initially identifies homologous genes using BLAST and then applies three filtering steps to select orthologous gene pairs. The first one uses BLAST score values to identify trivial paralogs. The second filter uses the shared identity percentages of found trivial paralogs as internal witnesses of non-orthology to set orthology cutoff values. The third filtering step uses conditional probabilities of orthology and non-orthology to define new cutoffs and generate supportive information of orthology assignations. Additionally, a subsidiary tool, called q-GeM, was also developed to mine traits of interest using logistic regression (LR) or linear discriminant analysis (LDA) classifiers. q-GeM is more efficient in the use of computing resources than Gem-Pro but needs an initial classified set of homologous genes in order to train LR and LDA classifiers. Hence, q-GeM could be used to analyze new set of strains with available genome sequences, without the need to rerun a complete Gem-Pro analysis. Finally, Gem-Pro and q-GeM perform a synteny analysis to evaluate the integrity and genomic arrangement of specific pathways of interest to infer their presence. The tools were applied to more than 2 million homologous pairs encoded by Bacillus strains generating statistical supported predictions of trait contents. The different patterns of encoded traits of interest were successfully used to perform a descriptive bacterial profiling.

Cavitation is a phenomenon that often occurs in the centrifugal pumps. The impact of cavitation is a decrease in pump performance which will affect the ongoing production process in the industries. It is important to have a method to detect the phenomenon of cavitation early. The vibration signal is a parameter that is often used in detecting cavitation or other faulty components. One of the methods is based on the pattern recognition i.e. machine learning. Linear Discriminant Analysis (LDA) is a machine learning algorithm that has the advantage of reducing the parameters used into low dimensions without reducing the accuracy of their classification. The study proposes LDA to classify normal conditions, initial cavitation, intermediate cavitation and severe cavitation. The recording of the vibration signal is taken using the an accelerometer mounted on the inlet of the centrifugal pump. The vibration signal is then extracted using 10 statistic parameters of time domain as the LDA feature selection, namely mean, RMS, standard deviation, kurtosis, skewness, crest factor, clearance factor, shape factor, variance and peak value. The results shows that the LDA classifier can detect and classify cavitation conditions with an accuracy rate of 98.8% on training and 99.6% on testing. The shape factor, kurtosis, skewness and RMS parameters are a combination of parameters that have a large contribution to the classifier to detect and classify cavitation conditions.Keywords: Linear Discriminant Analysis (LDA), cavitation, centrifugal pump, statistical parameter

In this dissertation we develop numerical algorithms and software tools to facilitate parallel solutions of nonlinear programming (NLP) problems. In particular, we address large-scale, block-structured problems with an intrinsic decomposable configuration. These problems arise in a great number of engineering applications, including parameter estimation, optimal control, network optimization, and stochastic programming. The structure of these problems can be leveraged by optimization solvers to accelerate solutions and overcome memory limitations, and we propose variants to two classes of optimization algorithms: augmented Lagrangian (AL) schemes and Schur-complement interior-point methods. The convergence properties of augmented Lagrangian decomposition schemes like the alternating direction method of multipliers (ADMM) and progressive hedging (PH) are well established for convex optimization but convergence guarantees in non-convex settings are still poorly understood. In practice, however, ADMM and PH often perform satisfactorily in complex non-convex NLPs. In this work, we study connections between the method of multipliers (MM), ADMM, and PH to derive benchmarking metrics that explain why PH and ADMM work in practice. We illustrate the concepts using challenging dynamic optimization problems. Our exposition seeks to establish more formalism in benchmarking ADMM, PH, and AL schemes and to motivate algorithmic improvements. The effectiveness of nonlinear interior-point solvers for solving large-scale problems relies quite heavily on the solution of the underlying linear algebra systems. The schur-complement decomposition is very effective for parallelizing the solution of linear systems with modest coupling. However, for systems with large number of coupling variables the schur-complement method does not scale favorably. We implement an approach that uses a Krylov solver (GMRES) preconditioned with ADMM to solve block-structured linear systems that arise in the interior-point method. We show that this ADMM-GMRES approach overcomes the well-known scalability issues of Schur decomposition. One important drawback of using decomposition approaches like ADMM and PH is their convergence rate. Unlike Schur-complement interior-point algorithms that have super-linear convergence, augmented Lagrangian approaches typically exhibit linear and sublinear rates. We exploit connections between ADMM and the Schur-complement decomposition to derive an accelerated version of ADMM. Specifically, we study the effectiveness of performing a Newton-Raphson algorithm to compute multiplier estimates for augmented Lagrangian methods. We demonstrate using two-stage stochastic programming problems that our multiplier update achieves convergence in fewer iterations for MM on general nonlinear problems. In the case of ADMM, the newton update significantly reduces the number of subproblem solves for convex quadratic programs (QPs). Moreover, we show that using newton multiplier updates makes the method robust to the selection of the penalty parameter. Traditionally, state-of-the-art optimization solvers are implemented in low-level programming languages. In our experience, the development of decomposition algorithms in these frameworks is challenging. They present a steep learning curve and can slow the development and testing of new numerical algorithms. To mitigate these challenges, we developed PyNumero, a new open source framework implemented in Python and C++. The package seeks to facilitate development of optimization algorithms for large-scale optimization within a high-level programming environment while at the same time minimizing the computational burden of using Python. The efficiency of PyNumero is illustrated by implementing algorithms for problems arising in stochastic programming and optimal control. Timing results are presented for both serial and parallel implementations. Our computational studies demonstrate that with the appropriate balance between compiled code and Python, efficient implementations of optimization algorithms are achievable in these high-level languages.

Complexity-reduced implementations of complete and null-space-based linear discriminant analysis

Current studies document the effectiveness of multi-sensing technique implementation to mimic or to complement human senses. This work demonstrated the successful application of multi-sensing techniques such electronic tongue (e-tongue), electronic nose (e-nose) and Fourier Transform Infrared Spectroscopy (FTIR). The fusion of these modalities enhance the classification of pure Tualang honey using Linear Discriminant Analysis (LDA), Probabilistic Neural Network (PNN), Support Vector Machine (SVM) and k-Nearest Neighbour (KNN). KNN and PNN are able to classify between pure and adulterated honey samples, outperform LDA and SVM. By performing data fusion, SVM and LDA classifier can achieved more than 80% accuracy, while KNN and PNN obtained greater precision, up to 96% correct classification. The findings confirmed that, multi-sensing technique; either KNN or PNN was significantly superior compared to SVM and LDA classification methods. Thus, both analyses are able to discriminate between pure and adulterated honey.

Dimensionality reduction of image features using the canonical contextual correlation projection

The linear discriminant analysis (LDA) is a very popular linear feature extraction approach. The algorithms of LDA usually perform well under the following two assumptions. The first assumption is that the global data structure is consistent with the local data structure. The second assumption is that the input data classes are Gaussian distributions. However, in real-world applications, these assumptions are not always satisfied. In this paper, we propose an improved LDA framework, the local LDA (LLDA), which can perform well without needing to satisfy the above two assumptions. Our LLDA framework can effectively capture the local structure of samples. According to different types of local data structure, our LLDA framework incorporates several different forms of linear feature extraction approaches, such as the classical LDA and principal component analysis. The proposed framework includes two LLDA algorithms: a vector-based LLDA algorithm and a matrix-based LLDA (MLLDA) algorithm. MLLDA is directly applicable to image recognition, such as face recognition. Our algorithms need to train only a small portion of the whole training set before testing a sample. They are suitable for learning large-scale databases especially when the input data dimensions are very high and can achieve high classification accuracy. Extensive experiments show that the proposed algorithms can obtain good classification results.

In this paper a novel non-linear subspace method for face verification is proposed. The problem of face verification is considered as a two-class problem (genuine versus impostor class). The typical Fisher's linear discriminant analysis (FLDA) gives only one or two projections in a two-class problem. This is a very strict limitation to the search of discriminant dimensions. As for the FLDA for N class problems (N is greater than two) the transformation is not person specific. In order to remedy these limitations of FLDA, exploit the individuality of human faces and take into consideration the fact that the distribution of facial images, under different viewpoints, illumination variations and facial expression is highly complex and non-linear, novel kernel discriminant algorithms are proposed. The new methods are tested in the face verification problem using the XM2VTS database where it is verified that they outperform other commonly used kernel approaches.

The Fisher's linear discriminant analysis (FLDA) is a successful linear feature extraction method which aims to maximize the between-class separability and to minimize the within-class variability. In order to accomplish its goal, FLDA maximizes the Fisher's linear discriminant analysis criterion given in the paper. In this paper we first address the limitations of the classical FLDA criterion and then discuss new criterion functions which were introduced to overcome those limitations. Finally, we demonstrate the capability of the new criteria both theoretically and empirically.