Kernel quadratic discriminant analysis for small sample size problem

This paper aims to understand how the discriminative information of neuronal population activity evolves and accumulates over time. We present two classes of approaches namely the probability-based and response-based approaches to predict the perceptual reports of a trained macaque monkey on a single-trial basis by integrating neural signals from multiple electrodes across time. We extend the probability-based integration originally using only the quadratic discriminant analysis (QDA) by considering also the linear discriminant analysis (LDA) and logistic regression methods. Furthermore, we introduce the response-based integration for the QDA, LDA and logistic regression methods. Experimental examples demonstrate the effectiveness of these approaches for determining the perceptual state of a brain under study by integrating its localized spatiotemporal neuronal activity.

Linear Discriminant Analysis (LDA) has been one of the popular subspace methods for face recognition. But this method suffers from the small sample size (SSS) problem, also known as 'curse of dimensionality'. Various techniques have been proposed in literature to overcome this limitation. But it is still unclear which method provides the best solution to SSS problem. In this paper, we have investigated the performance of some popular subspace methods such as principal component analysis (PCA), PCA + LDA, LDA via QR decomposition, Null-space LDA, Exponential Discriminant Analysis (EDA), PCA+EDA etc. Extensive experiments have been performed on five publically available face datasets viz. AR, CMU-PIE, PIX, Yale and YaleB. The performance is measured in terms of average classification accuracy. Experimental results show that the performance increases with the increase in the number of images per person in training set irrespective of the datasets. There is no clear winner among the subspace methods under investigation. But, the performance of PCA+LDA and PCA+EDA is consistent in tackling SSS problem irrespective of the dataset and can also handle the illumination variation in face recognition.

Many contemporary studies involve the classification of a subject into two classes based on n observations of the p variables associated with the subject. Under the assumption that the variables are normally distributed, the well-known linear discriminant analysis (LDA) assumes a common covariance matrix over the two classes while the quadratic discriminant analysis (QDA) allows different covariance matrices. When p is much smaller than n, even if they both diverge, the LDA and QDA have the smallest asymptotic misclassification rates for the cases of equal and unequal covariance matrices, respectively. However, modern statistical studies often face classification problems with the number of variables much larger than the sample size n, and the classical LDA and QDA can perform poorly. In fact, we give an example in which the QDA performs as poorly as random guessing even if we know the true covariances. Under some sparsity conditions on the unknown means and covariance matrices of the two classes, we propose a sparse QDA based on thresholding that has the smallest asymptotic misclassification rate conditional on the training data. We discuss an example of classifying normal and tumor colon tissues based on a set of p = 2; 000 genes and a sample of size n = 62, and another example of a cardiovascular study for n = 222 subjects with p = 2; 434 genes. A simulation is also conducted to check the performance of the proposed method.

This paper presents a novel and effective method for facial expression recognition including happiness, disgust, fear, anger, sadness, surprise, and neutral state. The proposed method utilizes a regularized discriminant analysis-based boosting algorithm (RDAB) with effective Gabor features to recognize the facial expressions. Entropy criterion is applied to select the effective Gabor feature which is a subset of informative and nonredundant Gabor features. The proposed RDAB algorithm uses RDA as a learner in the boosting algorithm. The RDA combines strengths of linear discriminant analysis (LDA) and quadratic discriminant analysis (QDA). It solves the small sample size and ill-posed problems suffered from QDA and LDA through a regularization technique. Additionally, this study uses the particle swarm optimization (PSO) algorithm to estimate optimal parameters in RDA. Experiment results demonstrate that our approach can accurately and robustly recognize facial expressions.

The objective of this study is to concentrate on the classification method of the logistic regression and the discriminant analysis by using the simulation dataset and the liver patients as the actual data. These datasets are used the binary dependent variable depending on the correlated independent variables or called multicollinearity data. The standard classification method is logistic regression, which uses the logit function’s probability to conduct the dichotomous dependent variable. The iteration process can be solved to estimate logit function parameters and explain the relationship between a dependent binary variable and independent variables. Discriminant analysis is a powerful classification based on linear discriminant analysis (LDA), quadratic discriminant analysis (QDA), and regularized discriminant analysis (RDA). These methods consider the decision boundaries by building a classifier model on the multivariate normal distribution. LDA defines the standard covariance matrix, but QDA has an individual covariance matrix. RDA extends from QDA by setting the regularized parameter to estimate the covariance matrix. In the case of the simulation study, the independent variables are generated by defining the constant correlation on the multivariate normal distribution that made the multicollinearity problem. Then the binary response variable can be approximated from the logit function. For application to actual data, we expressed the classification of type liver and non-liver patients as the dependent variables and obtained patient personal information on the nine independent variables. The highest average percentage of accuracy determines the performance of these methods. The results have shown that the logistic regression was successful when using small independent variables, but the RDA performed when using large independent variables.

Regularized discriminant analysis for the small sample size problem in face recognition

Linear discriminant analysis (LDA) is a popular feature extraction technique in statistical pattern recognition. However, it often suffers from the small sample size problem when dealing with the high dimensional image data. Moreover, while LDA is guaranteed to find the best directions when each class has a Gaussian density with a common covariance matrix, it can fail if the class densities are more general. In this paper, a new feature extraction method, nonparametric maximum margin criterion (NMMC), is proposed. NMMC finds the important discriminant directions without assuming the class densities belong to any particular parametric family, and it does not depend on the nonsingularity of the within-class scatter matrix. Our experimental results on the ATT and FERET face databases demonstrate that NMMC outperforms the existing variant LDA methods and the other state-of-art face recognition approaches.

Sufficient dimension-reduction methods provide effective ways to visualize discriminant analysis problems. For example, Cook and Yin showed that the dimension-reduction method of sliced average variance estimation (SAVE) identifies variates that are equivalent to a quadratic discriminant analysis (QDA) solution. This article makes this connection explicit to motivate the use of SAVE variates in exploratory graphics for discriminant analysis. Classification can then be based on the SAVE variates using a suitable distance measure. If the chosen measure is Mahalanobis distance, then classification is identical to QDA using the original variables. Just as canonical variates provide a useful way to visualize linear discriminant analysis (LDA), so do SAVE variates help visualize QDA. This would appear to be particularly useful given the lack of graphical tools for QDA in current software. Furthermore, whereas LDA and QDA can be sensitive to nonnormality, SAVE is more robust.

Comparison of chemometrics and AOCS official methods for predicting the shelf life of edible oil

The performance of seven classification methods, binary logistic (BLR), probit (PR) and cumulative probit (CPR) regression, linear (LDA) and quadratic (QDA) discriminant analysis, artificial neural networks (ANN), and naïve Bayes classification (NBC), is examined in skeletal sex estimation. These methods were tested using cranial and pelvic sexually dimorphic traits recorded on a modern documented collection, the Athens Collection. For their implementation, an R package has been written to perform cross-validated (CV) sex classification and give the discriminant function of each of the methods studied. A simple algorithm that combines two discriminant functions is also proposed. It was found that the differences in the classification performance between BLR, PR, CPR, LDA, QDA, ANN, and NBC are overall small. However, LDA is simpler and more flexible than CPR, QDA, and ANN and has a small but clear advantage over BLR, NBC, and PR. Consequently, LDA may be preferred in skeletal sex estimation. Finally, it is striking that the combination of pelvic and cranial traits via their discriminant functions, determined either by BLR or LDA, removes practically any population-specificity and yields much better predictions than the individual functions; in fact, the prediction accuracy increases above 97%.

Motor oil classification is important for quality control and the identification of oil adulteration. In this work, we propose a simple, rapid, inexpensive and nondestructive approach based on image analysis and pattern recognition techniques for the classification of nine different types of motor oils according to their corresponding color histograms. For this, we applied color histogram in different color spaces such as red green blue (RGB), grayscale, and hue saturation intensity (HSI) in order to extract features that can help with the classification procedure. These color histograms and their combinations were used as input for model development and then were statistically evaluated by using linear discriminant analysis (LDA), quadratic discriminant analysis (QDA), and support vector machine (SVM) techniques. Here, two common solutions for solving a multiclass classification problem were applied: (1) transformation to binary classification problem using a one-against-all (OAA) approach and (2) extension from binary classifiers to a single globally optimized multilabel classification model. In the OAA strategy, LDA, QDA, and SVM reached up to 97% in terms of accuracy, sensitivity, and specificity for both the training and test sets. In extension from binary case, despite good performances by the SVM classification model, QDA and LDA provided better results up to 92% for RGB-grayscale-HSI color histograms and up to 93% for the HSI color map, respectively. In order to reduce the numbers of independent variables for modeling, a principle component analysis algorithm was used. Our results suggest that the proposed method is promising for the identification and classification of different types of motor oils.

Feature extraction is a very important preprocessing step for classification of hyperspectral images. The linear discriminant analysis (LDA) method fails to work in small sample size situations. Moreover, LDA has poor efficiency for non-Gaussian data. LDA is optimized by a global criterion. Thus, it is not sufficiently flexible to cope with the multi-modal distributed data. We propose a new feature extraction method in this paper, which uses the boundary semi-labeled samples for solving small sample size problem. The proposed method, which called hybrid feature extraction based on boundary semi-labeled samples (HFE-BSL), uses a hybrid criterion that integrates both the local and global criteria for feature extraction. Thus, it is robust and flexible. The experimental results with three real hyperspectral images show the good efficiency of HFE-BSL compared to some popular and state-of-the-art feature extraction methods.

For purposes of dimensionality reduction in classification Linear Discriminant Analysis (LDA) is probably the most common approach. In fact, LDA is a linear dimension reduction technique that also returns a classification rule. In the case of heteroscedasticity of the classes, Quadratic Discriminant Analysis (QDA) can be used to determine an appropriate classification rule, but QDA does not serve for dimensionality reduction. Sliced Average Variance Estimation (SAVE) has been shown to be adequate in such situations as implemented in R in the package dr. This paper presents an alternative approach for linear dimensionality reduction for situations of heteroscedastic intraclass covariances, namely Heteroscedastic Discriminant Analysis (HDA) as well as its R implementation. Furthermore, tests are suggested in order to determine the dimension for the discriminative data subspace and a generalization of HDA by regularization of the covariance matrix estimates is proposed. Examples for application of HDA in R are demonstrated as well as a small simulation study turning out that HDA is preferable to SAVE in a situation where the classes differ in both means and covariances.

We evaluated the directional asymmetry between right and left wings and quantified the intraspecific and interspecific variation of the wing shape of 601 specimens of the genus Diachlorus to determine to what extent the geometrical variation discriminates six species distributed in six protected areas of Colombia. Geometric analyses were performed, integrating Procrustes methods, principal component analyses, cluster analyses, linear and quadratic discriminant analyses, and evaluations of shape changes. In Diachlorus, left and right wings did not present significant asymmetry but a geometrical analysis was allowed for species identification and, in some cases, the origin of the specimens using the variation of wing shape; the best-assigned species was Diachlorus leticia Wilkerson & Fairchild, while the worst was Diachlorus jobbinsi Fairchild, which also had the highest intraspecific variation, while Diachlorus fuscistigma Lutz had the lowest variation. Diachlorus fuscistigma and Diachlorus leucotibialis Wilkerson & Fairchild were the most similar species, while D. leucotibialis and Diachlorus nuneztovari Fairchild & Ortiz were the most disimilar. The specimens with the most different wing shape belonged to Chocó (especially those of D. jobbinsi), the geographically farthest area from the others in the study; however, no correlation was observed between geometric and geographical distances. Linear discriminants were better than nonlinear (quadratic) discriminant analyses in predicting species membership, but the opposite was true for predicting area membership. Based on our data, we hypothesized that other species of Diachlorus could also be discriminated using geometric morphometry of the wing shape.

The abundant movement information carried by the surface electromyography (sEMG) signal can comprehensively reflect the muscle movement patterns on the surface of the human body, and the characteristics of physiological signals involving movement information have promoted the vigorous development in the field of human-computer interaction in machine learning. This paper takes subtle gestures as the research objects, and proposes a subtle gestures recognition system that uses two-channel sEMG signal and utilizes Quadratic Discriminant Analysis (QDA) and Linear Discriminant Analysis (LDA) which are parameter estimation classifiers based on probability density. First, combining the raw sEMG signal and the envelope sEMG signal, the number of channels for sensors and corresponding position of the muscle groups are preferably selected. Then, we propose to use the Pearson correlation coefficient to optimize the four types of features extracted in the time domain, frequency domain, time-frequency domain and AR model parameters, which effectively ensures the uniqueness of each gesture and eliminates individual differences. Finally, this paper compares four classifiers of QDA, LDA, Support Vector Machine (SVM), and K-Nearest Neighbor (KNN), and explores the pattern recognition classifier suitable for subtle actions. The experiment collects 9 kinds of subtle gestures from 8 subjects. The results show that the system can use only two channels of sEMG signal to recognize 9 kinds of subtle gestures which include multi-finger clicking or pinching or holding objects, and the most suitable classifiers QDA and LDA have the average recognition rates of 95.79% and 95.01%, respectively.