Regularized discriminant analysis for high dimensional, low sample size data

The objective of this study is to concentrate on the classification method of the logistic regression and the discriminant analysis by using the simulation dataset and the liver patients as the actual data. These datasets are used the binary dependent variable depending on the correlated independent variables or called multicollinearity data. The standard classification method is logistic regression, which uses the logit function’s probability to conduct the dichotomous dependent variable. The iteration process can be solved to estimate logit function parameters and explain the relationship between a dependent binary variable and independent variables. Discriminant analysis is a powerful classification based on linear discriminant analysis (LDA), quadratic discriminant analysis (QDA), and regularized discriminant analysis (RDA). These methods consider the decision boundaries by building a classifier model on the multivariate normal distribution. LDA defines the standard covariance matrix, but QDA has an individual covariance matrix. RDA extends from QDA by setting the regularized parameter to estimate the covariance matrix. In the case of the simulation study, the independent variables are generated by defining the constant correlation on the multivariate normal distribution that made the multicollinearity problem. Then the binary response variable can be approximated from the logit function. For application to actual data, we expressed the classification of type liver and non-liver patients as the dependent variables and obtained patient personal information on the nine independent variables. The highest average percentage of accuracy determines the performance of these methods. The results have shown that the logistic regression was successful when using small independent variables, but the RDA performed when using large independent variables.

Lung cancer is a type of cancer that starts from the lungs with abnormal cell growth. Lung cancer is a type of cancer with the largest number of sufferers. In this research, Regularized Discriminant Analysis (RDA) analysis is used to classify the types of lung cancer that exist. Regularized Discriminant Analysis (RDA) is a generalization of Linear Discriminant Analysis (LDA) and Quadratic Discriminant Analysis (QDA). If the alpha parameter is set to 1, this operator performs an LDA. Likewise if the alpha parameter is set to 0, this operator performs QDA. Discriminant analysis is used to determine which variables distinguish between two or more groups that occur naturally. With an accuracy value of 60%, it can be ascertained the type of lung cancer suffered by patients for ease of care and treatment.

Sex estimation is an important part of skeletal analysis and forensic identification. Traditionally pelvic traits are utilized for accurate sex estimation. However, the long bones, especially humerus, have been proved to be as effective for determine the sex of the individual.The aim of this study was to compare the predictive accuracy of seven statistical modelling techniques including classical statistical methods and machine learning algorithms, to assess the sexual dimorphism of humerus on a French sample based on a metric analysis of 26 measurements. A total of 98 humeral bones (divided in two samples) were measured. Seven statistical models were compared: Linear Discriminant Analysis (LDA), Regularized Discriminant Analysis (RDA), Penalized Logistic Regression (PLR), Flexible Discriminant Analysis (FDA), Support Vector Machine (SVM), and Artificial Neural Network (ANN) and Random Forest (RF).With cross validation, classification accuracy was greater than 90% (ranges between 92% and 98%) for all models without variable selection methods. The simplification of the models has improved the accuracy between 98% and 100% and also a reduction of the number of variables to 6 or less. Penalized logistic regression (PLR), Random Forest (RF) and Linear discriminant analysis (LDA) were the best accuracy models.The measurements made at the proximal part of the humerus (WTT, CSD), at distal part (BEW, WT, MAW, THT) and of the entire bone (PLCT) stand out among the various models.The present study suggests that the humerus is an interesting alternative for sex estimation and that non-classical statistical models can provide a new approach.

In this thesis, we focus on studying the performance of general regularized discriminant analysis (RDA) classifiers. The data used for analysis is assumed to follow Gaussian mixture model with different means and covariances. RDA offers a rich class of regularization options, covering as special cases the regularized linear discriminant analysis (RLDA) and the regularized quadratic discriminant analysis (RQDA) classi ers. We analyze RDA under the double asymptotic regime where the data dimension and the training size both increase in a proportional way. This double asymptotic regime allows for application of fundamental results from random matrix theory. Under the double asymptotic regime and some mild assumptions, we show that the asymptotic classification error converges to a deterministic quantity that only depends on the data statistical parameters and dimensions. This result not only implicates some mathematical relations between the misclassification error and the class statistics, but also can be leveraged to select the optimal parameters that minimize the classification error, thus yielding the optimal classifier. Validation results on the synthetic data show a good accuracy of our theoretical findings. We also construct a general consistent estimator to approximate the true classification error in consideration of the unknown previous statistics. We benchmark the performance of our proposed consistent estimator against classical estimator on synthetic data. The observations demonstrate that the general estimator outperforms others in terms of mean squared error (MSE).

We identify canopy species in a Hawaiian tropical forest using supervised classification applied to airborne hyperspectral imagery acquired with the Carnegie Airborne Observatory-Alpha system. Nonparametric methods (linear and radial basis function support vector machine, artificial neural network, and k-nearest neighbor) and parametric methods (linear, quadratic, and regularized discriminant analysis) are compared for a range of species richness values and training sample sizes. We find a clear advantage in using regularized discriminant analysis, linear discriminant analysis, and support vector machines. No unique optimal classifier was found for all conditions tested, but we highlight the possibility of improving support vector machine classification with a better optimization of its free parameters. We also confirm that a combination of spectral and spatial information increases accuracy of species classification: we combine segmentation and species classification from regularized discriminant analysis to produce a map of the 17 discriminated species. Finally, we compare different methods to assess spectral separability and find a better ability of Bhattacharyya distance to assess separability within and among species. The results indicate that species mapping is tractable in tropical forests when using high-fidelity imaging spectroscopy.

Sufficient dimension-reduction methods provide effective ways to visualize discriminant analysis problems. For example, Cook and Yin showed that the dimension-reduction method of sliced average variance estimation (SAVE) identifies variates that are equivalent to a quadratic discriminant analysis (QDA) solution. This article makes this connection explicit to motivate the use of SAVE variates in exploratory graphics for discriminant analysis. Classification can then be based on the SAVE variates using a suitable distance measure. If the chosen measure is Mahalanobis distance, then classification is identical to QDA using the original variables. Just as canonical variates provide a useful way to visualize linear discriminant analysis (LDA), so do SAVE variates help visualize QDA. This would appear to be particularly useful given the lack of graphical tools for QDA in current software. Furthermore, whereas LDA and QDA can be sensitive to nonnormality, SAVE is more robust.

Comparison of the performance of multiclass classifiers in chemical data: Addressing the problem of overfitting with the permutation test

Multigroup discriminant analysis is an important supervised learning technique in the classification framework, with applications in various disciplines. Its objective is to approximate underlying class distributions based on data attributes or features. After the class distributions are estimated, the classification task can be readily carried out for data points with unknown labels. Linear discriminant analysis (LDA) as well as quadratic discriminant analysis (QDA) are statistical procedures widely utilized by practitioners due to their practicality and generally good performance. Both procedures rely on the assumption of normally distributed classes and can be affected by deviations from multivariate normality. To address this model deficiency, we propose an extension of LDA and QDA that relies on the idea of transformation and can readily accommodate asymmetry and skewness in data classes. Through the set of simulation studies and applications to real-life data sets, we demonstrate that the developed technique is promising as it demonstrates superior performance over competitors in a variety of cases.

The abundant movement information carried by the surface electromyography (sEMG) signal can comprehensively reflect the muscle movement patterns on the surface of the human body, and the characteristics of physiological signals involving movement information have promoted the vigorous development in the field of human-computer interaction in machine learning. This paper takes subtle gestures as the research objects, and proposes a subtle gestures recognition system that uses two-channel sEMG signal and utilizes Quadratic Discriminant Analysis (QDA) and Linear Discriminant Analysis (LDA) which are parameter estimation classifiers based on probability density. First, combining the raw sEMG signal and the envelope sEMG signal, the number of channels for sensors and corresponding position of the muscle groups are preferably selected. Then, we propose to use the Pearson correlation coefficient to optimize the four types of features extracted in the time domain, frequency domain, time-frequency domain and AR model parameters, which effectively ensures the uniqueness of each gesture and eliminates individual differences. Finally, this paper compares four classifiers of QDA, LDA, Support Vector Machine (SVM), and K-Nearest Neighbor (KNN), and explores the pattern recognition classifier suitable for subtle actions. The experiment collects 9 kinds of subtle gestures from 8 subjects. The results show that the system can use only two channels of sEMG signal to recognize 9 kinds of subtle gestures which include multi-finger clicking or pinching or holding objects, and the most suitable classifiers QDA and LDA have the average recognition rates of 95.79% and 95.01%, respectively.

PurposeMany studies have proposed predictive models for type 2 diabetes mellitus (T2DM). However, these predictive models have several limitations, such as user convenience and reproducibility. The purpose of this study was to develop a T2DM predictive model using electronic medical records (EMRs) and machine learning and to compare the performance of this model with traditional statistical methods.Materials and MethodsIn this study, a total of available 8454 patients who had no history of diabetes and were treated at the cardiovascular center of Korea University Guro Hospital were enrolled. All subjects completed 5 years of follow up. The prevalence of T2DM during follow up was 4.78% (404/8454). A total of 28 variables were extracted from the EMRs. In order to verify the cross-validation test according to the prediction model, logistic regression (LR), linear discriminant analysis (LDA), quadratic discriminant analysis (QDA), and K-nearest neighbor (KNN) algorithm models were generated. The LR model was considered as the existing statistical analysis method.ResultsAll predictive models maintained a change within the standard deviation of area under the curve (AUC) <0.01 in the analysis after a 10-fold cross-validation test. Among all predictive models, the LR learning model showed the highest prediction performance, with an AUC of 0.78. However, compared to the LR model, the LDA, QDA, and KNN models did not show a statistically significant difference.ConclusionWe successfully developed and verified a T2DM prediction system using machine learning and an EMR database, and it predicted the 5-year occurrence of T2DM similarly to with a traditional prediction model. In further study, it is necessary to apply and verify the prediction model through clinical research.

The performance of seven classification methods, binary logistic (BLR), probit (PR) and cumulative probit (CPR) regression, linear (LDA) and quadratic (QDA) discriminant analysis, artificial neural networks (ANN), and naïve Bayes classification (NBC), is examined in skeletal sex estimation. These methods were tested using cranial and pelvic sexually dimorphic traits recorded on a modern documented collection, the Athens Collection. For their implementation, an R package has been written to perform cross-validated (CV) sex classification and give the discriminant function of each of the methods studied. A simple algorithm that combines two discriminant functions is also proposed. It was found that the differences in the classification performance between BLR, PR, CPR, LDA, QDA, ANN, and NBC are overall small. However, LDA is simpler and more flexible than CPR, QDA, and ANN and has a small but clear advantage over BLR, NBC, and PR. Consequently, LDA may be preferred in skeletal sex estimation. Finally, it is striking that the combination of pelvic and cranial traits via their discriminant functions, determined either by BLR or LDA, removes practically any population-specificity and yields much better predictions than the individual functions; in fact, the prediction accuracy increases above 97%.

Many contemporary studies involve the classification of a subject into two classes based on n observations of the p variables associated with the subject. Under the assumption that the variables are normally distributed, the well-known linear discriminant analysis (LDA) assumes a common covariance matrix over the two classes while the quadratic discriminant analysis (QDA) allows different covariance matrices. When p is much smaller than n, even if they both diverge, the LDA and QDA have the smallest asymptotic misclassification rates for the cases of equal and unequal covariance matrices, respectively. However, modern statistical studies often face classification problems with the number of variables much larger than the sample size n, and the classical LDA and QDA can perform poorly. In fact, we give an example in which the QDA performs as poorly as random guessing even if we know the true covariances. Under some sparsity conditions on the unknown means and covariance matrices of the two classes, we propose a sparse QDA based on thresholding that has the smallest asymptotic misclassification rate conditional on the training data. We discuss an example of classifying normal and tumor colon tissues based on a set of p = 2; 000 genes and a sample of size n = 62, and another example of a cardiovascular study for n = 222 subjects with p = 2; 434 genes. A simulation is also conducted to check the performance of the proposed method.

People can identify emotion from facial expressions easily, but it is more difficult to do it with a computer. It is now feasible to identify feelings from images because of recent advances in computational intelligence. Emotional responses are those mental states of thoughts that develop without conscious effort and are naturally associated with facial muscles, resulting in different facial expressions such as happy, sad, angry, contempt, fear, surprise etc. Emotions play an important role in nonverbal cues that represent a person’s interior thoughts. Intimate robots are expanding in every domain, whether it is completing requirements of elderly people, addressing psychiatric patients, child rehabilitation, or even childcare, as the human–robot interface is grabbing on every day with the increased demand for automation in every industry. We evaluate and test machine learning algorithms on the FER 2013 data set to recognize human emotion from facial expressions, with some of them achieving the highest accuracy and the others failing to detect emotions. Many researchers have used various machine learning methods to identify human emotions during the last few years. In this research article, we analyze eight frequently used machine learning techniques on the FER 2013 dataset to determine which method performs best at categorizing human facial expression. After analyzing the results, it is found that the accuracy of some of the algorithms is quite satisfactory, with 37% for Logistic Regression, 33% for K-neighbors classifier, 100% for Decision Tree Classifier, 78% for Random Forests, 57% for Ada-Boost, 100% for Gaussian NB, 33% for LDA (Linear Discriminant Analysis), and 99% for QDA (Quadratic Discriminant Analysis). Furthermore, the experimental results show that the Decision Tree and Gaussian NB Classifier can correctly identify all of the emotions in the FER 2013 dataset with 100% classification accuracy, while Quadratic Discriminant Analysis can do so with 99% accuracy.KeywordsFacial expressionEmotionMachine learningKNNQDADecision treeGaussian NBRandom Forest

Tropical forests have exceptional floristic diversity, but their characterization remains incomplete, in part due to the resource intensity of in-situ assessments. Remote sensing technologies can provide valuable, cost-effective, large-scale insights. This study investigates the combined use of airborne LiDAR and imaging spectroscopy to map tree species at landscape scale in French Guiana. Binary classifiers were developed for each of 20 species using linear discriminant analysis (LDA), regularized discriminant analysis (RDA) and logistic regression (LR). Complementing visible and near infrared (VNIR) spectral bands with short wave infrared (SWIR) bands improved the mean average classification accuracy of the target species from 56.1% to 79.6%. Increasing the number of non-focal species decreased the success rate of target species identification. Classification performance was not significantly affected by impurity rates (confusion between assigned classes) in the non-focal class (up to 5% of bias), provided that an adequate criterion was used for adjusting threshold probability assignment. A limited number of crowns (30 crowns) in each species class was sufficient to retrieve correct labels effectively. Overall canopy area of target species was strongly correlated to their basal area over 118 ha at 1.5 ha resolution, indicating that operational application of the method is a realistic prospect (R2 = 0.75 for six major commercial tree species).

Sex and species estimation from skeletal remains is important in veterinary forensic medicine, comparative anatomy, and zooarchaeology. Radiographic osteometry has been studied in dogs and cats, but machine-learning approaches have not been well evaluated for this purpose. To assess the performance of machine-learning models and a multilayer perceptron for estimating sex and species from radiographic femoral measurements in dogs and cats. This retrospective study analyzed pelvic radiographs of 280 animals (140 dogs and 140 cats; 70 males and 70 females of each species) using 9 radiographic measurements. Random forest, decision tree, logistic regression, extra trees, linear discriminant analysis, quadratic discriminant analysis, and a multilayer perceptron were evaluated. Feature importance was explored with Shapley additive explanations. For sex classification, the extra trees classifier showed the highest accuracy in both dogs (0.79) and cats (0.75). For species classification, logistic regression, quadratic discriminant analysis, and decision tree each achieved an accuracy of 0.89, whereas the multilayer perceptron reached 0.93 after 500 and 1,000 training cycles. The most influential variables were femoral length for sex classification in cats, left intercondylar fossa width for sex classification in dogs, and inter-femoral-head distance for species classification. Radiographic femoral measurements permit moderate sex classification and high species classification in dogs and cats. These findings support the potential use of machine-learning analysis of femoral radiographs in veterinary forensic medicine and related morphometric fields.