Regularized Discriminant Analysis: A Large Dimensional Study

Linear and Quadratic Discriminant Analysis have been used widely in many areas of data mining, machine learning, and bioinformatics. Friedman proposed a compromise between Linear and Quadratic Discriminant Analysis, called Regularized Discriminant Analysis (RDA), which has been shown to be more flexible in dealing with various class distributions. RDA applies the regularization techniques by employing two regularization parameters, which are chosen to jointly maximize the classification performance. The optimal pair of parameters is commonly estimated via cross-validation from a set of candidate pairs. It is computationally prohibitive for high dimensional data, especially when the candidate set is large, which limits the applications of RDA to low dimensional data.In this paper, a novel algorithm for RDA is presented for high dimensional data. It can estimate the optimal regularization parameters from a large set of parameter candidates efficiently. Experiments on a variety of datasets confirm the claimed theoretical estimate of the efficiency, and also show that, for a properly chosen pair of regularization parameters, RDA performs favorably in classification, in comparison with other existing classification methods.

The objective of this study is to concentrate on the classification method of the logistic regression and the discriminant analysis by using the simulation dataset and the liver patients as the actual data. These datasets are used the binary dependent variable depending on the correlated independent variables or called multicollinearity data. The standard classification method is logistic regression, which uses the logit function’s probability to conduct the dichotomous dependent variable. The iteration process can be solved to estimate logit function parameters and explain the relationship between a dependent binary variable and independent variables. Discriminant analysis is a powerful classification based on linear discriminant analysis (LDA), quadratic discriminant analysis (QDA), and regularized discriminant analysis (RDA). These methods consider the decision boundaries by building a classifier model on the multivariate normal distribution. LDA defines the standard covariance matrix, but QDA has an individual covariance matrix. RDA extends from QDA by setting the regularized parameter to estimate the covariance matrix. In the case of the simulation study, the independent variables are generated by defining the constant correlation on the multivariate normal distribution that made the multicollinearity problem. Then the binary response variable can be approximated from the logit function. For application to actual data, we expressed the classification of type liver and non-liver patients as the dependent variables and obtained patient personal information on the nine independent variables. The highest average percentage of accuracy determines the performance of these methods. The results have shown that the logistic regression was successful when using small independent variables, but the RDA performed when using large independent variables.

Lung cancer is a type of cancer that starts from the lungs with abnormal cell growth. Lung cancer is a type of cancer with the largest number of sufferers. In this research, Regularized Discriminant Analysis (RDA) analysis is used to classify the types of lung cancer that exist. Regularized Discriminant Analysis (RDA) is a generalization of Linear Discriminant Analysis (LDA) and Quadratic Discriminant Analysis (QDA). If the alpha parameter is set to 1, this operator performs an LDA. Likewise if the alpha parameter is set to 0, this operator performs QDA. Discriminant analysis is used to determine which variables distinguish between two or more groups that occur naturally. With an accuracy value of 60%, it can be ascertained the type of lung cancer suffered by patients for ease of care and treatment.

We identify canopy species in a Hawaiian tropical forest using supervised classification applied to airborne hyperspectral imagery acquired with the Carnegie Airborne Observatory-Alpha system. Nonparametric methods (linear and radial basis function support vector machine, artificial neural network, and k-nearest neighbor) and parametric methods (linear, quadratic, and regularized discriminant analysis) are compared for a range of species richness values and training sample sizes. We find a clear advantage in using regularized discriminant analysis, linear discriminant analysis, and support vector machines. No unique optimal classifier was found for all conditions tested, but we highlight the possibility of improving support vector machine classification with a better optimization of its free parameters. We also confirm that a combination of spectral and spatial information increases accuracy of species classification: we combine segmentation and species classification from regularized discriminant analysis to produce a map of the 17 discriminated species. Finally, we compare different methods to assess spectral separability and find a better ability of Bhattacharyya distance to assess separability within and among species. The results indicate that species mapping is tractable in tropical forests when using high-fidelity imaging spectroscopy.

Comparison of the performance of multiclass classifiers in chemical data: Addressing the problem of overfitting with the permutation test

Fisher linear discriminant analysis (LDA) involves within-class and between-class covariance matrices. For 2-D data such as images, regularized LDA (RLDA) can improve LDA due to the regularized eigenvalues of the estimated within-class matrix. However, it fails to consider the eigenvectors and the estimated between-class matrix. To improve these two matrices simultaneously, we propose in this paper a new two-stage method for 2-D data, namely a bidirectional LDA (BLDA) in the first stage and the RLDA in the second stage, where both BLDA and RLDA are based on the Fisher criterion that tackles correlation. BLDA performs the LDA under special separable covariance constraints that incorporate the row and column correlations inherent in 2-D data. The main novelty is that we propose a simple but effective statistical test to determine the subspace dimensionality in the first stage. As a result, the first stage reduces the dimensionality substantially while keeping the significant discriminant information in the data. This enables the second stage to perform RLDA in a much lower dimensional subspace, and thus improves the two estimated matrices simultaneously. Experiments on a number of 2-D synthetic and real-world data sets show that BLDA+RLDA outperforms several closely related competitors.

Crop yield prediction is vital for global foodstuffs and intensely challenging due to its multitudinous factors such as soil, weather, rainfall, temperature, and their interactions. The influence of weather changes in India impacted in terms of the agricultural crop outcomes over the last decades. The proposed research focuses on the weather condition and assists the farmers to comprehend the crop yields before growing onto farming. This research illustrates a deep learning framework using artificial neural networks (ANN). The offered ANN model, along with further widespread approaches such as bagging classifier, extra tees classifier, linear discriminant analysis (LDA) classifier, quadratic discriminant analysis (QDA) classifier and stochastic gradient boosting (SGB) classifier are used for India's downloaded crop dataset. These proposed six models achieved quite good accuracies. The numerous substantial contributions of the prediction models are the ability to produce the exact predictions. These were succeeded by training the algorithms and naturally to the verify data. Eventually, we tend to compare our study with the previous study's research paper.

ABSTRACT- Firmness is one of the most important quality indicators for apple fruits, which is highly correlated with the storage time. The acoustic impulse response technique is one of the most commonly used nondestructive detection methods for evaluating apple firmness. This paper presents a non-destructive method for classification of Iranian apple (Malus domestica Borkh. cv. Golab) according to the duration of storage. Several data preprocessing methods were tested: normalization, detrending, Savitzky-Golay smoothing, first and second derivatives, multiplicative scatter correction, standard normal variate and moving average. It was observed that the maximum average Fβ value of classification on the test dataset (0.84) belongs to non-preprocessing. In this study, principal component analysis (PCA) technique was performed to determine the key variables that explain most differences in the spectra. Seven principal components were used to calibrate linear discriminant analysis (LDA) and quadratic discriminant analysis (QDA) classifiers. The classification accuracy for LDA and QDA models were about 80.56% and 83.33%, respectively. The results indicated that the acoustic impulse response method is potentially applicable for the detection of apple firmnes

Objective. Most brain–computer interfaces (BCIs) based on functional near-infrared spectroscopy (fNIRS) require that users perform mental tasks such as motor imagery, mental arithmetic, or music imagery to convey a message or to answer simple yes or no questions. These cognitive tasks usually have no direct association with the communicative intent, which makes them difficult for users to perform. Approach. In this paper, a 3-class intuitive BCI is presented which enables users to directly answer yes or no questions by covertly rehearsing the word ‘yes’ or ‘no’ for 15 s. The BCI also admits an equivalent duration of unconstrained rest which constitutes the third discernable task. Twelve participants each completed one offline block and six online blocks over the course of two sessions. The mean value of the change in oxygenated hemoglobin concentration during a trial was calculated for each channel and used to train a regularized linear discriminant analysis (RLDA) classifier. Main results. By the final online block, nine out of 12 participants were performing above chance (p < 0.001 using the binomial cumulative distribution), with a 3-class accuracy of 83.8% ± 9.4%. Even when considering all participants, the average online 3-class accuracy over the last three blocks was 64.1 % ± 20.6%, with only three participants scoring below chance (p < 0.001). For most participants, channels in the left temporal and temporoparietal cortex provided the most discriminative information. Significance. To our knowledge, this is the first report of an online 3-class imagined speech BCI. Our findings suggest that imagined speech can be used as a reliable activation task for selected users for development of more intuitive BCIs for communication.

Sex estimation is an important part of skeletal analysis and forensic identification. Traditionally pelvic traits are utilized for accurate sex estimation. However, the long bones, especially humerus, have been proved to be as effective for determine the sex of the individual.The aim of this study was to compare the predictive accuracy of seven statistical modelling techniques including classical statistical methods and machine learning algorithms, to assess the sexual dimorphism of humerus on a French sample based on a metric analysis of 26 measurements. A total of 98 humeral bones (divided in two samples) were measured. Seven statistical models were compared: Linear Discriminant Analysis (LDA), Regularized Discriminant Analysis (RDA), Penalized Logistic Regression (PLR), Flexible Discriminant Analysis (FDA), Support Vector Machine (SVM), and Artificial Neural Network (ANN) and Random Forest (RF).With cross validation, classification accuracy was greater than 90% (ranges between 92% and 98%) for all models without variable selection methods. The simplification of the models has improved the accuracy between 98% and 100% and also a reduction of the number of variables to 6 or less. Penalized logistic regression (PLR), Random Forest (RF) and Linear discriminant analysis (LDA) were the best accuracy models.The measurements made at the proximal part of the humerus (WTT, CSD), at distal part (BEW, WT, MAW, THT) and of the entire bone (PLCT) stand out among the various models.The present study suggests that the humerus is an interesting alternative for sex estimation and that non-classical statistical models can provide a new approach.

Although mostly used for pattern classification, linear discriminant analysis (LDA) may also be used for feature selection. When employed to select genes for microarray data, which has high dimensionality and small sample size, LDA encounters three problems, including singularity of scatter matrix, overfitting and prohibitive computational complexity. In this study, we propose a new regularization technique to address the singularity and overfitting problem. In addition, we develop a recursive implementation for LDA to reduce computational overhead. Experimental studies on 5 gene microarray problems show that the regularized linear discriminant analysis (RLDA) and its recursive implementation produce gene subsets with excellent classification performance.

Selection of suitable features plays a pivotal role in Electromyography pattern recognition (EMG-PR) based system designing. Time-domain features are widely used in EMG-PR based application and show improved proficiency in the development of rehabilitation robotics. Even though, the performance of existing features is not satisfactory. In this study, we proposed four novel time-domain features obtained by using first-order differentiation of original surface electromyogram (sEMG) signals feature. Here, sEMG signals were acquired from ten healthy volunteers with the help of myotrace400 device for six different arm movements. The data acquisition and pre-processing stage were carried out followed by the feature extraction process for better classification results. Four different classifiers namely, k-nearest neighbors (KNN), Linear discriminant analysis (LDA), Quadratic discriminant analysis (QDA) and Medium tree (MT) classifiers were utilized for the performance evaluation of proposed and conventional features. Experimental results demonstrate that proposed features extracted by using first-order differentiation of sEMG signals feature attained better classification accuracy with MT classifier as compared to the feature extracted from original sEMG signals with the conventional features. The accuracy of proposed feature based on first-order differentiation improved up to 6%. The results indicate that proposed features may be considered for developing the EMG-PR based system designing.

This paper presents a study of murmur recognition in heart sound signal with different machine learning algorithms using combined features. The heart sound signal is required to be pre-processed and segmented first before performing a feature generation and classification process. In this work, two hundred segmented heart cycle samples were parameterized based on temporal, spectral and geometric features as input of cubic Support Vector Machine (SVM), quadratic Discriminant Analysis (DA) and fine K-Nearest Neighbour (KNN) classifiers. The significant of this study is to support the hypothesis in statistic that said higher number of samples provide more accuracy of classifier and smaller margin error. To determine which classifier algorithm is outperformed, cross-validation technique is conducted and varied with different number of samples. The findings indicate cubic Support Vector Machine (SVM) classifier model yields the best performance, achieving an accuracy of 0.97 and F1-score of 0.98 at higher number of samples.

Boosting is a general method for improving the accuracy of any given learning algorithm. In this paper, we have proposed the boosting method for face recognition (FR) that improves the linear discriminant analysis (LDA)-based technique. The improvement is achieved by incorporating the regularized LDA (R-LDA) technique into the boosting framework. R-LDA is based on a new regularized Fisher’s discriminant criterion, which is particularly robust against the small sample size problem compared to the traditional one used in LDA. The AdaBoost technique is utilized within this framework to generalize a set of simple FR subproblems and their corresponding LDA solutions and combines the results from the multiple, relatively weak, LDA solutions to form a strong solution. The comparative experimental result on FERET database demonstrates that the proposed boosting method achieves more accurate results over the individual algorithms.

A new face recognition method is proposed in this paper. The proposed method is based on fuzzy regularized linear discriminant analysis (FR-LDA) and combines the regularized linear discriminant analysis (R-LDA) and the fuzzy set theory. R-LDA is based on a new regularized Fisher's discriminant criterion, which is particularly robust against the small sample size problem compared to the traditional one used in LDA. In the proposed method, we calculate the membership degree matrix by Fuzzy K-nearest neighbor (FKNN) and then incorporate the membership degree into the definition of the between-class and within-class scatter matrices and get the fuzzy between-class and within-class scatter matrices. Experimental results obtained on the FERET database demonstrate that the proposed method improves the classification rate performance.