Solutions of LDA for Small Sample Size Problems

Linear Discriminant Analysis (LDA) has been one of the popular subspace methods for face recognition. But this method suffers from the small sample size (SSS) problem, also known as 'curse of dimensionality'. Various techniques have been proposed in literature to overcome this limitation. But it is still unclear which method provides the best solution to SSS problem. In this paper, we have investigated the performance of some popular subspace methods such as principal component analysis (PCA), PCA + LDA, LDA via QR decomposition, Null-space LDA, Exponential Discriminant Analysis (EDA), PCA+EDA etc. Extensive experiments have been performed on five publically available face datasets viz. AR, CMU-PIE, PIX, Yale and YaleB. The performance is measured in terms of average classification accuracy. Experimental results show that the performance increases with the increase in the number of images per person in training set irrespective of the datasets. There is no clear winner among the subspace methods under investigation. But, the performance of PCA+LDA and PCA+EDA is consistent in tackling SSS problem irrespective of the dataset and can also handle the illumination variation in face recognition.

A new image feature extraction method for face recognition called Tri-direction 2D-Fisher Discriminant Analysis (T2D-FDA) is proposed to deal with the Small Sample Size (SSS) problem in conventional 1D-Fisher Discriminant Analysis (1D-FDA). Moreover, the essence of T2D-FDA is investigated, and the equivalence of the left-multiplying 2D-FDA of the original image matrices and the left-multiplying D2D-FDA of diagonal image matrices is verified if each column is viewed as a computational unit. Different from the 1D-LDA based approaches, the T2D-FDA is based on 2D image matrices rather than column vectors, so the image matrix does not need to be transformed into a long vector before feature extraction. The advantage arising in this way is that the SSS problem does not exist anymore because within-class scatter matrices constructed is full-rank in T2D-FDA. The proposed method is applied to face recognition where only few training images exist for each subject. Experiment results show T2D-FDA outperforms the current linear subspace methods in SSS problem on four public databases: ORL, Yale, AR and FERET face databases.

We made two contributions in this paper. First, a new method called two-dimensional fisher discriminant analysis (2D-FDA) is proposed to deal with the small sample size (SSS) problem in LDA based face recognition. Second, appearance and depth information are combined to improve face recognition rate. Different from the conventional 1D-FDA (PCA plus LDA) approaches, 2D-FDA is based on 2D image matrices rather than column vectors so the image matrix does not need to be transformed into a long vector before feature extraction. The advantage arising in this way is that the SSS problem does not exist any more because the between-class and within-class scatter matrices constructed in 2D-FDA are both of full-rank. 2D FDA and 1D FDA (PCA plus LDA) are evaluated respectively with a problem that combines appearance and depth information for face recognition. We investigate the complete range of linear combinations to reveal the interplay between these two paradigms. The recognition rate by the combination is better than either appearance alone or depth alone. It is verified that 2D-FDA outperforms 1D FDA

The small sample size (SSS) problem is one of the major problems encountered when traditional kernel discriminant analysis methods are applied to high-dimensional pattern recognition tasks. Different methods have been proposed to solve this problem. In this paper, we introduce a new kernel discriminant analysis algorithm, which is able to effectively address the SSS problem and extract a set of optimal discriminant vectors without any lose of useful discriminant information. Experiments performed on radar target recognition using range profiles indicate that the proposed method outperforms some existing kernel discriminant algorithms, such as generalized discriminant analysis and kernel direct discriminant analysis, in terms of recognition rate.

With recent success in supervised learning, artificial intelligence (AI) and machine learning (ML) can play a vital role in precision medicine. Deep learning neural networks have been used in drug discovery when larger data is available. However, applications of machine learning in clinical trials with small sample size (around a few hundreds) are limited. We propose a Similarity-Principle-Based Machine Learning (SBML) method, which is applicable for small and large sample size problems. In SBML, the attribute-scaling factors are introduced to objectively determine the relative importance of each attribute (predictor). The gradient method is used in learning (training), that is, updating the attribute-scaling factors. We evaluate SBML when the sample size is small and investigate the effects of tuning parameters. Simulations show that SBML achieves better predictions in terms of mean squared errors for various complicated nonlinear situations than full linear models, optimal and ridge regressions, mixed effect models, support vector machine and decision tree methods.

When locality preserving projection (LPP) method was originally proposed, it takes as the LPP solution the minimum eigenvalue solution of an eigenequation. After that, LPP has been used for image recognition problems such as face recognition. However, almost no researcher realizes that LPP usually encounters several difficulties when applied to the image recognition problem. For example, since image recognition problems are usually small sample size (SSS) problems, the corresponding eigenequation cannot be directly solved. In addition, it seems that even if one can obtain the solution of the eigenequation by using the numerical analysis approach, the obtained conventional LPP solution might produce the `presentation confusion' problem for samples from different classes, which is disadvantageous for the classification to procedure a high accuracy. In this paper we first thoroughly investigate the characteristics and drawbacks of the conventional LPP solution in the small sample size (SSS) problem in which the sample number is smaller than the data dimension. In order to overcome these drawbacks, we propose a new LPP solution for the SSS problem, which has clear physical meaning and can be directly and easily worked out because it is generated from a non-singular eigenequation. Experimental results the proposed solution scheme can produce a much lower classification error rate than the conventional LPP solution.

Most finger vein authentication systems suffer from the problem of small sample size. However, the data augmentation can alleviate this problem to a certain extent but did not fundamentally solve the problem of category diversity. So the researchers resort to pre-training or multi-source data joint training methods, but these methods will lead to the problem of user privacy leakage. In view of the above issues, this paper proposes a federated learning-based finger vein authentication framework (FedFV) to solve the problem of small sample size and category diversity while protecting user privacy. Through training under FedFV, each client can share the knowledge learned from its user’s finger vein data with the federated client without causing template leaks. In addition, we further propose an efficient personalized federated aggregation algorithm, named federated weighted proportion reduction (FedWPR), to tackle the problem of non-independent identically distribution caused by client diversity, thus achieving the best performance for each client. To thoroughly evaluate the effectiveness of FedFV, comprehensive experiments are conducted on nine publicly available finger vein datasets. Experimental results show that FedFV can improve the performance of the finger vein authentication system without directly using other client data. To the best of our knowledge, FedFV is the first personalized federated finger vein authentication framework, which has some reference value for subsequent biometric privacy protection research.

Complexity-reduced implementations of complete and null-space-based linear discriminant analysis

The recently proposed Marginal Fisher Analysis (MFA) achieved high performance for face recognition. However, MFA suffers from the small sample size problem and the optimal basis vectors obtained by the MFA are nonorthogonal. In this paper, we present a new method called Direct Orthogonal Marginal Fisher Analysis (DOMFA), which is able to extract all the orthogonal discriminant vectors simultaneously in the high-dimensional feature space without pre-processing using PCA and does not suffer the small sample size problem. Experimental results on ORL database indicate that the proposed DOMFA method achieves better recognition rate than the MFA method and some other orthogonal feature extraction algorithms.

Novel Fisher discriminant classifiers

In this paper, we propose a method by engaging the One Class Support Vector Machine (OC-SVM) in the identification of Diffractive Optically Variable Images (DOVIs). OC-SVM, as a special SVM, can solve the problems of high-dimensional data sets and small sample size (SSS) with positive and negative unbalance training data. Image feature matrix is built by extracting image features from texture aspects. OC-SVM can be trained with the high-dimensional matrix directly, and does not have to reduce the dimensionality of feature matrix as the usual methods. The experiment results show the effectiveness of the proposed approach against Linear Discriminant Analysis. Considering time cost and correct classification rate, OC-SVM is suitable for the identification of DOVIs.

This study is dedicated to modeling the spatial variation in COVID-19 prevalence using the adaptive neuro-fuzzy inference system (ANFIS) when dealing with nonlinear relationships, especially useful for small areas or small sample size problems. We compiled a broad range of socio-demographic, environmental, and climatic factors along with potentially related urban land uses to predict COVID-19 prevalence in rural districts of the Golestan province northeast of Iran with a very high-case fatality ratio (9.06%) during the first year of the pandemic (2020–2021). We also compared the ANFIS and principal component analysis (PCA)-ANFIS methods for modeling COVID-19 prevalence in a geographical information system framework. Our results showed that combined with the PCA, the ANFIS accuracy significantly increased. The PCA-ANFIS model showed a superior performance (R2 (determination coefficient) = 0.615, MAE (mean absolute error) = 0.104, MSE (mean square error) = 0.020, and RMSE (root mean square error) = 0.139) than the ANFIS model (R2 = 0.543, MAE = 0.137, MSE = 0.034, and RMSE = 0.185). The sensitivity analysis of the ANFIS model indicated that migration rate, employment rate, the number of days with rainfall, and residential apartment units were the most contributing factors in predicting COVID-19 prevalence in the Golestan province. Our findings indicated the ability of the ANFIS model in dealing with nonlinear parameters, particularly for small sample sizes. Identifying the main factors in the spread of COVID-19 may provide useful insights for health policymakers to effectively mitigate the high prevalence of the disease.

To solve the Small Sample Size (SSS) problem, the recent linear discriminant analysis using the 2D matrix-based data representation model has demonstrated its superiority over that using the conventional vector-based data representation model in face recognition [7]. But the explicit reason why the matrix-based model is better than vectorized model has not been given until now. In this paper, a framework of Generalized 2D Fisher Discriminant Analysis (G2DFDA) is proposed. Three contributions are included in this framework: 1) the essence of these ’2D’ methods is analyzed and their relationships with conventional ’1D’ methods are given, 2) a Bilateral and 3) a Kernel-based 2D Fisher Discriminant Analysis methods are proposed. Extensive experiment results show its excellent performance.

Because the Small Sample Size (SSS) problem usually leads to singularity of the within-class scatter matrix,there will be an ill-posed problem in solving the generalized character equation. An improved algorithm based on former ones was proposed. It introduced mirror image to enlarge sample capacity and adopted the method of the null space of the within-class scatter matrix S w to get the best solution of Fisher criterion. Experimental results on ORL face database and Yale face database show that the proposed algorithm is more effective than traditional Linear Discriminant Analysis (LDA),Independent Components Analysis (ICA) and 2-Dimensional Symmetric Principal Component Analysis (2DSPCA).

LPP solution schemes for use with face recognition