- Research Article
17
- 10.1016/j.neucom.2015.12.068
DC programming and DCA for sparse optimal scoring problem
- Jan 06, 2016
- Neurocomputing
- Hoai An Le Thi + 1 more +1
DC programming and DCA for sparse optimal scoring problem
Linear discriminant analysis (LDA) has attracted many attentions as a classical tool for both classification and dimensionality reduction. Classical LDA performs quite well in simple and low dimensional setting while it is not suitable for small sample size data (SSS). Feature selection is an effective way to solve this problem. As a variant of LDA, sparse optimal scoring (SOS) with $\ell _{0}$ -norm regularization is considered in this paper. By using a new continuous nonconvex nonsmooth function to approximate $\ell _{0}$ -norm, we propose a novel difference of convex functions algorithm (DCA) for sparse optimal scoring. The most favorable property of the proposed DCA is its subproblem admits an analytical solution. The effectiveness of the proposed method is validated via theoretical analysis as well as some illustrative numerical experiments.
Loading PDF
DC programming and DCA for sparse optimal scoring problem
DC programming and DCA for sparse optimal scoring problem
A New Linear Discriminant Analysis Method to Address the Over-Reducing Problem
Linear discriminant analysis (LDA) is an effective and efficient linear dimensionality reduction and feature extraction method. It has been used in a broad range of pattern recognition tasks including face recognition, document recognition and image retrieval. When applied to fewer-class classification tasks (such as binary classification), however, LDA suffers from the over-reducing problem – insufficient number of features are extracted for describing the class boundaries. This is due to the fact that LDA results in a fixed number of reduced features, which is one less the number of classes. As a result, the classification performance will suffer, especially when the classification data space has high dimensionality. To cope with the problem we propose a new LDA variant, orLDA (i.e., LDA for over-reducing problem), which promotes the use of individual data instances instead of summary data alone in generating the transformation matrix. As a result orLDA will obtain a number of features that is independent of the number of classes. Extensive experiments show that orLDA has better performance than the original LDA and two LDA variants – uncorrelated LDA and orthogonal LDA.
Read moreA hybrid PCA-LDA model for dimension reduction
Several variants of Linear Discriminant Analysis (LDA) have been investigated to address the vanishing of the within-class scatter under projection to a low-dimensional subspace in LDA. However, some of these proposals are ad hoc and some others do not address the problem of generalization to new data. Meanwhile, even though LDA is preferred in many application of dimension reduction, it does not always outperform Principal Component Analysis (PCA). In order to optimize discrimination performance in a more generative way, a hybrid dimension reduction model combining PCA and LDA is proposed in this paper. We also present a dimension reduction algorithm correspondingly and illustrate the method with several experiments. Our results have shown that the hybrid model outperform PCA, LDA and the combination of them in two separate stages.
Read moreDeterministic and Stochastic DCA for DC Programming
In the context of big data analysis, stochastic optimization algorithms are widely used as effective tools to handle data complexity and data uncertainty. These algorithms usually aim to solve problems modeled as stochastic programs. Some of these problems admit nonconvex objective functions. On the other hand, DCA (difference of convex functions algorithm) has proven its strength in tackling a large class of smooth or nonsmooth, nonconvex optimization problems called DC programming. The key advantages of DCA come from its simplicity and flexibility that allows it to treat large-scale problems arising in various contexts. This chapter concerns methods incorporating ideas of stochastic optimization in an online manner into DCA framework to create new algorithms called online stochastic DCA. The first section introduces the chapter. The second section accounts for deterministic DC programming and DCA. The third section briefly reviews stochastic optimization. The fourth section is dedicated to stochastic DC programming and DCA, where we propose two online stochastic DCA schemes for solving a class of stochastic DC programs. The last section concludes the chapter with discussions about promising aspects of the topic.
Read moreAutomatic recognition and analysis of human faces and facial expression by LDA using wavelet transform
Linear Discriminant Analysis (LDA) is one of the principal techniques used in face recognition systems. The Linear Discriminant Analysis (LDA) is well-known scheme for feature extraction and dimension reduction. It provides improved performance over the standard Principal Component Analysis (PCA) method of face recognition by introducing the concept of classes and distance between classes. This paper provides an overview of PCA, the various variants of LDA and their basic drawbacks. The proposed method includes a development over classical LDA (i.e. LDA using wavelets transform approach) that enhances performance such as accuracy and time complexity. Experiments on ORL face database clearly demonstrate this and the graphical comparison of the algorithms clearly showcases the improved recognition rate in case of the proposed algorithm.
Read moreSparse Trace Ratio LDA for Supervised Feature Selection.
Classification is a fundamental task in the field of data mining. Unfortunately, high-dimensional data often degrade the performance of classification. To solve this problem, dimensionality reduction is usually adopted as an essential preprocessing technique, which can be divided into feature extraction and feature selection. Due to the ability to obtain category discrimination, linear discriminant analysis (LDA) is recognized as a classic feature extraction method for classification. Compared with feature extraction, feature selection has plenty of advantages in many applications. If we can integrate the discrimination of LDA and the advantages of feature selection, it is bound to play an important role in the classification of high-dimensional data. Motivated by the idea, we propose a supervised feature selection method for classification. It combines trace ratio LDA with l2,p -norm regularization and imposes the orthogonal constraint on the projection matrix. The learned row-sparse projection matrix can be used to select discriminative features. Then, we present an optimization algorithm to solve the proposed method. Finally, the extensive experiments on both synthetic and real-world datasets indicate the effectiveness of the proposed method.
Read moreSparse semi-supervised support vector machines by DC programming and DCA
Sparse semi-supervised support vector machines by DC programming and DCA
Regularized Discriminant Analysis Incorporating Prior Knowledge on Gene Functional Groups
In the last decade, the renaissance of interest in discriminant analysis has been primarily motivated by possible applications to tumor classification using highdimensional microarray-based data. In this thesis, we do three things: 1. First, we introduce a new regularizing covariance estimation procedure we refer to as SHIP: SHrinking and Incorporating Prior knowledge. The resulting covariance estimator is based on the shrinkage estimator by Ledoit and Wolf [31, 33, 32], but additionally incorporates prior knowledge on gene functional groups extracted from the database KEGG. In order to integrate this knowledge into the shrinkage estimator, we develop multiple options. Instead of using a standard cross-validation procedure for determining the optimal shrinkage intensity, we determine it analytically as introduced by Ledoit and Wolf. 2. Second, we propose a variant of regularized linear discriminant analysis. This method generalizes the idea of the shrinkage estimator from above into the linear discriminant analysis (LDA). 3. Third, we apply our method to public gene expression data sets and examine the classification performance in both the binary and the c-nary case, where c > 2. We choose the diagonal linear discriminant analysis and the nearest shrunken centroids method [15] as competitors. It is shown that the rlda.TG one of our variants of LDA ‘via the SHIP’ performs well in all classification problems and even outperforms, albeit marginally, the competitors in some situations. Unexpectedly, we find that another variant of LDA which is based on the shrinkage estimator by Ledoit and Wolf and which does not incorporate any biological knowledge is as competitive as the rlda.TG.
Read moreA DC Programming Approach for Sparse Linear Discriminant Analysis
We consider the supervised pattern classification in the high-dimensional setting, in which the number of features is much larger than the number of observations. We present a novel approach to the sparse linear discriminant analysis (LDA) using the zero-norm. The resulting optimization problem is non-convex, discontinuous and very hard to solve. We overcome the discontinuity by using an appropriate continuous approximation to zero-norm such that the resulting problem can be formulated as a DC (Difference of Convex functions) program to which DC programming and DC Algorithms (DCA) can be investigated. The computational results show the efficiency and the superiority of our approach versus the l 1 regularization model on both feature selection and classification.KeywordsClassificationSparse Fisher linear discriminant analysisDC programmingDCA
Read moreA DC Programming Approach for Sparse Optimal Scoring
We consider the supervised classification problem in the high-dimensional setting. High-dimensionality makes the application of most classification difficult. We present a novel approach to the sparse linear discriminant analysis (LDA) based on its optimal scoring interpretation and the zero-norm. The difficulty in treating the zero-norm is overcome by using an appropriate continuous approximation such that the resulting problem can be formulated as a DC (Difference of Convex functions) program to which DCA (DC Algorithms) is investigated. The computational results on both simulated data and real microarray cancer data show the efficiency of the proposed algorithm in feature selection as well as classification.
Read moreComparison between visual assessments and different variants of linear discriminant analysis to the classification of Raman patterns of inkjet printer inks
Comparison between visual assessments and different variants of linear discriminant analysis to the classification of Raman patterns of inkjet printer inks
Read morePerformance Evaluation of Exponential Discriminant Analysis with Feature Selection for Steganalysis
The performance of supervised learning-based seganalysis depends on the choice of both classifier and features which represent the image. Features extracted from images may contain irrelevant and redundant features which makes them inefficient for machine learning. Relevant features not only decrease the processing time to train a classifier but also provide better generalisation. Linear discriminant classifier which is commonly used for classification may not be able to classify in better way non-linearly separable data. Recently, exponential discriminant analysis, a variant of linear discriminant analysis (LDA), is proposed which transforms the scatter matrices to a new space by distance diffusion mapping. This provides exponential discriminant analysis (EDA) much more discriminant power to classify non-linearly separable data and helps in improving classification accuracy in comparison to LDA. In this paper, the performance of EDA in conjunction with feature selection methods has been investigated. For feature selection, Kullback divergence, Chernoff distance measures and linear regression measures are used to determine relevant features from higher-order statistics of images. The performance is evaluated in terms classification error and computation time. Experimental results show that exponential discriminate analysis in conjunction with linear regression significantly performs better in terms of both classification error and compilation time of training classifier. Defence Science Journal, 2012, 62(1), pp.19 -24 , DOI:http://dx.doi.org/10.14429/dsj. 62.1437
Read moreD.C. programming for sparse proximal support vector machines
D.C. programming for sparse proximal support vector machines
Balance Ratio Sum Versus Maximization Ratio Sum for Linear Discriminant Analysis.
Linear discriminant analysis (LDA) is a widely used dimensionality reduction (DR) technique that is effective in extracting discriminative features across various fields. Ratio sum LDA (RSLDA), a variant of LDA, was developed to address the shortcoming of the LDA method, which tends to obtain features with weak discriminative information. However, the traditional ratio sum formulation is dominated by the maximum ratio, which makes it difficult to select highly discriminative features and contradicts the original goal of the ratio sum. In this article, we analyzed the underlying causes of the dominance problem in RSLDA. A novel discriminant feature learning method via balance ratio sum discriminant analysis (BRSDA) is proposed. BRSDA effectively balances the ratios of the model formulation, thereby mitigating the domination problem. It focuses on optimizing low-quality projection directions, thereby yielding a well-balanced solution with consistently strong projection quality. First, a minimization ratio sum (Min-RS) criterion is adopted, which leverages the balance property of the harmonic mean to balance the gaps between the ratios. Second, Min-RS is integrated with the $\ell _{p}$ -norm to further balance gaps between ratios. By amplifying the differences across projection directions, the $\ell _{p}$ -norm drives BRSDA to emphasize the optimization of low-quality directions, thus raising the lower bound of direction quality. Finally, since obtaining a closed-form solution for the BRSDA problem is challenging, the gradient descent method is employed to solve its optimization. Sufficient experimental results verify the effectiveness of BRSDA, and BRSDA can effectively solve the domination problem and extract discriminative features.
Read more<title>Boosting multiple classifiers by hybrid discriminant analysis for face identification</title>
Linear Discriminant Analysis (LDA) has been widely applied in the field of face identification because of its simplicity and efficiency in capturing the most discriminant features. However LDA often fails when facing the change in illumination, pose or small training size. To overcome those difficulties, Principal Component Analysis (PCA), which recover the most descriptive/informative features in the reduced dimension feature space, are often used in preprocessing stage. Although there is a trend of preferring LDA over PCA in classification, it has been found that PCA may perform better than LDA in some cases, especially when the size of the training set is small. To better combine the merits of PCA and LDA, some rule-based parametric combination of PCA and LDA methods have been proposed. However in those methods the optimal parameter setting is not guaranteed and can only be approximated by exhaustive search. In this paper we propose a learning-based framework that can unify PCA and LDA in adaptively finding both discriminant and descriptive feature. To eliminate the parameter selection, we incorporate a non-linear boosting process to enhance a pool of hybrid classifiers and combine them into a more accurate one. To evaluate the performance of our boosted hybrid method, we compare it to state-of-the-art LDA variants and traditional PCA-LDA technique on three widely used face image benchmark databases. The experiment results show that our novel boosted hybrid discriminant analysis outperforms the other techniques and the best single hybrid classifier.
Read more