A feature selection method for nonparallel plane support vector machine classification

Least squares twin support vector machine (LSTSVM) is a new machine learning method, as opposed to solving two quadratic programming problems in twin support vector machine (TWSVM), which generates two nonparallel hyperplanes by solving a pair of linear system of equations. However, LSTSVM obtains the resultant classifier by giving same importance to all training samples which may be important for classification performance. In this paper, by considering the fuzzy membership value for each sample, we propose an entropy-based fuzzy least squares twin support vector machine where fuzzy membership values are assigned based on the entropy values of all training samples. The proposed method not only retains the superior characteristics of LSTSVM which is simple and fast algorithm, but also implements the structural risk minimization principle to overcome the possible over- fitting problem. Experiments are performed on several synthetic as well as benchmark datasets and the experimental results illustrate the effectiveness of our method.

In order to reduce the higher training cost of support vector machine (SVM) and its sensitivity towards noise and outliers, two fuzzy based approaches are proposed in this paper. The proposed approaches are based on least squares twin support vector machine (LSTWSVM) and fuzzy support vector machine (FSVM). The effects of noise and outliers are reduced by assigning lower membership values to the data points which are away from the class centers. Further, 2-norm of the slack vectors of the LSTWSVM formulation is taken after multiplying to their respective diagonal matrices of the membership values to effectively utilize the fuzzy membership principle and to make the optimization problem strongly convex. Moreover, the proposed approaches solve linear equations instead of quadratic programming problems which help in training faster. The effectiveness of the proposed approaches are established by comparing the classification accuracies and training time with support vector machine, fuzzy support vector machine, twin support vector machine and least squares twin support vector machine.

Sparse pinball twin support vector machines

In this paper, a novel unconstrained convex minimization problem formulation for the Lagrangian dual of the recently introduced twin support vector machine (TWSVM) in simpler form is proposed for constructing binary classifiers. Since the objective functions of the modified minimization problems contain non-smooth `plus' function, we solve them by Newton iterative method either by considering their generalized Hessian matrices or replacing the `plus' function by a smooth approximation function. Numerical experiments were performed on a number of interesting real-world benchmark data sets. Computational results clearly illustrates the effectiveness and the applicability of the proposed approach as comparable or better generalization performance with faster learning speed is obtained in comparison with SVM, least squares TWSVM (LS-TWSVM) and TWSVM.

Weighted Twin Support Vector Machines with Local Information and its application

Support vector machine (SVM) has a good effect in the supervised classification of hyperspectral images. In view of the shortcomings of the existing parallel structure SVM, this article proposes a non-parallel SVM model. Based on the traditional parallel boundary structure vector machine, this model adds an additional empirical risk minimization term to the original optimization problem by adding the least square term of the sample and obtains two non-parallel hyperplanes, respectively, forming a new non-parallel SVM algorithm to minimize the additional empirical risk of non-parallel SVM (Additional Empirical Risk Minimization Non-parallel Support Vector Machine, AERM-NPSVM). On the basis of AERM-NPSVM, the bias constraint is added to it, and AERM-NPSVM (BC-AERM-NPSVM) is further obtained. The experimental results show that, compared with the traditional parallel SVM model and the classical non-parallel SVM model, Twin Support Vector Machine (TWSVM), the new model, has a better effect in hyperspectral image classification and better generalization performance.

Discriminative information-based nonparallel support vector machine

Twin support vector machine (TWSVM) and twin support vector regression (TSVR) are newly emerging efficient machine learning techniques which offer promising solutions for classification and regression challenges respectively. TWSVM is based upon the idea to identify two nonparallel hyperplanes which classify the data points to their respective classes. It requires to solve two small sized quadratic programming problems (QPPs) in lieu of solving single large size QPP in support vector machine (SVM) while TSVR is formulated on the lines of TWSVM and requires to solve two SVM kind problems. Although there has been good research progress on these techniques; there is limited literature on the comparison of different variants of TSVR. Thus, this review presents a rigorous analysis of recent research in TWSVM and TSVR simultaneously mentioning their limitations and advantages. To begin with, we first introduce the basic theory of support vector machine, TWSVM and then focus on the various improvements and applications of TWSVM, and then we introduce TSVR and its various enhancements. Finally, we suggest future research and development prospects.

Classical support vector machine (SVM) and its twin variant twin support vector machine (TWSVM) utilize the Hinge loss that shows linear behaviour, whereas the least squares version of SVM (LSSVM) and twin least squares support vector machine (LSTSVM) uses L2-norm of error which shows quadratic growth. The robust Huber loss function is considered as the generalization of Hinge loss and L2-norm loss that behaves like the quadratic L2-norm loss for closer error points and the linear Hinge loss after a specified distance. Three functional iterative approaches based on generalized Huber loss function are proposed in this paper to solve support vector classification problems of which one is based on SVM, i.e. generalized Huber support vector machine and the other two are in the spirit of TWSVM, namely generalized Huber twin support vector machine and regularization on generalized Huber twin support vector machine. The proposed approaches iteratively find the solutions and eliminate the requirements to solve any quadratic programming problem (QPP) as for SVM and TWSVM. The main advantages of the proposed approach are: firstly, utilize the robust Huber loss function for better generalization and for lesser sensitivity towards noise and outliers as compared to quadratic loss; secondly, it uses functional iterative scheme to find the solution that eliminates the need to solving QPP and also makes the proposed approaches faster. The efficacy of the proposed approach is established by performing numerical experiments on several real-world datasets and comparing the result with related methods, viz. SVM, TWSVM, LSSVM and LSTSVM. The classification results are convincing.

Support vector machine (SVM), as a supervised learning method, has different kinds of varieties with significant performance. In recent years, more research focused on nonparallel SVM, where twin SVM (TWSVM) is the typical one. In order to reduce the influence of outliers, more robust distance measurements are considered in these methods, but the discriminability of the models is neglected. In this article, we propose robust manifold twin bounded SVM (RMTBSVM), which considers both robustness and discriminability. Specifically, a novel norm, that is, capped L₁-norm, is used as the distance metric for robustness, and a robust manifold regularization is added to further improve the robustness and classification performance. In addition, we also use the kernel method to extend the proposed RMTBSVM for nonlinear classification. We introduce the optimization problems of the proposed model. Subsequently, effective algorithms for both linear and nonlinear cases are proposed and proved to be convergent. Moreover, the experiments are conducted to verify the effectiveness of our model. Compared with other methods under the SVM framework, the proposed RMTBSVM shows better classification accuracy and robustness.

abstractUniversum, a third class not belonging to either class of the classification problem, allows to incorporate the prior knowledge into the learning process. A lot of previous work have demonstrated that the Universum is helpful to the supervised and semi-supervised classification. Moreover, Universum has already been introduced into the support vector machine (SVM) and twin support vector machine (TSVM) to enhance the generalisation performance. To further increase the generalisation performance, we propose a least squares TSVM with Universum data (-TSVM) in this paper. Our -TSVM possesses the following advantages: first, it exploits Universum data to improve generalisation performance. Besides, it implements the structural risk minimisation principle by adding a regularisation to the objective function. Finally, it costs less computing time by solving two small-sized systems of linear equations instead of a single larger-sized quadratic programming problem. To verify the validity of our proposed algorithm, we conduct various experiments around the size of labelled samples and the number of Universum data on data-sets including seven benchmark data-sets, Toy data, MNIST and Face images. Empirical experiments indicate that Universum contributes to making prediction accuracy improved even stable. Especially when fewer labelled samples given, -TSVM is far superior to the improved LS-TSVM (ILS-TSVM), and slightly superior to the -TSVM.

The traditional vector-based algorithms, such as Support Vector Machine (SVM) and Twin Support Vector Machine (TSVM), have many limitations especially when tensor is considered as input matrix. In this paper, we proposed a novel algorithm with a tensor-based classification paradigm to utilize the structural information. The new proposed algorithm is called  -Twin Support Tensor Machine ( -TSTM), which is an extension of  -Twin Support Vector Machine ( -TSVM). Similarly,  -TSTM solves a pair of smaller-sized Quadratic Programming Problems (QPPs). It reduces the computational complexity substantially. Besides, we formulate  -TSTM, which separates samples in the tensor space with two non-parallel hyperplanes, and the pair of parameters ( ) have theoretical interpretation which are used to control the bounds of the fractions of support tensors and the error margins. What’s more, the structure information of data is retained by the direct use of tensor representation. The proposed  -TSTM can preferably overcome overfitting problem and deal with big data while most vector-based algorithms could hardly compare. In addition, it has better performances on high dimensional and small-sample-size (S3) problem. The efficiency and superiority of the proposed method are demonstrated by experiments on various datasets.

Twin support vector machine (TWSVM) is a new machine learning method, as opposed to solving a single quadratic programming problem in support vector machine (SVM), which generates two nonparallel hyperplanes by solving two smaller size quadratic programming problems. However, the TWSVM obtains the final classifier by giving the same importance to all training samples which may be important for classification performance. In order to address this problem, in this paper, we propose a novel entropy-based fuzzy twin bounded support vector machine (EFTBSVM) for binary classification problems. By considering the fuzzy membership value for each sample and assigning it based on the entropy value, the samples with higher class certainty are assigned to relatively larger fuzzy membership. In addition, the proposed EFTBSVM not only maintains the superior characteristics of the TWSVM but also exploits the structural risk minimization principle by introducing a regularization term. The experimental results achieved on synthetic datasets and benchmark datasets illustrate the effectiveness of the proposed method.

Color image classification and retrieval through ternary decision structure based multi-category TWSVM

Twin support vector machines (TWSVM) is based on the idea of proximal SVM based on generalized eigenvalues (GEPSVM), which determines two nonparallel planes by solving two related SVM-type problems, so that its computing cost in the training phase is 1/4 of standard SVM. In addition to keeping the superior characteristics of GEPSVM, the classification performance of TWSVM significantly outperforms that of GEPSVM. However, the stand-alone method requires the solution of two smaller quadratic programming problems. This paper mainly reviews the research progress of TWSVM. Firstly, it analyzes the basic theory and the algorithm thought of TWSVM, then tracking describes the research progress of TWSVM including the learning model and specific applications in recent years, finally points out the research and development prospects.