Comprehensive review on twin support vector machines

The traditional vector-based algorithms, such as Support Vector Machine (SVM) and Twin Support Vector Machine (TSVM), have many limitations especially when tensor is considered as input matrix. In this paper, we proposed a novel algorithm with a tensor-based classification paradigm to utilize the structural information. The new proposed algorithm is called  -Twin Support Tensor Machine ( -TSTM), which is an extension of  -Twin Support Vector Machine ( -TSVM). Similarly,  -TSTM solves a pair of smaller-sized Quadratic Programming Problems (QPPs). It reduces the computational complexity substantially. Besides, we formulate  -TSTM, which separates samples in the tensor space with two non-parallel hyperplanes, and the pair of parameters ( ) have theoretical interpretation which are used to control the bounds of the fractions of support tensors and the error margins. What’s more, the structure information of data is retained by the direct use of tensor representation. The proposed  -TSTM can preferably overcome overfitting problem and deal with big data while most vector-based algorithms could hardly compare. In addition, it has better performances on high dimensional and small-sample-size (S3) problem. The efficiency and superiority of the proposed method are demonstrated by experiments on various datasets.

Weighted Twin Support Vector Machines with Local Information and its application

Twin support vector machine (TWSVM) is a new machine learning method, as opposed to solving a single quadratic programming problem in support vector machine (SVM), which generates two nonparallel hyperplanes by solving two smaller size quadratic programming problems. However, the TWSVM obtains the final classifier by giving the same importance to all training samples which may be important for classification performance. In order to address this problem, in this paper, we propose a novel entropy-based fuzzy twin bounded support vector machine (EFTBSVM) for binary classification problems. By considering the fuzzy membership value for each sample and assigning it based on the entropy value, the samples with higher class certainty are assigned to relatively larger fuzzy membership. In addition, the proposed EFTBSVM not only maintains the superior characteristics of the TWSVM but also exploits the structural risk minimization principle by introducing a regularization term. The experimental results achieved on synthetic datasets and benchmark datasets illustrate the effectiveness of the proposed method.

Over the past decades, 1-norm techniques based on algorithms are widely used to suppress input features. Quite different from traditional 1-norm support vector machine (SVM), direct 1-norm optimization based on the primal problem of nonparallel plane classifiers like generalized proximal support vector machine, twin support vector machine (TWSVM) and least squares twin support vector machine (LSTSVM) are not capable of generating very sparse solutions that are vital for classification and can make them easier to store and faster to compute. To address the issue, in this paper, we develop a feature selection method for LSTSVM, called a feature selection method for nonparallel plane support vector machine classification (FLSTSVM), which is specially designed for strong feature suppression. We incorporate a Tikhonov regularization term to the objective of LSTSVM, and then minimize its 1-norm measure. Solution of FLSTSVM can follow directly from solving two smaller quadratic programming problems (QPPs) arising from two primal QPPs as opposed to two dual ones in TWSVM. FLSTSVM is capable of generating very sparse solutions. This means that FLSTSVM can reduce input features, for the linear case. When a nonlinear classifier is used, few kernel functions determine the classifier. In addition to having strong feature suppression, the edge of our method still lies in its faster computing time compared to that of TWSVM, Newton Method for Linear Programming SVM (NLPSVM) and LPNewton. Lastly, this algorithm is compared on public data sets, as well as an Exclusive Or (XOR) example.

In this paper, for an improved twin support vector machine (TWSVM), we give it a theoretical explanation based on the concept of Universum and then name it Self-Universum support vector machine (SUSVM). For the binary classification problem, SUSVM takes the positive class and negative class as Universum separately to construct two classification problems with Universum; therefore, two nonparallel hyperplanes are derived. SUSVM has several improved advantages compared with TWSVMs. Furthermore, we improve SUSVM by formulating it as a pair of linear programming problems instead of quadratic programming problems (QPPs), which leads to the better generalization performance and less computational time. The effectiveness of the enhanced method is demonstrated by experimental results on several benchmark datasets.

The training algorithm of classical twin support vector regression (TSVR) can be attributed to the solution of a pair of quadratic programming problems (QPPs) with inequality constraints in the dual space. However, this solution is affected by time and memory constraints when dealing with large datasets. In this paper, we present a least squares version for TSVR in the primal space, termed primal least squares TSVR (PLSTSVR). By introducing the least squares method, the inequality constraints of TSVR are transformed into equality constraints. Furthermore, we attempt to directly solve the two QPPs with equality constraints in the primal space instead of the dual space; thus, we need only to solve two systems of linear equations instead of two QPPs. Experimental results on artificial and benchmark datasets show that PLSTSVR has comparable accuracy to TSVR but with considerably less computational time. We further investigate its validity in predicting the opening price of stock.

Classical support vector machine (SVM) and its twin variant twin support vector machine (TWSVM) utilize the Hinge loss that shows linear behaviour, whereas the least squares version of SVM (LSSVM) and twin least squares support vector machine (LSTSVM) uses L2-norm of error which shows quadratic growth. The robust Huber loss function is considered as the generalization of Hinge loss and L2-norm loss that behaves like the quadratic L2-norm loss for closer error points and the linear Hinge loss after a specified distance. Three functional iterative approaches based on generalized Huber loss function are proposed in this paper to solve support vector classification problems of which one is based on SVM, i.e. generalized Huber support vector machine and the other two are in the spirit of TWSVM, namely generalized Huber twin support vector machine and regularization on generalized Huber twin support vector machine. The proposed approaches iteratively find the solutions and eliminate the requirements to solve any quadratic programming problem (QPP) as for SVM and TWSVM. The main advantages of the proposed approach are: firstly, utilize the robust Huber loss function for better generalization and for lesser sensitivity towards noise and outliers as compared to quadratic loss; secondly, it uses functional iterative scheme to find the solution that eliminates the need to solving QPP and also makes the proposed approaches faster. The efficacy of the proposed approach is established by performing numerical experiments on several real-world datasets and comparing the result with related methods, viz. SVM, TWSVM, LSSVM and LSTSVM. The classification results are convincing.

abstractUniversum, a third class not belonging to either class of the classification problem, allows to incorporate the prior knowledge into the learning process. A lot of previous work have demonstrated that the Universum is helpful to the supervised and semi-supervised classification. Moreover, Universum has already been introduced into the support vector machine (SVM) and twin support vector machine (TSVM) to enhance the generalisation performance. To further increase the generalisation performance, we propose a least squares TSVM with Universum data (-TSVM) in this paper. Our -TSVM possesses the following advantages: first, it exploits Universum data to improve generalisation performance. Besides, it implements the structural risk minimisation principle by adding a regularisation to the objective function. Finally, it costs less computing time by solving two small-sized systems of linear equations instead of a single larger-sized quadratic programming problem. To verify the validity of our proposed algorithm, we conduct various experiments around the size of labelled samples and the number of Universum data on data-sets including seven benchmark data-sets, Toy data, MNIST and Face images. Empirical experiments indicate that Universum contributes to making prediction accuracy improved even stable. Especially when fewer labelled samples given, -TSVM is far superior to the improved LS-TSVM (ILS-TSVM), and slightly superior to the -TSVM.

In this paper, a novel nonparallel hyperplane based classifier termed as “angle-based twin support vector machine” (ATWSVM) has been proposed which is motivated by the concept of twin support vector machine (TWSVM). TWSVM obtains two nonparallel hyperplanes by solving a pair of quadratic programming problems (QPPs). ATWSVM presents a generic classification model, where the first problem can be formulated using a TWSVM-based classifier and the second problem is an unconstrained minimization problem (UMP) which is reduced to solving a system of linear equations. The second hyperplane is determined so that it is proximal to its own class and the angle between the normals to the two hyperplanes is maximized. The notion of angle has been introduced to have maximum separation between the two hyperplanes. In this work, we have presented two versions of ATWSVM: one that solves a QPP and a UMP; second which formulates both the problems as UMPs. The training time of ATWSVM is much less than that of TWSVM because ATWSVM solves the second problem as UMP instead of QPP. To test the efficacy of the proposed classifier, experiments have been conducted on synthetic and benchmark datasets and it is observed that the proposed classifier achieves classification accuracy comparable or better than that of TWSVM. This work also proposes application of ATWSVM for color image segmentation.

Sparse pinball twin support vector machines

Ship dynamic models serve as the foundation for designing ship controllers, trajectory planning, and obstacle avoidance. Support vector regression (SVR) is a commonly used nonparametric modelling method for ship dynamics. Achieving high accuracy SVR models requires a substantial amount of training samples. Additionally, as the number of training samples increases, the computational efficiency for solving the quadratic programming problem (QPP) of SVR decreases. Ship controllers demand dynamic models with both high accuracy and computational efficiency. Therefore, to enhance the prediction accuracy and computational efficiency of SVR, this paper proposes a nonparametric modelling method based on twin SVR (TSVR). TSVR replaces a large QPP with a set of smaller QPPs, significantly enhancing generalizability and computational efficiency. To further improve the predictive accuracy of TSVR, the puma optimizer algorithm is employed to determine the optimal hyperparameters. The performance of the proposed method is validated using a Mariner class vessel. Gaussian white noise is introduced into the modelling data to simulate measurement error. The TSVR model accurately predicts various zigzag and turning circle manoeuvring motions under disturbance conditions, demonstrating its robustness and generalizability. Compared to the SVR model, the TSVR model achieves lower root mean square error and computational time, confirming its superior predictive accuracy and computational efficiency.

Color image classification and retrieval through ternary decision structure based multi-category TWSVM

Support vector machine (SVM) has attracted substantial interest in the community of machine learning. As the extension of SVM, nonparallel hyperplane SVM (NHSVM) classification algorithms have become current researching hot spots in machine learning during the last few years. For binary classification tasks, the idea of NHSVM algorithms is to find a hyperplane for each class, such that each hyperplane is proximal to the data points of one class and far from the data points of the other class. Compared with the classical SVM, NHSVM algorithms have lower computational complexity, work better on XOR problems and can get better generalization performance. This paper reviews three representative NHSVM algorithms, including generalized eigenvalue proximal SVM (GEPSVM), twin SVM (TWSVM) and projection twin SVM (PTSVM), and gives the research progress of them. The aim of this overview is to provide an insightful organization of current developments of NHSVM algorithms, identify their limitations and give suggestions for further research.

KNN weighted reduced universum twin SVM for class imbalance learning

The main purpose of twin support vector regression (TSVR) is to find linear or nonlinear relationships in sample data, and then predict future data. TSVR is the decomposition of a large convex quadratic programming problem into two small convex quadratic programming problems. Therefore, TSVR not only has the advantages of fast computation and low computational complexity, but also has better regression performance. Classic SVR, TSVR is assuming that accords with mean zero, variance of noise with the variance of the gaussian distribution or not to consider the effects of noise, but in some practical applications, such as wind speed forecasting, noise characteristic is more in line with mean zero, variance for heteroscedasticity of gaussian distribution, therefore, the return of the existing technology is not the best. In this paper, the characteristics of heteroscedasticity Gaussian noise are introduced into the model TSVR, and the twin support vector regression model based on heteroscedasticity Gaussian noise (TSVR-HGN) is constructed. The Lagrange multiplier method is used to solve the problem, and the optimization algorithm is used to find the global optimization. The artificial data set, UCI data set and wind speed data set were selected for experimental comparison. The experimental results show that TSVR-HGN has better prediction accuracy.