Nonparallel Support Vector Machine with L2-norm Loss and its DCD-type Solver

Union nonparallel support vector machines framework with consistency

Support vector machine (SVM) has a good effect in the supervised classification of hyperspectral images. In view of the shortcomings of the existing parallel structure SVM, this article proposes a non-parallel SVM model. Based on the traditional parallel boundary structure vector machine, this model adds an additional empirical risk minimization term to the original optimization problem by adding the least square term of the sample and obtains two non-parallel hyperplanes, respectively, forming a new non-parallel SVM algorithm to minimize the additional empirical risk of non-parallel SVM (Additional Empirical Risk Minimization Non-parallel Support Vector Machine, AERM-NPSVM). On the basis of AERM-NPSVM, the bias constraint is added to it, and AERM-NPSVM (BC-AERM-NPSVM) is further obtained. The experimental results show that, compared with the traditional parallel SVM model and the classical non-parallel SVM model, Twin Support Vector Machine (TWSVM), the new model, has a better effect in hyperspectral image classification and better generalization performance.

Nonparallel support vector machine with large margin distribution for pattern classification

Over the past decades, 1-norm techniques based on algorithms are widely used to suppress input features. Quite different from traditional 1-norm support vector machine (SVM), direct 1-norm optimization based on the primal problem of nonparallel plane classifiers like generalized proximal support vector machine, twin support vector machine (TWSVM) and least squares twin support vector machine (LSTSVM) are not capable of generating very sparse solutions that are vital for classification and can make them easier to store and faster to compute. To address the issue, in this paper, we develop a feature selection method for LSTSVM, called a feature selection method for nonparallel plane support vector machine classification (FLSTSVM), which is specially designed for strong feature suppression. We incorporate a Tikhonov regularization term to the objective of LSTSVM, and then minimize its 1-norm measure. Solution of FLSTSVM can follow directly from solving two smaller quadratic programming problems (QPPs) arising from two primal QPPs as opposed to two dual ones in TWSVM. FLSTSVM is capable of generating very sparse solutions. This means that FLSTSVM can reduce input features, for the linear case. When a nonlinear classifier is used, few kernel functions determine the classifier. In addition to having strong feature suppression, the edge of our method still lies in its faster computing time compared to that of TWSVM, Newton Method for Linear Programming SVM (NLPSVM) and LPNewton. Lastly, this algorithm is compared on public data sets, as well as an Exclusive Or (XOR) example.

Discriminative information-based nonparallel support vector machine

Support vector machine (SVM), as a supervised learning method, has different kinds of varieties with significant performance. In recent years, more research focused on nonparallel SVM, where twin SVM (TWSVM) is the typical one. In order to reduce the influence of outliers, more robust distance measurements are considered in these methods, but the discriminability of the models is neglected. In this article, we propose robust manifold twin bounded SVM (RMTBSVM), which considers both robustness and discriminability. Specifically, a novel norm, that is, capped L₁-norm, is used as the distance metric for robustness, and a robust manifold regularization is added to further improve the robustness and classification performance. In addition, we also use the kernel method to extend the proposed RMTBSVM for nonlinear classification. We introduce the optimization problems of the proposed model. Subsequently, effective algorithms for both linear and nonlinear cases are proposed and proved to be convergent. Moreover, the experiments are conducted to verify the effectiveness of our model. Compared with other methods under the SVM framework, the proposed RMTBSVM shows better classification accuracy and robustness.

In this paper, we propose a nonparallel support vector machine with pinball loss (Pin-NPSVM) that deals with the noise sensitivity and resampling instability of NPSVM. More specifically, we redefine a pinball loss funtion and build a pair of quantile hyper-planes. Each quantile hyper-plane is constructed by using the new pinball loss instead of ɛ -insensitive loss, which makes the new classification model be insensitive to noise samples, especially for feature noise samples around the decision boundary. Moreover, instead of hinge loss, Pin-NPSVM also builds a pair of decision boundaries based on traditional pinball loss, which further improves the anti-nosie ability of the classification model. In a word, Pin-NPSVM not only inherits the characteristics of the nonparallel optimal hyper-planes, but also has a consistent model with Pin-SVM, which can process noise data well. Finally, numerical experimental results show that the Pin-NPSVM has more obvious advantages than other models in classification performance, especially for noise datasets.

Classical support vector machine (SVM) and its twin variant twin support vector machine (TWSVM) utilize the Hinge loss that shows linear behaviour, whereas the least squares version of SVM (LSSVM) and twin least squares support vector machine (LSTSVM) uses L2-norm of error which shows quadratic growth. The robust Huber loss function is considered as the generalization of Hinge loss and L2-norm loss that behaves like the quadratic L2-norm loss for closer error points and the linear Hinge loss after a specified distance. Three functional iterative approaches based on generalized Huber loss function are proposed in this paper to solve support vector classification problems of which one is based on SVM, i.e. generalized Huber support vector machine and the other two are in the spirit of TWSVM, namely generalized Huber twin support vector machine and regularization on generalized Huber twin support vector machine. The proposed approaches iteratively find the solutions and eliminate the requirements to solve any quadratic programming problem (QPP) as for SVM and TWSVM. The main advantages of the proposed approach are: firstly, utilize the robust Huber loss function for better generalization and for lesser sensitivity towards noise and outliers as compared to quadratic loss; secondly, it uses functional iterative scheme to find the solution that eliminates the need to solving QPP and also makes the proposed approaches faster. The efficacy of the proposed approach is established by performing numerical experiments on several real-world datasets and comparing the result with related methods, viz. SVM, TWSVM, LSSVM and LSTSVM. The classification results are convincing.

Kernel partial least squares (KPLS) has been known as a generic kernel regression method and proven to be competitive with other kernel regression methods such as support vector machines for regression (SVM) and kernel ridge regression. Kernel boosted latent features (KBLF) is a variant of KPLS for any differentiable convex loss functions. It provides a more flexible framework for various predictive modeling tasks such as classification with logistic loss and robust regression with L1 norm loss, etc. However, KPLS and KBLF solutions are dense and thus not suitable for large-scale computations. Sparsification of KPLS solutions has been studied for dual and primal forms. For dual sparsity, it requires solving a nonlinear optimization problem at every iteration step and its computational burden limits its applicability to general regression tasks.In this paper, we propose simple heuristics to approximate sparse solutions for KPLS and the framework is also applied for sparsifying KBLF solutions. The algorithm provides an interesting "path" from a maximum residual criterion based algorithm with orthogonality conditions to the dense KPLS/KBLF. With the orthogonality, it differentiates itself from many existing forward selection-type algorithms. The computational advantage is illustrated by benchmark datasets and comparison to SVM is done.

A novel projection nonparallel support vector machine for pattern classification

Integrating genomic datasets from homogenous or disparate sources to identify genes that are commonly or uniquely expressed remains a largely underexplored area. Such integrative analysis can reveal biologically relevant genes that are common or exclusive across datasets or within specific conditions or cohorts. Identifying these gene expression profiles and employing them to classify disease status can aid in the development of vaccines, diagnostics and targeted therapeutics with efficacy against difficult-to-treat medically important pathogens and cancer. This work develops new methodologies to integrate transcriptomic patterns from the lungs and spleen tissues infected by Francisella tularensis – Schu4 and Live Vaccine Strain (LVS). Our objective is to (i) identify biologically relevant gene features indicative of respiratory infection, disease severity, and bacterial dissemination to the spleen, and (ii) develop a Weighted $$\ell _1$$-norm Non-Parallel Support Vector Machines ($$\ell _1$$-WNPSVM) that will utilize the selected genes to predict disease status. The $$\ell _1$$-WNPSVM is trained on the lungs data and validated on the spleen data, introducing a form of transfer learning, with uninfected controls and Schu4 or LVS samples as classes. Currently, a direct application of existing NPSVM-type methods to analyze gene expression datasets, where the number of genes significantly exceeds the number of samples, is computationally impractical due to their large memory requirements. This work addresses these challenges and also generalizes to models of similar formulations by incorporating dimensionality reduction and gene selection into the NPSVM-type frameworks. The $$\ell _1$$-WNPSVM method outperforms traditional machine learning techniques such as ANN, XGBoost, AdaBoost, GradBoost, KNN, SVM, Naive Bayes, Random Forest, Logistic Regression, and Decision Tree, achieving a $$97\%$$ balanced accuracy on imbalanced data. We discovered sets of 235 genes exclusively expressed in the lungs and spleen tissues and utilized them to classify bacterial strains and controls, enabling prediction of disease status. Gene ontology is performed to reveal underlying metabolic pathways. Our analysis shows that signal transduction and disease (cancer) pathways are the most significant pathways activated in the lungs while gene expression (transcription), immune system, and disease (cancer) pathways are activated in the spleen. Collectively, these pathways indicate a significant host response to infection, including how the bacteria interact with host tissues during dissemination.

Fast Nonparallel Support Vector Machine with the Margin Hyper-planes and Its Iterative Solver

In this paper, we propose a novel clustering method with feature selection in a synchronized manner, called iterative tighter nonparallel support vector clustering with simultaneous feature selection (IT-NHSVC-SFS). A certain iterative (alternating) optimization strategy for clustering is applied to a learning model with twin hyperplanes, in which two types of regularizers, namely the Euclidean and infinite norms, are introduced to achieve the enhancement of clustering generalization performance and coordinated feature selection. The L-infinite norm actually conducts implicit feature elimination process to reduce clustering noises resulting from irrelevant features, thus guaranteeing clustering accuracy. Meanwhile, since the formulation of the proposed model embodies the large-margin spirit,good generalization can also be ensured.Unlike twin support vector machine and its variants, nonparallel hyperplane SVM (NHSVM) is chosen to be a baseline model,thus only a single quadratic programming problem is needed to solve for the optimal twin hyperplanes, making it convenient to design a synchronized feature selection process in two hyperplanes. Additionally, two more groups of equality constraints are enforced into the original constraint set of NHSVM, thus the inverse operation of two large matrices can be avoided to reduce the computational complexity. Furthermore,the hinge loss function of NHSVM is replaced by the Laplacian loss measure to prevent the premature convergence. Numerical experiments are performed on benchmark datasets to investigate the validity of the proposed algorithm. The experimental results indicate that IT-NHSVC-SFS has better performance than other existing clustering methods mainly in terms of clustering accuracy.

A novel robust nonparallel support vector classifier based on one optimization problem