- Research Article
65
- 10.1016/j.knosys.2022.108578
KNN weighted reduced universum twin SVM for class imbalance learning
- Mar 23, 2022
- Knowledge-Based Systems
- M.A Ganaie + 1 more +1
KNN weighted reduced universum twin SVM for class imbalance learning
Purpose Support vector machine (SVM) and twin support vector machine (TSVM) are plane-based classifiers that perform well on balanced datasets. However, their performance significantly degrades on imbalanced datasets due to their limited ability to model class proportions, noise, and outliers. To address this issue, this paper proposes an entropy-based intuitionistic fuzzy least squares twin support vector machine for class imbalance learning (EIFLSTSVM-CIL). Design/methodology/approach This study proposes a class imbalance learning method that embeds entropy-driven intuitionistic fuzzy modeling into the least squares twin support vector machine framework. Specifically, each sample is assigned the degrees of membership and nonmembership by jointly considering its spatial distribution in the feature space and the uncertainty of its class association measured by information entropy. In addition, sample weights are adjusted based on the global imbalance ratio, which enhances the penalization of majority class outliers and noisy instances. The proposed method is evaluated on 21 synthetic datasets and 35 real-world benchmark datasets with varying imbalance ratios. Findings Experimental results on 21 synthetic datasets and 35 real-world datasets demonstrate that EIFLSTSVM-CIL exhibits superior robustness and generalization performance in imbalanced scenarios, showing a significant advantage over classical models. Originality/value This study enhances the twin least squares SVM by jointly incorporating entropy-based intuitionistic fuzzy modeling and class-imbalance weighting. The proposed approach integrates both aspects in a unified framework, improving robustness and generalization on imbalanced data.
KNN weighted reduced universum twin SVM for class imbalance learning
KNN weighted reduced universum twin SVM for class imbalance learning
Ensemble of classification models with weighted functional link network
Ensemble of classification models with weighted functional link network
Regularized robust fuzzy least squares twin support vector machine for class imbalance learning
Twin support vector machines (TWSVM) have been successfully applied to the classification problems. TWSVM is computationally efficient model of support vector machines (SVM). However, in real world classification problems issues of class imbalance and noise provide great challenges. Due to this, models lead to the inaccurate classification either due to higher tendency towards the majority class or due to the presence of noise. We provide an improved version of robust fuzzy least squares twin support vector machine (RFLSTSVM) known as regularized robust fuzzy least squares twin support vector machine (RRFLSTSVM) to handle the imbalance problem. The advantage of RRFLSTSVM over RFLSTSVM is that the proposed RRFLSTSVM implements the structural risk minimization principle by the introduction of regularization term in the primal formulation of the objective functions. This modification leads to the improved classification as it embodies the marrow of statistical learning theory. The proposed RRFLSTSVM doesn't require any extra assumption as the matrices resulting in the dual are positive definite. However, RFLSTSVM is based on the assumption that the inverse of the matrices resulting in the dual always exist as the matrices are positive semi-definite. To subsidize the effects of class imbalance and noise, the data samples are assigned weights via fuzzy membership function. The fuzzy membership function incorporates the imbalance ratio knowledge and assigns appropriate weights to the data samples. Unlike TWSVM which solves a pair of quadratic programming problem (QPP), the proposed RRFLSTSVM method solves a pair of system of linear equations and hence is computationally efficient. Experimental and statistical analysis show the efficacy of the proposed RRFLSTSVM method.
Read moreEntropy based fuzzy least squares twin support vector machine for class imbalance learning
In classification problems, the data samples belonging to different classes have different number of samples. Sometimes, the imbalance in the number of samples of each class is very high and the interest is to classify the samples belonging to the minority class. Support vector machine (SVM) is one of the widely used techniques for classification problems which have been applied for solving this problem by using fuzzy based approach. In this paper, motivated by the work of Fan et al. (Knowledge-Based Systems 115: 87–99 2017), we have proposed two efficient variants of entropy based fuzzy SVM (EFSVM). By considering the fuzzy membership value for each sample, we have proposed an entropy based fuzzy least squares support vector machine (EFLSSVM-CIL) and entropy based fuzzy least squares twin support vector machine (EFLSTWSVM-CIL) for class imbalanced datasets where fuzzy membership values are assigned based on entropy values of samples. It solves a system of linear equations as compared to the quadratic programming problem (QPP) as in EFSVM. The least square versions of the entropy based SVM are faster than EFSVM and give higher generalization performance which shows its applicability and efficiency. Experiments are performed on various real world class imbalanced datasets and compared the results of proposed methods with new fuzzy twin support vector machine for pattern classification (NFTWSVM), entropy based fuzzy support vector machine (EFSVM), fuzzy twin support vector machine (FTWSVM) and twin support vector machine (TWSVM) which clearly illustrate the superiority of the proposed EFLSTWSVM-CIL.
Read moreFSVM-CIL: Fuzzy Support Vector Machines for Class Imbalance Learning
Support vector machines (SVMs) is a popular machine learning technique, which works effectively with balanced datasets. However, when it comes to imbalanced datasets, SVMs produce suboptimal classification models. On the other hand, the SVM algorithm is sensitive to outliers and noise present in the datasets. Therefore, although the existing class imbalance learning (CIL) methods can make SVMs less sensitive to class imbalance, they can still suffer from the problem of outliers and noise. Fuzzy SVMs (FSVMs) is a variant of the SVM algorithm, which has been proposed to handle the problem of outliers and noise. In FSVMs, training examples are assigned different fuzzy-membership values based on their importance, and these membership values are incorporated into the SVM learning algorithm to make it less sensitive to outliers and noise. However, like the normal SVM algorithm, FSVMs can also suffer from the problem of class imbalance. In this paper, we present a method to improve FSVMs for CIL (called FSVM-CIL), which can be used to handle the class imbalance problem in the presence of outliers and noise. We thoroughly evaluated the proposed FSVM-CIL method on ten real-world imbalanced datasets and compared its performance with five existing CIL methods, which are available for normal SVM training. Based on the overall results, we can conclude that the proposed FSVM-CIL method is a very effective method for CIL, especially in the presence of outliers and noise in datasets.
Read moreDensity-weighted intuitionistic fuzzy least squares twin support vector machines for class imbalance learning
The existing models generated by twin support vector machines cannot effectively deal with datasets containing noises and outliers and often do not consider the inherent distribution of data when tackling the class imbalance data. To moderate the above problems, we propose a density-weighted intuitionistic fuzzy least squares twin support vector machines for class imbalance learning. Firstly, the density-weighted values of each data point based on data inherent distribution are calculated by the k-nearest neighbor method, the values of density-weighted are introduced into the intuitionistic fuzzy twin support vector machines (IFTSVM), and thus the density-weighted IFTSVM are proposed for class imbalance learning. Besides, to avoid the issue of overfitting, a regularization term is added to the objective function of the density-weighted IFTSVM. Finally, to improve the computational efficiency of the density-weighted IFTSVM, we use least squares instead of quadratic programming to solve it and the algorithm DW-IFLSTSVM-CIL is proposed. The experimental results and corresponding statistical analyses on some benchmark class imbalance datasets verify that the proposed algorithm can obtain better performance because the proposed method fully takes the inherent distribution of the training datasets into account.
Read moreA robust fuzzy least squares twin support vector machine for class imbalance learning
A robust fuzzy least squares twin support vector machine for class imbalance learning
Fuzzy twin SVM based on Hilbert–Schmidt independence criterion and information entropy for imbalanced learning
Purpose Imbalanced learning presents a significant challenge in the field of machine learning. Although traditional support vector machine (SVM) demonstrate relatively robust performance when handling imbalanced datasets, they assign equal learning contributions to all samples, which can lead to decision boundaries that are biased toward the majority class, especially in the presence of outliers or noise. To address this issue, this paper proposes a fuzzy SVM model based on the Hilbert–Schmidt independence criterion (HSIC) heuristic strategy and information entropy (HEFTSVM) for imbalanced learning. Design/methodology/approach This study introduces an effective fuzzy membership allocation strategy combining HSIC heuristic strategies and information entropy. The fuzzy membership function leverages structural information derived from both the input and feature spaces. Specifically, entropy assesses membership within the input space, whereas HSIC evaluates it in the feature space. The final fuzzy membership function is derived by multiplying the memberships from both spaces. This approach is integrated with the twin support vector machine (TSVM) algorithm to create the HEFTSVM algorithm. We evaluated the model’s effectiveness through comparative experiments on 39 datasets with varying imbalance levels. Findings Experimental results validate the effectiveness of HEFTSVM in addressing class imbalance classification problems, achieving an average geometric mean (GM) of 86.71% on low-imbalance datasets and 82.13% on high-imbalance datasets. These findings demonstrate that HEFTSVM exhibits better robustness and generalization performance than existing learning models. Originality/value This study proposes a fuzzy membership degree allocation strategy based on HSIC heuristic and information entropy, effectively addressing the class imbalance issue, mitigating the sensitivity of TSVM to noise and introducing the noise-robust HEFTSVM model.
Read moreLeast squares twin support vector machine with Universum data for classification
abstractUniversum, a third class not belonging to either class of the classification problem, allows to incorporate the prior knowledge into the learning process. A lot of previous work have demonstrated that the Universum is helpful to the supervised and semi-supervised classification. Moreover, Universum has already been introduced into the support vector machine (SVM) and twin support vector machine (TSVM) to enhance the generalisation performance. To further increase the generalisation performance, we propose a least squares TSVM with Universum data (-TSVM) in this paper. Our -TSVM possesses the following advantages: first, it exploits Universum data to improve generalisation performance. Besides, it implements the structural risk minimisation principle by adding a regularisation to the objective function. Finally, it costs less computing time by solving two small-sized systems of linear equations instead of a single larger-sized quadratic programming problem. To verify the validity of our proposed algorithm, we conduct various experiments around the size of labelled samples and the number of Universum data on data-sets including seven benchmark data-sets, Toy data, MNIST and Face images. Empirical experiments indicate that Universum contributes to making prediction accuracy improved even stable. Especially when fewer labelled samples given, -TSVM is far superior to the improved LS-TSVM (ILS-TSVM), and slightly superior to the -TSVM.
Read moreRobust twin bounded support vector machines for outliers and imbalanced data
Truncated loss functions are robust to class noise and outliers. A robust twin bounded support vector machine is proposed in this paper that truncates the growth of its loss functions at a pre-specified point, thus, flattens the function that pre-specified score afterwards. Moreover, to make the proposed method capable of handling datasets with different imbalance ratio, cost-sensitive learning is implemented by scaling the total error of the classes based on the number of samples of each class. However, the adopted loss functions take a non-convex structure which does not always assure global optimum. To handle this issue, we suggest concave-convex procedure (CCCP) to ensure global convergence by decomposing the cost functions into additions of one convex and one concave part. The behaviour of the proposed method for varied imbalance ratio is analysed experimentally and depicted graphically. Classification performance of the proposed method in terms of AUC, F-Measure and G-Mean is compared with other related methods, viz. Hinge loss support vector machine (SVM), Ramp loss SVM (RSVM), twin SVM (TWSVM), twin bounded SVM (TBSVM), Pinball loss SVM (pin-SVM), entropy-based fuzzy SVM (EFSVM), non-parallel hyperplane Universum SVM (U-NHSVM), stochastic gradient twin support vector machine (SGTSVM), k-nearest neighbor (KNN)-based maximum margin and minimum volume hyper-sphere machine (KNN-M3VHM) and affinity and class probability-based fuzzy SVM (ACFSVM) on several real-world datasets with imbalance ratio raging from low to high. Further, to establish the significance of the proposed method in pattern classification, pair-wise statistical comparison of the methods is performed based on their average ranks on AUC. Experimental results are convincing.
Read moreInverse free reduced universum twin support vector machine for imbalanced data classification
Inverse free reduced universum twin support vector machine for imbalanced data classification
Adaptively weighted learning for twin support vector machines via Bregman divergences
Some versions of weighted (twin) support vector machines have been developed to handle the contaminated data. However, the weights of samples are generally obtained from the prior knowledge of data in advance. This article develops an adaptively weighted twin support vector machine via Bregman divergences. To better handle the contaminated data, we employ an insensitive loss function to control the fitting error of the samples in one class and introduce the weight (fuzzy membership) of each sample into the proposed model. The alternating optimization technique is utilized to solve the proposed model due to the characteristics of the model. The accelerated version of first-order methods is used to solve a quadratic programming problem, and the fuzzy membership of each sample is achieved analytically in the case of Bregman divergences. Experiments on some data sets have been conducted to show that our method gains better classification performance than previous methods, especially for the open set experiment.
Read moreLarge-Scale Fuzzy Least Squares Twin SVMs for Class Imbalance Learning
Twin support vector machines (TSVMs) have been successfully employed for binary classification problems. With the advent of machine learning algorithms, data have proliferated and there is a need to handle or process large-scale data. TSVMs are not successful in handling large-scale data due to the following: 1) the optimization problem solved in the TSVM needs to calculate large matrix inverses, which makes it an ineffective choice for large-scale problems; 2) the empirical risk minimization principle is employed in the TSVM and, hence, may suffer due to overfitting; and 3) the Wolfe dual of TSVM formulation involves positive-semidefinite matrices, and hence, singularity issues need to be resolved manually. Keeping in view the aforementioned shortcomings, in this article, we propose a novel large-scale fuzzy least squares TSVM for class imbalance learning (LS-FLSTSVM-CIL). We formulate the LS-FLSTSVM-CIL such that the proposed optimization problem ensures that: 1) no matrix inversion is involved in the proposed LS-FLSTSVM-CIL formulation, which makes it an efficient choice for large-scale problems; 2) the structural risk minimization principle is implemented, which avoids the issues of overfitting and results in better performance; and 3) the Wolfe dual formulation of the proposed LS-FLSTSVM-CIL model involves positive-definite matrices. In addition, to resolve the issues of class imbalance, we assign fuzzy weights in the proposed LS-FLSTSVM-CIL to avoid bias in dominating the samples of class imbalance problems. To make it more feasible for large-scale problems, we use an iterative procedure known as the sequential minimization principle to solve the objective function of the proposed LS-FLSTSVM-CIL model. From the experimental results, one can see that the proposed LS-FLSTSVM-CIL demonstrates superior performance in comparison to baseline classifiers. To demonstrate the feasibility of the proposed LS-FLSTSVM-CIL on large-scale classification problems, we evaluate the classification models on the large-scale normally distributed clustered (NDC) dataset. To demonstrate the practical applications of the proposed LS-FLSTSVM-CIL model, we evaluate it for the diagnosis of Alzheimer’s disease and breast cancer disease. Evaluation on NDC datasets shows that the proposed LS-FLSTSVM-CIL has feasibility in large-scale problems as it is fast in comparison to the baseline classifiers.
Read morePAC-Bayes bounds for twin support vector machines
PAC-Bayes bounds for twin support vector machines
Wavelet twin support vector machines based on glowworm swarm optimization
Wavelet twin support vector machines based on glowworm swarm optimization