Support vector machine ensemble based on independent component analysis and fuzzy kernel clustering

Flood susceptibility mapping using a novel ensemble weights-of-evidence and support vector machine models in GIS

Support Vector Machines (SVMs) ensembles have been widely used to improve classification accuracy in complicated pattern recognition tasks. In this work we propose to apply an ensemble of SVMs coupled with feature-subset selection methods to aleviate the curse of dimensionality associated with expression-based classification of DNA microarray data. We compare the single SVM classifier to SVM ensembles applying two different feature-subset selection techniques, namely random selection and k-means clustering, the base classifiers are combined using either majority vote or SVM fusion. Two real-world benchmarks datasets are used to evaluate and compare the performance. Experimental results show that the SVM ensemble of SVM base classifiers using k-means clustering for feature-subset selection and employing an SVM combiner achieve the best classification accuracy, and that feature-subset-selection methods can have considerable impact on the classification accuracy.

In this paper, the algorithm design of the support vector machines (SVMs) ensemble in a practical multi-stage framework is analyzed which can be implemented efficiently by evolutionary multi-objective optimization algorithm. The designing of SVMs ensemble is considered in three stages: first, the bootstrap method and a strategy of dynamical parameter range adjustment are used to generate more diverse base SVMs, and the NSGA-II algorithm which can efficiently tune the parameters of SVMs is applied to ensure the accuracy of base SVMs; Second, the NSGA-II algorithm is used again to select the member of ensemble based on the accuracy and diversity of the ensemble we have measured; Last, the reliability of different class is computed and combined to decide the outputs of ensemble in terms of the decision values of base SVMs. The proposed algorithm is applied to the UCI datasets, some useful results has been concluded for the future work in this field.

Flood is one of the most commonly occurred natural hazards worldwide. Severe flood occurrences in Kelantan, Malaysia cause damage to both life and property every year. Due to the huge losses in this area, development of appropriate flood modeling is required for the government. Remote sensing and geographic information system techniques can support overall flood management as they can produce rapid data collection and analysis for hydrological studies. The existing models for flood mapping have some weak points that may improve through more sophisticated and ensemble methods. The current research aimed to propose a novel ensemble method by integrating support vector machine (SVM) and frequency ratio (FR) to produce spatial modeling in flood susceptibility assessment. In the literature, mostly statistical and machine learning methods are used individually; however, their integration can enhance the final output. The FR model can perform bivariate statistical analysis and evaluate the correlation between the flooding and classes of each conditioning factors. The weights achieved by FR can be assigned to each conditioning factor and the resulted factors can be used in SVM analysis. In order to examine the efficiency of the proposed ensemble method and to show the proficiency of SVM, another machine learning algorithm such as decision tree (DT) was applied and the results were compared. To perform the methods, the upper catchment of the Kelantan basin in Malaysia was chosen. First, a flood inventory map with a total of 155 flood locations were extracted from various sources over the study area. The flood inventory map was randomly divided into two dataset; 70 % (115 flood locations) for the purpose of training and the remaining 30 % (40 flood locations) was used for validation. The spatial database included digital elevation model, curvature, geology, river, stream power index, rainfall, land use/cover, soil type, topographic wetness index and slope. For model validation, area under curve method was used and both success and prediction rate curves were calculated. The validation results for ensemble method showed 88.71 and 85.21 % for success rate and prediction rate respectively. The DT model showed 87.00 and 82.00 % for the success rate and prediction rate respectively. It is evident that the accuracies were increased using the ensemble method. The acquired results proved the efficiency of the proposed ensemble method as rapid, accurate and reasonable in flood susceptibility assessment.

In order to develop Human-centric Driver Fatigue Monitoring Systems (HDFMS) with aims to increase driving safety, an efficient combined features extraction from Curvelet transform and Gabor wavelet transform for fatigue expressions descriptions of vehicle drivers is proposed, and Random Subspace Ensemble (RSE) of Support Vector Machines (SVMs) with polynomial kernel as the base classifier is then exploited for classification of three predefined fatigue expressions classes, namely, awake expressions, moderate fatigue expressions, and severe fatigue expressions. The results of holdout and cross-validation experiments show that CF by RSE of SVMs with polynomial kernel outperforms other seven classifiers, i.e., Curvelet features by SVMs classifier, Gabor features by SVMs classifier, CF by five individual SVMs classifiers. With CF and RSE of SVMs with polynomial kernel, the classification accuracies of drivers' fatigue expressions are over 90% in both of the holdout and cross-validation experiments, which show the proposed approach of combined features extraction and RSE of SVMs can be used for developing Human-centric Driver Fatigue Monitoring Systems to increase driving safety.

Accurate fault diagnosis for large machines is very important for its economic meaning. In essence, fault diagnosis is a pattern classification and recognition problem of judging whether the operation status of the system is normal or not. Support vector machines (SVMs), well motivated theoretically, have been introduced as effective methods for solving classification problems. However, the generalization performance of SVMs is sometimes far from the expected level due to their limitations in practical applications. Ensemble learning of SVMs provides a promising way for such cases. In this paper, a new ensemble learning method based on genetic algorithm is presented. Genetic algorithm is introduced into ensemble learning so as to search for accurate and diverse classifiers to construct a good ensemble. The presented method works on a higher level and is more direct than other search based ensemble learning methods. Different strategies of constructing diverse base classifiers are also studied. Experimental results on a steam turbine fault diagnosis problem show that the presented method can achieve much better generalization performance than other methods including single SVM, Bagging and Adaboost.M1 if the strategies used are appropriate for the practical problem.

Bond rating is regarded as an important event for measuring financial risk of companies and for determining the investment returns of investors. As a result, it has been a popular research topic for researchers to predict companies' credit ratings by applying statistical and machine learning techniques. The statistical techniques, including multiple regression, multiple discriminant analysis (MDA), logistic models (LOGIT), and probit analysis, have been traditionally used in bond rating. However, one major drawback is that it should be based on strict assumptions. Such strict assumptions include linearity, normality, independence among predictor variables and pre-existing functional forms relating the criterion variablesand the predictor variables. Those strict assumptions of traditional statistics have limited their application to the real world. Machine learning techniques also used in bond rating prediction models include decision trees (DT), neural networks (NN), and Support Vector Machine (SVM). Especially, SVM is recognized as a new and promising classification and regression analysis method. SVM learns a separating hyperplane that can maximize the margin between two categories. SVM is simple enough to be analyzed mathematical, and leads to high performance in practical applications. SVM implements the structuralrisk minimization principle and searches to minimize an upper bound of the generalization error. In addition, the solution of SVM may be a global optimum and thus, overfitting is unlikely to occur with SVM. In addition, SVM does not require too many data sample for training since it builds prediction models by only using some representative sample near the boundaries called support vectors. A number of experimental researches have indicated that SVM has been successfully applied in a variety of pattern recognition fields. However, there are three major drawbacks that can be potential causes for degrading SVM's performance. First, SVM is originally proposed for solving binary-class classification problems. Methods for combining SVMs for multi-class classification such as One-Against-One, One-Against-All have been proposed, but they do not improve the performance in multi-class classification problem as much as SVM for binary-class classification. Second, approximation algorithms (e.g. decomposition methods, sequential minimal optimization algorithm) could be used for effective multi-class computation to reduce computation time, but it could deteriorate classification performance. Third, the difficulty in multi-class prediction problems is in data imbalance problem that can occur when the number of instances in one class greatly outnumbers the number of instances in the other class. Such data sets often cause a default classifier to be built due to skewed boundary and thus the reduction in the classification accuracy of such a classifier. SVM ensemble learning is one of machine learning methods to cope with the above drawbacks. Ensemble learning is a method for improving the performance of classification and prediction algorithms. AdaBoost is one of the widely used ensemble learning techniques. It constructs a composite classifier by sequentially training classifiers while increasing weight on the misclassified observations through iterations. The observations that are incorrectly predicted by previous classifiers are chosen more often than examples that are correctly predicted. Thus Boosting attempts to produce new classifiers that are better able to predict examples for which the current ensemble's performance is poor. In this way, it can reinforce the training of the misclassified observations of the minority class. This paper proposes a multiclass Geometric Mean-based Boosting (MGM-Boost) to resolve multiclass prediction problem. Since MGM-Boost introduces the notion of geometric mean into AdaBoost, it can perform learning process considering the geometric mean-based accuracy and errors of multiclass. This study applies MGM-Boost to the real-world bond rating case for Korean companies to examine the feasibility of MGM-Boost. 10-fold cross validations for threetimes with different random seeds are performed in order to ensure that the comparison among three different classifiers does not happen by chance. For each of 10-fold cross validation, the entire data set is first partitioned into tenequal-sized sets, and then each set is in turn used as the test set while the classifier trains on the other nine sets. That is, cross-validated folds have been tested independently of each algorithm. Through these steps, we have obtained the results for classifiers on each of the 30 experiments. In the comparison of arithmetic mean-based prediction accuracy between individual classifiers, MGM-Boost (52.95%) shows higher prediction accuracy than both AdaBoost (51.69%) and SVM (49.47%). MGM-Boost (28.12%) also shows the higher prediction accuracy than AdaBoost (24.65%) and SVM (15.42%)in terms of geometric mean-based prediction accuracy. T-test is used to examine whether the performance of each classifiers for 30 folds is significantly different. The results indicate that performance of MGM-Boost is significantly different from AdaBoost and SVM classifiers at 1% level. These results mean that MGM-Boost can provide robust and stable solutions to multi-classproblems such as bond rating.

기업의 부도를 예측하는 것은 회계나 재무 분야에서 중요한 연구주제이다. 지금까지 기업 부도예측을 위해 여러 가지 데이터마이닝 기법들이 적용되었으나 주로 단일 모형을 사용함으로서 복잡한 분류 문제에의 적용에 한계를 갖고 있었다. 본 논문에서는 최근에 각광받고 있는 SVM (support vector machine) 모형들을 결합한 앙상블 SVM 모형 (ensemble SVM model)을 부도예측에 사용하고자 한다. 제안된 앙상블 모형은 v-조각 교차 타당성 (v-fold cross-validation)에 의해 얻어진 여러 가지 모형 중에서 성능이 좋은 상위 k개의 단일 모형으로 구성하고 과반수 투표 방식 (majority voting)을 사용하여 미지의 클래스를 분류한다. 본 논문에서 제안된 앙상블 SVM 모형의 성능을 평가하기 위해 실제 기업의 재무비율 자료와 모의실험자료를 가지고 실험하였고, 실험결과 제안된 앙상블 모형이 여러 가지 평가척도 하에서 단일 SVM 모형들보다 좋은 성능을 보임을 알 수 있었다. Corporate bankruptcy prediction has been an important topic in the accounting and finance field for a long time. Several data mining techniques have been used for bankruptcy prediction. However, there are many limits for application to real classification problem with a single model. This study proposes ensemble SVM (support vector machine) model which assembles different SVM models with each different kernel functions. Our ensemble model is made and evaluated by v-fold cross-validation approach. The k top performing models are recruited into the ensemble. The classification is then carried out using the majority voting opinion of the ensemble. In this paper, we investigate the performance of ensemble SVM classifier in terms of accuracy, error rate, sensitivity, specificity, ROC curve, and AUC to compare with single SVM classifiers based on financial ratios dataset and simulation dataset. The results confirmed the advantages of our method: It is robust while providing good performance.

Online dynamic security assessment of wind integrated power system using SDAE with SVM ensemble boosting learner

Support Vector Machines (SVMs) are considered to be one of the most powerful classification tools, widely used in many applications. However, in numerous scenarios the classes are not equally represented and the predictive performance of SVMs on such data can drop dramatically. Different methods have been proposed to address moderate class imbalance issues, but there are few methods that can be successful at detecting the minority class while also keeping high accuracy, especially when applied to datasets with significant level of imbalance. In this paper, we consider SVM ensembles that are built by using a bootstrap-based undersampling technique. We target to reduce the bias induced by class imbalances via multiple undersampling procedures and then decrease the variance using SVM ensembles. For combining the SVMs, we propose a new technique that deals with class imbalance problems of varying levels. Experiments on several datasets demonstrate the performance of the proposed scheme compared to state-of-the-art balancing methods.

The Support Vector Machine (SVM) (Cortes and Vapnik, 1995; Vapnik, 1995; Burges, 1998) is intended to generate an optimal separating hyperplane by minimizing the generalization error without the assumption of class probabilities such as Bayesian classifier. The decision hyperplane of SVM is determined by the most informative data instances, called Support Vectors (SVs). In practice, these SVMs are a subset of the entire training data. By now, SVMs have been successfully applied in many applications, such as face detection, handwritten digit recognition, text classification, and data mining. Osuna et al. (1997) applied SVMs for face detection. Heisele et al. (2004) achieved high face detection rate by using 2nd degree SVM. They applied hierarchical classification and feature reduction methods to speed up face detection using SVMs. Feature extraction and reduction are two primary issues in feature selection that is essential in pattern classification. Whether it is for storage, searching, or classification, the way the data are represented can significantly influence performances. Feature extraction is a process of extracting more effective representation of objects from raw data to achieve high classification rates. For image data, many kinds of features have been used, such as raw pixel values, Principle Component Analysis (PCA), Independent Component Analysis (ICA), wavelet features, Gabor features, and gradient values. Feature reduction is a process of selecting a subset of features with preservation or improvement of classification rates. In general, it intends to speed up the classification process by keeping the most important class-relevant features.

Fine-grained classification is a recognition task where subtle differences distinguish between different classes. To tackle this classification problem, part-based classification methods are mostly used. Partbased methods learn an algorithm to detect parts of the observed object and extract local part features for the detected part regions. In this paper we show that not all extracted part features are always useful for the classification. Furthermore, given a part selection algorithm that actively selects parts for the classification we estimate the upper bound for the fine-grained recognition performance. This upper bound lies way above the current state-of-the-art recognition performances which shows the need for such an active part selection method. Though we do not present such an active part selection algorithm in this work, we propose a novel method that is required by active part selection and enables sequential part-based classification. This method uses a support vector machine (SVM) ensemble and allows to classify an image based on arbitrary number of part features. Additionally, the training time of our method does not increase with the amount of possible part features. This fact allows to extend the SVM ensemble with an active part selection component that operates on a large amount of part feature proposals without suffering from increasing training time.

In this paper we propose a system for visual object detection and tracking based on the extended structural tensor and the ensemble of one-class support vector machines. First, the input color image is transformed with the anisotropic process into the extended structural tensor. Then the tensor space is clustered into the number of partitions which are used to train a corresponding number of one-class support vector machines composing an ensemble of classifiers. In run-time the ensemble classifies the input video stream into an object and background. Thanks to high discriminative properties of the extended structural tensor and to the diversity of the ensemble of classifiers the method shows very good properties which were shown by experiments on real video sequences.

The nodules and the multiple times larger non-nodules of the regions of interested(ROIs) in lung areas are achieved, that would lead to a serious imbalance on the sample data. Many scholars have proposed some algorithms to solve this problem. In this paper, in order to guarantee that there is no correlation among the extracted characteristics, the PCA method is adopted to optimize and reduce dimensions, and then the modified support vector machine(SVM) classifiers using the sequential minimal optimization(SMO) algorithm and the grid research method are proposed to improve the computing efficiency. Furthermore, the abundant lung CT images from the hospital partnership could confirm the experimental results. We compare the classification performance between the ensemble SVM(ESVM) classifier and the cost-sensitive SVM(CSVM) classifier to deal with this problem. Experimental results show the performance of the CSVM classifier based on grid search is satisfactory than the ESVM.

Modeling radiation-induced lung injury risk with an ensemble of support vector machines