Sample selection via clustering to construct support vector-like classifiers

Despite the superior classification performance of support vector machines (SVMs), training SVMs on large datasets is still a challenging problem. Sample reduction methods have been proposed and shown to reduce the training complexity significantly, but more or less trade off the generalization performance. This paper presents an efficient sample reduction method for multi-class classification using one-vs-rest SVMs, called Multi-class Sample Selection (MUSS). For each binary one-vs-rest classification problem, positive samples and negative samples are selected based on the distances from the cluster centers of positive class, assuming that positive samples with large distances from the positive centers and negative samples with small distances from the positive centers are near the classification boundary. The intention of clustering is to improve the computation efficiency of sample selection, other than to select from cluster centers as previous methods did. Experiments on a wide variety of datasets demonstrate the superiority of the proposed MUSS over other competitive algorithms in respect of the tradeoff between reduced sample size and classification performance. The experimental results show that MUSS also works well for binary classification problems.

Bond rating is regarded as an important event for measuring financial risk of companies and for determining the investment returns of investors. As a result, it has been a popular research topic for researchers to predict companies' credit ratings by applying statistical and machine learning techniques. The statistical techniques, including multiple regression, multiple discriminant analysis (MDA), logistic models (LOGIT), and probit analysis, have been traditionally used in bond rating. However, one major drawback is that it should be based on strict assumptions. Such strict assumptions include linearity, normality, independence among predictor variables and pre-existing functional forms relating the criterion variablesand the predictor variables. Those strict assumptions of traditional statistics have limited their application to the real world. Machine learning techniques also used in bond rating prediction models include decision trees (DT), neural networks (NN), and Support Vector Machine (SVM). Especially, SVM is recognized as a new and promising classification and regression analysis method. SVM learns a separating hyperplane that can maximize the margin between two categories. SVM is simple enough to be analyzed mathematical, and leads to high performance in practical applications. SVM implements the structuralrisk minimization principle and searches to minimize an upper bound of the generalization error. In addition, the solution of SVM may be a global optimum and thus, overfitting is unlikely to occur with SVM. In addition, SVM does not require too many data sample for training since it builds prediction models by only using some representative sample near the boundaries called support vectors. A number of experimental researches have indicated that SVM has been successfully applied in a variety of pattern recognition fields. However, there are three major drawbacks that can be potential causes for degrading SVM's performance. First, SVM is originally proposed for solving binary-class classification problems. Methods for combining SVMs for multi-class classification such as One-Against-One, One-Against-All have been proposed, but they do not improve the performance in multi-class classification problem as much as SVM for binary-class classification. Second, approximation algorithms (e.g. decomposition methods, sequential minimal optimization algorithm) could be used for effective multi-class computation to reduce computation time, but it could deteriorate classification performance. Third, the difficulty in multi-class prediction problems is in data imbalance problem that can occur when the number of instances in one class greatly outnumbers the number of instances in the other class. Such data sets often cause a default classifier to be built due to skewed boundary and thus the reduction in the classification accuracy of such a classifier. SVM ensemble learning is one of machine learning methods to cope with the above drawbacks. Ensemble learning is a method for improving the performance of classification and prediction algorithms. AdaBoost is one of the widely used ensemble learning techniques. It constructs a composite classifier by sequentially training classifiers while increasing weight on the misclassified observations through iterations. The observations that are incorrectly predicted by previous classifiers are chosen more often than examples that are correctly predicted. Thus Boosting attempts to produce new classifiers that are better able to predict examples for which the current ensemble's performance is poor. In this way, it can reinforce the training of the misclassified observations of the minority class. This paper proposes a multiclass Geometric Mean-based Boosting (MGM-Boost) to resolve multiclass prediction problem. Since MGM-Boost introduces the notion of geometric mean into AdaBoost, it can perform learning process considering the geometric mean-based accuracy and errors of multiclass. This study applies MGM-Boost to the real-world bond rating case for Korean companies to examine the feasibility of MGM-Boost. 10-fold cross validations for threetimes with different random seeds are performed in order to ensure that the comparison among three different classifiers does not happen by chance. For each of 10-fold cross validation, the entire data set is first partitioned into tenequal-sized sets, and then each set is in turn used as the test set while the classifier trains on the other nine sets. That is, cross-validated folds have been tested independently of each algorithm. Through these steps, we have obtained the results for classifiers on each of the 30 experiments. In the comparison of arithmetic mean-based prediction accuracy between individual classifiers, MGM-Boost (52.95%) shows higher prediction accuracy than both AdaBoost (51.69%) and SVM (49.47%). MGM-Boost (28.12%) also shows the higher prediction accuracy than AdaBoost (24.65%) and SVM (15.42%)in terms of geometric mean-based prediction accuracy. T-test is used to examine whether the performance of each classifiers for 30 folds is significantly different. The results indicate that performance of MGM-Boost is significantly different from AdaBoost and SVM classifiers at 1% level. These results mean that MGM-Boost can provide robust and stable solutions to multi-classproblems such as bond rating.

For the classical k-means clustering algorithm, the problem of selecting an appropriate k is a hard problem and affects the performance of k-means strongly. When used for clustering analysis, the conventional competitive learning (CL) algorithms also have a similar crucial problem-the selection of an appropriate number of neural units. The performance of frequency sensitive competitive learning (FSCL)-one version of the improved CL algorithms, also significantly deteriorates when the number of units is inappropriately selected. The paper proposes a new algorithm called rival penalized competitive learning (RPCL), which has the ability of automatically allocating an appropriate number of units for an input data set. The experimental results have shown that RPCL outperforms FSCL significantly when they are used for unsupervised classification, and supervised classification through the radial basis function net. >

Support vector machines (SVMs) are powerful, state-of-the-art machine learning tools. With the aim to integrate SVM training capability into embedded systems while being able to meet area and performance constraints, a parallel and scalable digital architecture for training SVMs on-line is proposed and implemented on a field-programmable gate array (FPGA). Experiments show that the proposed SVM processor can solve the channel equalization problem effectively with fixed-point arithmetic and exhibits good scalability with respect to the number of cache update units (CUUs). This architecture is particularly suitable to be applied in embedded environments, as designers can easily trade off between area and performance to meet the constraints.

High spatial resolution (1–5 m) remotely sensed datasets are increasingly being used to map land covers over large geographic areas using supervised machine learning algorithms. Although many studies have compared machine learning classification methods, sample selection methods for acquiring training and validation data for machine learning, and cross-validation techniques for tuning classifier parameters are rarely investigated, particularly on large, high spatial resolution datasets. This work, therefore, examines four sample selection methods—simple random, proportional stratified random, disproportional stratified random, and deliberative sampling—as well as three cross-validation tuning approaches—k-fold, leave-one-out, and Monte Carlo methods. In addition, the effect on the accuracy of localizing sample selections to a small geographic subset of the entire area, an approach that is sometimes used to reduce costs associated with training data collection, is investigated. These methods are investigated in the context of support vector machines (SVM) classification and geographic object-based image analysis (GEOBIA), using high spatial resolution National Agricultural Imagery Program (NAIP) orthoimagery and LIDAR-derived rasters, covering a 2,609 km2 regional-scale area in northeastern West Virginia, USA. Stratified-statistical-based sampling methods were found to generate the highest classification accuracy. Using a small number of training samples collected from only a subset of the study area provided a similar level of overall accuracy to a sample of equivalent size collected in a dispersed manner across the entire regional-scale dataset. There were minimal differences in accuracy for the different cross-validation tuning methods. The processing time for Monte Carlo and leave-one-out cross-validation were high, especially with large training sets. For this reason, k-fold cross-validation appears to be a good choice. Classifications trained with samples collected deliberately (i.e., not randomly) were less accurate than classifiers trained from statistical-based samples. This may be due to the high positive spatial autocorrelation in the deliberative training set. Thus, if possible, samples for training should be selected randomly; deliberative samples should be avoided.

For last two decades, various techniques have been advanced for classification of satellite imageries. A vast majority of them are supervised/semisupervised requiring manual selection of samples for each class. Some of the unsupervised approaches are based on hard thresholds and particular to image acquisition modules. In this study, we propose a spectral-slope-based classification technique and subsequently summarize the changes in temporal image sets. Using the properties of spectral slopes, we propose a set of rules for selection of training samples from Landsat imageries for classifying the land cover. The images are initially classified into three classes: water, vegetation, and vegetation-void. Further, vegetation and vegetation-void regions are classified into proper vegetation and dry cropland, and urban land and bare land, respectively. The initial classification is performed by support vector machines, and the second-level classification is performed through $k$ -means clustering and subsequent labeling of clusters to subclasses. Considering temporal images of the same scene, postclassification change summarization is carried out to quantize the land variations, both qualitatively (type of change) and quantitatively (volume of change).The approach has been used in the analysis of images acquired by different sensors operating under similar wavelength ranges.

We perceive the digital watermark detection as classification problem in image processing. We classify watermarked images as positive class whilst unwatermarked images as negative class. Support Vector Machine (SVM) is used as classifier of the watermarked and unwatermarked digital images. Two watermarking schemes i.e. Cox's spread spectrum (SS) and Single Value Decomposition (SVD) are used to embed watermark into digital images. These algorithms are selected based on their different level of robustness to Stirmark attacks. The payload of the watermark used for both algorithms is consistent at certain number of bits. SVM is trained with both the watermarked and unwatermarked images. Receiver Operating Characteristics (ROC) graphs are plotted to assess the statistical detection behavior of both the correlation detector and SVM classifier. We found that straight forward application of SVM leads to generalization problem. We suggest remedies to preprocess the training data in order to achieve substantially better performance from SVM classifier than those resulting from the straightforward application of SVM. Both watermarked and unwatermarked images are attacked under Stirmark and are then tested with the correlation detectors and SVM classifier. A comparison of the ROC of the correlation detectors and SVM classifier is performed to assess the accuracy of SVM classifier relative to correlation detectors. We found that SVM classifier has higher robustness to Stirmark attacks.

With reference to the literature worldwide, it is obvious that Support Vector Machine (SVM), a machine learning algorithm has proven records for excellent results regarding Classification of Image. But, Remote Sensing Images are considered as most complex in nature as far as classification is concern. Supervised classification of Remote Sensing Images needs more precise machine learning models, which will be fast and efficient. SVM do satisfy researchers all over the world as far as Remote Sensing Images are concern. Basically, SVM is non-parametric statistical learning based model, which acts like binary classifier. SVM represents a group of superior machine learning algorithms, where it decomposes the parameter of the problem into a quadratic optimization technique. Hence, SVM is used to locate optimum boundaries between classes, which in return generalize to unseen samples with least error among all possible boundaries separating two classes. SVM uses density estimation function for developing easy and efficient learning parameters. Like other supervised algorithms, SVM also undergo into Training, Learning and Testing Phase for classifying any image. Besides all parameters, training sample selection and optimization is crucial part that affects the classification accuracy of remote sensing images. We need to address this issue in our project so as to devise noble algorithm or approach, which could make SVM, a more robust statistical learning model.

Immediately after an earthquake, rapid disaster management is the main challenge for relevant organizations. While satellite images have been used in the past two decades for building-damage mapping, they have rarely been utilized for the timely damage monitoring required for rescue operations. Unmanned aerial vehicles (UAVs) have recently become very popular due to their agile deployment to sites, super-high spatial resolution, and relatively low operating cost. This paper proposes a novel deep-learning-based method for rapid post-earthquake building damage detection. The method detects damages in four levels and consists of three steps. First, three different feature types—non-deep, deep, and their fusion—are investigated to determine the optimal feature extraction method. A “one-epoch convolutional autoencoder (OECAE)” is used to extract deep features from non-deep features. Then, a rule-based procedure is designed for the automatic selection of the proper training samples required by the classification algorithms in the next step. Finally, seven famous machine learning (ML) algorithms—including support vector machine (SVM), random forest (RF), gradient boosting (GB), extreme gradient boosting (XGB), decision trees (DT), k-nearest neighbors (KNN), and adaBoost (AB)—and a basic deep learning algorithm (i.e., multi-layer perceptron (MLP)) are implemented to obtain building damage maps. The results indicated that auto-training samples are feasible and superior to manual ones, with improved overall accuracy (OA) and kappa coefficient (KC) over 22% and 33%, respectively; SVM (OA = 82% and KC = 74.01%) was the most accurate AI model with a slight advantage over MLP (OA = 82% and KC = 73.98%). Additionally, it was found that the fusion of deep and non-deep features using OECAE could significantly enhance damage-mapping efficiency compared to those using either non-deep features (by an average improvement of 6.75% and 9.78% in OA and KC, respectively) or deep features (improving OA by 7.19% and KC by 10.18% on average) alone.

This paper proposed a novel gender classification system based on selected texture-based features and Support Vector Machine (SVM) classifier. In this study, t-test is applied as a feature selection technique to select significant features. Firstly, we extract texture-based features comprising Local Binary Patterns (LBP) and Histogram of Oriented Gradient (HOG) of face images from FERET face database. Then t-test is employed to determine each feature if it has significant difference between male and female categories. Next, the SVM model is trained with the significant features, which are selected by p-value selection of training samples. Finally, the accuracy of the trained gender classifier is estimated by using testing samples. The experimental results show that with the proposed t-test-based gender classification the number of features is decreased dramatically from 5195 to 1563, a 70% reduction, and the accuracy also shows slight improvement which is from 91.5% to 92.2%.

Abstract. In this paper, we present a method for automatic refinement of training data. Many classifiers from machine learning used in applications in the remote sensing domain, rely on previously labelled training data. This labelling is often done by human operators and is bound to time constraints. Hence, selection of training data must be kept practical which implies a certain inaccuracy. This results in erroneously tagged regions enclosed within competing classes. For that purpose, we propose a method that removes outliers from training data by using an iterative training-classification scheme. Outliers are detected by their newly determined class membership as well as through analysis of uncertainty of classified samples. The sample selection method which incorporates quality of neighbouring samples is presented and compared to alternative strategies. Additionally, iterative approaches tend to propagate errors which might lead to degenerating classes. Therefore, a robust stopping criterion based on training data characteristics is described. Our experiments using a support vector machine (SVM) show, that outliers are reliably removed, allowing a more convenient sample selection. The classification result for unknown scenes of the accordant validation set improves from 70.36% to 79.12% on average. Additionally, the average complexity of the SVM model is decreased by 82.75% resulting in similar reduction of processing time.

A machine learning framework to predict kidney graft failure with class imbalance using Red Deer algorithm

This study aims to explore the effects of different non-landslide sampling strategies on machine learning models in landslide susceptibility mapping. Non-landslide samples are inherently uncertain, and the selection of non-landslide samples may suffer from issues such as noisy or insufficient regional representations, which can affect the accuracy of the results. In this study, a positive-unlabeled (PU) bagging semi-supervised learning method was introduced for non-landslide sample selection. In addition, buffer control sampling (BCS) and K-means (KM) clustering were applied for comparative analysis. Based on landslide data from Qiaojia County, Yunnan Province, China, collected in 2014, three machine learning models, namely, random forest, support vector machine, and CatBoost, were used for landslide susceptibility mapping. The results show that the quality of samples selected using different non-landslide sampling strategies varies significantly. Overall, the quality of non-landslide samples selected using the PU bagging method is superior, and this method performs best when combined with CatBoost for predicting (AUC = 0.897) landslides in very high and high susceptibility zones (82.14%). Additionally, the KM results indicated overfitting, displaying high accuracy for validation but poor statistical outcomes for zoning. The BCS results were the worst.

The learning-based methods have been widely applied to design a fault diagnosis model for rolling element bearing. However, the mainstream methods can only deal with the large training dataset, which is always violated in practical application. In this paper, we propose a physical knowledge-based hierarchical extreme learning machine(H-ELM) approach to adapt the problem of fault diagnosis for bearing with the small and imbalanced dataset. First, the proposed method uses the simple feature extraction algorithm to build a knowledge base for sample selection from the historical database, and the given training dataset is augmented with knowledge base. Second, a modified H-ELM algorithm is developed to identify fault location and recognize fault severity ranking based on the augmented dataset. Third, we design a self-optimizing module to optimize the sample selection and improve the performance of the H-ELM network. To evaluate the effectiveness of the proposed approach, the H-ELM without knowledge base and data augmentation-based support vector machine(SVM), back propagation neuron networks(BPNN) and deep belief networks(DBN) are tested in the numerical experiments to present a comprehensive comparison. The experimental results demonstrate that our approach outperforms in accuracy than other counterparts when dealing with the small and imbalanced datasets.

In this chapter, one gives an introduction to different kinds of particle swarm optimization (PSO) algorithms. One also introduces artificial neural networks (ANNs), support vector machines (SVMs) and evolutionary computing to show how PSO may be used to determine optimal parameters using an ANN or SVM regime, for classification of DNA strings. In addition, PSO is used in the design of an SVM-based clustering algorithm. Ant colony optimization (ACO) algorithms are also introduced in the chapter. Using ACO algorithms has been of great success to solve many discrete optimization and non-deterministic polynomial (NP)-hard problems, for instance the travelling salesman problem. The behaviour of ants is also used, for instance, to design an algorithm for data clustering. We want to develop later a similar application based on this clustering algorithm and compare it with the SVM one using PSO.