Fuzzy rules extraction from support vector machines for multi-class classification

This paper proposes a fuzzy rule extraction method from support vector machines. Support vector machines (SVM) are learning systems based on statistical learning theory that have been successfully applied to a wide variety of application. However, SVM are "black box" models, that is, they generate a solution with linear combination of kernel functions which has a quite difficult interpretation. Methods for rule extraction from trained SVM have already been proposed, however, the rules generated by these methods have, in their antecedents, intervals or functions. This format decreases the interpretability of the generated rules and jeopardizes the knowledge extraction capability. Hence, to increase the linguistic interpretability of the generated rules, we propose in this paper a methodology for extracting fuzzy rules from a trained SVM, where the rule's antecedents are associated with fuzzy sets.

Rule extraction from data is one of the key technologies for solving the bottlenecks in artificial intelligence. Artificial neural networks are well suited for representing any knowledge in given data. Extraction of logical/fuzzy rules from the trained artificial neural network is of great importance to researchers in the fields of artificial intelligence and soft computing. Fuzzy rule sets are capable of approximating any nonlinear mapping relationships. Extraction of rules from data has been discussed in terms of fuzzy modeling, fuzzy clustering, and classification with fuzzy rule sets. This special issue entitled"Rule Extraction from Data" is aimed at providing the readers with good insights into the advanced studies in the field of rule extraction from data using neural networks/fuzzy rule sets. I invited seven research papers best suited for the theme of this special issue. All the papers were reviewed rigorously by two reviewers each. The first paper proposes an interesting rule extraction method from data using neural networks. Ishikawa presents a combination of learning with an immediate critic and a structural learning with forgetting. This method is capable of generating skeletal networks for logical rule extraction from data with correct and wrong answers. The proposed method is applied to rule extraction from lense data. The second paper presents a new methodology for logical rule extraction based on transformation of MLP (multilayered perceptron) to a logical network. Duck et al. applied their C-MLP2LN to the Iris benchmark classification problem as well as real-world medical data with very good results. In the third paper, Geczy and Usui propose fuzzy rule extraction from trained artificial neural networks. The proposed algorithm is implied from their theoretical study, not from heuristics. Their study enables to initially consider derivation of crisp rules from trained artificial neural network, and in case of conflict, application of fuzzy rules. The proposed algorithm is experimentally demonstrated with the Iris benchmark classification problem. The fourth paper presents a new framework for fuzzy modeling using genetic algorithm. The authors have broken new ground of fuzzy rule extraction from neural networks. For the fuzzy modeling, they have proposed a particular type of neural networks containing nodes representing membership functions. In this fourth paper, the authors discuss input variable selection for the fuzzy modeling under multiple criteria with different importance. A target system with a strong nonlinearity is used for demonstrating the proposed method. Kasabov, et al. present, in the fifth paper, a method for extraction of fuzzy rules that have different level of abstraction depending on several modifiable thresholds. Explanation quality becomes better with higher threshold values. They apply the proposed method to the Iris benchmark classification problem and to a real world problem. J. Yen and W. Gillespie address interpretability issue of Takagi-Sugeno-Kang model, one of the most popular fuzzy mdoels, in the fifth paper. They propose a new approach of fuzzy modeling that ensures not only a high approximation of the input-output relationship in the data, but also good insights about the local behavior of the model. The proposed method is applied to fuzzy modeling of sinc function and Mackey-Glass chaotic time series data. The last paper discusses fuzzy rule extraction from numerical data for high-dimensional classification problems. H.Ishibuchi, et al. have been pioneering methods for classification of data using fuzzy rules and genetic algorithm. In this last paper, they introduced a new criterion, simplicity of each rule, together with the conventional ones, compactness of rule base and classification ability, for high-dimensional problem. The Iris data is used for demonstrating their new classification method. They applied it also to wine data and credit data. I hope that the readers will be encouraged to explore the frontier to establish a new paradigm in the field of knowledge representation and rule extraction.

The responsibility weighted Mahalanobis kernel for semi-supervised training of support vector machines for classification

Bond rating is regarded as an important event for measuring financial risk of companies and for determining the investment returns of investors. As a result, it has been a popular research topic for researchers to predict companies' credit ratings by applying statistical and machine learning techniques. The statistical techniques, including multiple regression, multiple discriminant analysis (MDA), logistic models (LOGIT), and probit analysis, have been traditionally used in bond rating. However, one major drawback is that it should be based on strict assumptions. Such strict assumptions include linearity, normality, independence among predictor variables and pre-existing functional forms relating the criterion variablesand the predictor variables. Those strict assumptions of traditional statistics have limited their application to the real world. Machine learning techniques also used in bond rating prediction models include decision trees (DT), neural networks (NN), and Support Vector Machine (SVM). Especially, SVM is recognized as a new and promising classification and regression analysis method. SVM learns a separating hyperplane that can maximize the margin between two categories. SVM is simple enough to be analyzed mathematical, and leads to high performance in practical applications. SVM implements the structuralrisk minimization principle and searches to minimize an upper bound of the generalization error. In addition, the solution of SVM may be a global optimum and thus, overfitting is unlikely to occur with SVM. In addition, SVM does not require too many data sample for training since it builds prediction models by only using some representative sample near the boundaries called support vectors. A number of experimental researches have indicated that SVM has been successfully applied in a variety of pattern recognition fields. However, there are three major drawbacks that can be potential causes for degrading SVM's performance. First, SVM is originally proposed for solving binary-class classification problems. Methods for combining SVMs for multi-class classification such as One-Against-One, One-Against-All have been proposed, but they do not improve the performance in multi-class classification problem as much as SVM for binary-class classification. Second, approximation algorithms (e.g. decomposition methods, sequential minimal optimization algorithm) could be used for effective multi-class computation to reduce computation time, but it could deteriorate classification performance. Third, the difficulty in multi-class prediction problems is in data imbalance problem that can occur when the number of instances in one class greatly outnumbers the number of instances in the other class. Such data sets often cause a default classifier to be built due to skewed boundary and thus the reduction in the classification accuracy of such a classifier. SVM ensemble learning is one of machine learning methods to cope with the above drawbacks. Ensemble learning is a method for improving the performance of classification and prediction algorithms. AdaBoost is one of the widely used ensemble learning techniques. It constructs a composite classifier by sequentially training classifiers while increasing weight on the misclassified observations through iterations. The observations that are incorrectly predicted by previous classifiers are chosen more often than examples that are correctly predicted. Thus Boosting attempts to produce new classifiers that are better able to predict examples for which the current ensemble's performance is poor. In this way, it can reinforce the training of the misclassified observations of the minority class. This paper proposes a multiclass Geometric Mean-based Boosting (MGM-Boost) to resolve multiclass prediction problem. Since MGM-Boost introduces the notion of geometric mean into AdaBoost, it can perform learning process considering the geometric mean-based accuracy and errors of multiclass. This study applies MGM-Boost to the real-world bond rating case for Korean companies to examine the feasibility of MGM-Boost. 10-fold cross validations for threetimes with different random seeds are performed in order to ensure that the comparison among three different classifiers does not happen by chance. For each of 10-fold cross validation, the entire data set is first partitioned into tenequal-sized sets, and then each set is in turn used as the test set while the classifier trains on the other nine sets. That is, cross-validated folds have been tested independently of each algorithm. Through these steps, we have obtained the results for classifiers on each of the 30 experiments. In the comparison of arithmetic mean-based prediction accuracy between individual classifiers, MGM-Boost (52.95%) shows higher prediction accuracy than both AdaBoost (51.69%) and SVM (49.47%). MGM-Boost (28.12%) also shows the higher prediction accuracy than AdaBoost (24.65%) and SVM (15.42%)in terms of geometric mean-based prediction accuracy. T-test is used to examine whether the performance of each classifiers for 30 folds is significantly different. The results indicate that performance of MGM-Boost is significantly different from AdaBoost and SVM classifiers at 1% level. These results mean that MGM-Boost can provide robust and stable solutions to multi-classproblems such as bond rating.

Support vector machine (SVM) and artificial neural network (ANN) systems were applied to a drug/nondrug classification problem as an example of binary decision problems in early-phase virtual compound filtering and screening. The results indicate that solutions obtained by SVM training seem to be more robust with a smaller standard error compared to ANN training. Generally, the SVM classifier yielded slightly higher prediction accuracy than ANN, irrespective of the type of descriptors used for molecule encoding, the size of the training data sets, and the algorithm employed for neural network training. The performance was compared using various different descriptor sets and descriptor combinations based on the 120 standard Ghose-Crippen fragment descriptors, a wide range of 180 different properties and physicochemical descriptors from the Molecular Operating Environment (MOE) package, and 225 topological pharmacophore (CATS) descriptors. For the complete set of 525 descriptors cross-validated classification by SVM yielded 82% correct predictions (Matthews cc = 0.63), whereas ANN reached 80% correct predictions (Matthews cc = 0.58). Although SVM outperformed the ANN classifiers with regard to overall prediction accuracy, both methods were shown to complement each other, as the sets of true positives, false positives (overprediction), true negatives, and false negatives (underprediction) produced by the two classifiers were not identical. The theory of SVM and ANN training is briefly reviewed.

Support vector machines (SVMs) can solve structured multioutput learning problems such as multilabel classification, multiclass classification, and vector regression. SVM training is expensive, especially for large and high-dimensional data sets. The bottleneck of the SVM training often lies in the kernel value computation. In many real-world problems, the same kernel values are used in many iterations during the training, which makes the caching of kernel values potentially useful. The majority of the existing studies simply adopt the least recently used (LRU) replacement strategy for caching kernel values. However, as we analyze in this article, the LRU strategy generally achieves high hit ratio near the final stage of the training but does not work well in the whole training process. Therefore, we propose a new caching strategy called EFU (less frequently used), which replaces the EFU kernel values that enhance least frequently used (LFU). Our experimental results show that EFU often has 20% higher hit ratio than LRU in the training with the Gaussian kernel. To further optimize the strategy, we propose a caching strategy called hybrid caching for the SVM training (HCST), which has a novel mechanism to automatically adapt the better caching strategy in different stages of the training. We have integrated the caching strategy into ThunderSVM, a recent SVM library on many-core processors. Our experiments show that HCST adaptively achieves high hit ratios with little runtime overhead among different problems including multilabel classification, multiclass classification, and regression problems. Compared with other existing caching strategies, HCST achieves 20% more reduction in training time on average.

Support vector machine is a new and promising technique for pattern classification and regression estimation. The training of support vector machine is characterized by convex optimization problem, up to the determination of a few additional tuning parameters. Moreover, the model complexity follows from this convex optimization problem. In this paper we introduce sequential support vector machine for regression estimation. The support vector machine is trained by the particle filter. By the method of training support vector machine we can design the controller based on support vector machine. Afterwards we discuss feedforward control based on support vector machine. Support vector machine builds the inverse model of plant in the feedforward control based on support vector machine. In order to demonstrate the availability of this new controller designing method, we give a simulation of the simple nonlinear system for output tracking problem. The results of simulation demonstrate this method of new controller designing is very effective.

I335 the training of Support Vector Machine (SVM) classification model, some problems such as over-fitting, under-fitting, slow training speed and prediction affected by penalty and kernel function parameters are exposed. A fault classification method of rolling bearing based on Complete Ensemble Empirical Mode Decomposition with Adaptive Noise (CEEMDAN) and Grey Wolf Optimization (GWO) -SVM is proposed. Firstly, the acquire signals are adaptively decomposed into a plurality of Intrinsic Mode Function(IMF) component by CEEMDAN, and the energy entropy of each IMF component is extracted to form a set of high dimensional feature vector. Then, the processed feature vectors are introduced into the diagnosis network of GWO-SVM algorithm to build a fault classification model of motor rolling bearing. The classification results show that the CEEMDAN and GWO-SVM fault classification network of motor rolling bearing has higher accuracy and shorter diagnosis time than Particle Swarm Optimization (PSO) -SVM and Cross Validation (CV) -SVM.

When applying traditional methods to train approximately linear support vector machine (SVM), we will get a kernel matrix which occupy mass computer memory and lead a slow convergence speed. In order to improve the convergence speed of SVM, a method of training approximately linear support vector machine based on variational inequality (VIALSVM) was proposed. The method turns the convex quadratic programming into variational inequality in the process of training approximately linear support vector machine. The method can find the optimal separating hyperplane by solving variational inequality, and can not produce a great deal of data that occupy the computer memory. So the method can improve the training and test speed of SVM in classification greatly. We apply VIALSVM into the multidimensional iris training samples. Experiments show that VIALSVM not only has less misclassification rate, but also has more fast convergence speed than traditional SVM on the foundation of the equal misclassification, and especially in the high dimension training samples.

Automatic seizure detection is very essential for monitoring and rehabilitation of epilepsy patients and will open up new treatment possibilities for saving the lives of epileptic patients. In recent years, many algorithms for the automatic seizure detection have been proposed and applied, in which Support vector machines proved to be a robust machine learning algorithm. The purpose of this study is to compute relevant EEG features and apply a feature selection algorithm to select an optimum set of features for use in a classification scheme for epileptic seizure detection. Thus S VM will thereby yield a better accuracy compared to other algorithms. Effective features such as energy, relative amplitude, standard deviation, coefficient of variation, fluctuation index etc are selected and then these features are fed into the support vector machine for training and classification. This algorithm makes use of Radial Basis Function Kernels for training data and thus obtains more accurate results.

BackgroundInformation on reproductive success is crucial to understanding population dynamics but can be difficult to obtain, particularly for species that birth while denning. For grizzly (Ursus arctos) and polar bears (U. maritimus), den visits are impractical because of safety and logistical considerations. Reproduction is typically documented through direct observation, which can be difficult, costly, and often occurs long after den departure. Reproduction could be documented remotely, however, from post-denning movement data if discernable differences exist between females with and without cubs.MethodsWe trained support vector machines (SVMs) with eight variables derived from telemetry data of female grizzly (2000–2022) and polar bears (1985–2016) with or without cubs during seven periods with lengths ranging from 5 to 60 days starting at den departure. We assessed SVM classification accuracy by withholding two samples (one cub-present, one cub-absent), training SVMs with the remaining data, predicting classification of the withheld samples, and repeating this process for each sample combination. Additionally, we evaluated how classification accuracy for grizzly bears was influenced by sample size, length of the post-departure period, and frequency of standardized location estimates.ResultsAccuracy of predicting cub presence or absence was 87% for grizzly bears with only 5 days of post-departure data and increased to a maximum of 92% with 20 days of data. For polar bears, accuracy was 86% at 5 days post-departure and increased to a maximum of 93% at 50 days. Classification accuracy for grizzly bears increased from 76 to 90% when sample size increased from 10 to 30 bears while holding period length constant (30 days) but did not increase at larger sample sizes. When sample size was held constant, increasing the length of the post-departure period did not affect classification accuracy markedly.ConclusionPresence or absence of grizzly and polar bear cubs can be identified with high accuracy even when SVM models are trained with limited data. Detecting cub presence or absence remotely could improve estimates of reproductive success and litter survival, enhancing our understanding of factors affecting cub recruitment.

A natural way to determine the knowledge embedded within connectionist models is to generate symbolic rules. Nevertheless, extracting rules from Multi Layer Perceptrons (MLPs) is NP-hard. With the advent of social networks, techniques applied to Sentiment Analysis show a growing interest, but rule extraction from connectionist models in this context has been rarely performed because of the very high dimensionality of the input space. To fill the gap we present a case study on rule extraction from ensembles of Neural Networks and Support Vector Machines (SVMs), the purpose being the characterization of the complexity of the rules on two particular Sentiment Analysis problems. Our rule extraction method is based on a special Multi Layer Perceptron architecture for which axis-parallel hyperplanes are precisely located. Two datasets representing movie reviews are transformed into Bag-of-Words vectors and learned by ensembles of neural networks and SVMs. Generated rules from ensembles of MLPs are less accurate and less complex than those extracted from SVMs. Moreover, a clear trade-off appears between rules’ accuracy, complexity and covering. For instance, if rules are too complex, less complex rules can be re-extracted by sacrificing to some extent their accuracy. Finally, rules can be viewed as feature detectors in which very often only one word must be present and a longer list of words must be absent.

This paper presents an optimized support vector machine (SVM) training processor employing the alternative direction method of multipliers (ADMM) optimizer. Low-rank approximation is exploited to reduce the dimension of the kernel matrix by employing the Nyström method. Verified in four datasets, the proposed ADMM-based training processor with rank approximation reduces 32× of matrix dimension with only 2% drop in inference accuracy. Compared to the conventional sequential minimal optimization (SMO) algorithm, the ADMM-based training algorithm is able to achieve a 9.8×10 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">7</sup> shorter latency for training 2048 samples. Hardware optimization techniques, including pre-computation and memory sharing, are proposed to reduce the computational complexity by 62% and the memory usage by 60%. As a proof of concept, an epileptic seizure detector is designed to demonstrate the effectiveness of the proposed optimization techniques. The chip achieves a 153,310× higher energy efficiency and a 364× higher throughput-to-area ratio for SVM training than a high-end CPU. This work provides a promising solution for edge devices which require low-power and real-time training.

This paper presents an assessment method for the corrosion evaluation of carbon steel. The digital image processing technology is used to extract the texture and color characteristics of rusted carbon steel images. First, Q235 carbon steel images with different degrees of corrosion were collected by digital camera and microscope. Second, the texture features of rusted carbon steel surface are extracted based on Gray Level Co-occurrence Matrix, the color features are extracted using color moments. Third, the extracted features are combined and normalized as the inputs of Support Vector Machine (SVM) for training and classification. We use Particle Swarm Optimization (PSO) algorithm to optimize the SVM classifier. The accuracy of the classification is up to 97.5%.

Support Vector Machines (SVM) are believed to be as powerful as Artificial Neural Networks (ANN) in modeling complex problems while avoiding some of the drawbacks of the latter such as local minimæ or reliance on architecture. However, a question that remains to be answered is whether SVM users may expect improvements in the interpretability of their models, namely by using rule extraction methods already available to ANN users. This study successfully applies the Orthogonal Search-based Rule Extraction algorithm (OSRE) to Support Vector Machines. The study evidences the portability of rules extracted using OSRE, showing that, in the case of SVM, extracted rules are as accurate and consistent as those from equivalent ANN models. Importantly, the study also shows that the OSRE method benefits from SVM specific characteristics, being able to extract less rules from SVM than from equivalent ANN models.KeywordsData MiningSupport Vector MachinesArtificial Neural NetworksOrthogonal Search-based AlgorithmOSREPedagogicalDecompositionalRule Extraction