Comparing the performance of support vector machines to regression with structural risk minimisation

The statistical learning theory has formulated the Structural Risk Minimization (SRM) principle, based upon the functional form of risk bound on the generalization performance of a learning machine. This paper addresses the application of this formula, which is equivalent to a complexity penalty, to model selection tasks for decision trees, whereas the quantization of the machine capacity for decision trees is estimated using an empirical approach. Experimental results show that, for either classification or regression problems, this novel strategy of decision tree pruning performs better than alternative methods. We name classification and regression trees pruned by virtue of this methodology as Statistical Learning Intelligent Trees (SLIT).

Objective: Support Vector Machine (SVM) is a machine-learning method, based on the principle of structural risk minimization, which performs well when applied to data outside the training set. In this paper, SVM was applied to predict 5-year survival status of patients with nasopharyngeal carcinoma (NPC) after treatment, we expect to find a new way for prognosis studies in cancer so as to assist right clinical decision for individual patient. Methods: Two modelling methods were used in the study; SVM network and a standard parametric logistic regression were used to model 5-year survival status. And the two methods were compared on a prospective set of patients not used in model construction via receiver operating characteristic (ROC) curve analysis. Results: The SVM1, trained with the 25 original input variables without screening, yielded a ROC area of 0.868, at sensitivity to mortality of 79.2% and the specificity of 94.5%. Similarly, the SVM2, trained with 9 input variables which were obtained by optimal input variable selection from the 25 original variables by logistic regression screening, yielded a ROC area of 0.874, at a sensitivity to mortality of 79.2% and the specificity of 95.6%, while the logistic regression yielded a ROC area of 0.751 at a sensitivity to mortality of 66.7% and gave a specificity of 83.5%. Conclusion: SVM found a strong pattern in the database predictive of 5-year survival status. The logistic regression produces somewhat similar, but better, results. These results show that the SVM models have the potential to predict individual patient's 5-year survival status after treatment, and to assist the clinicians for making a good clinical decision.

Support vector machine (SVM) is a new sort of machine learning method based on Structure Risk Minimization (SRM) principle, which has high generalization capability. Many problems with small samples, nonlinearity or high dimension in pattern recognition could be solved by the method. In this paper, the traffic data on freeway were taken as research objects and an information fusion algorithm based on SVM about freeway incident detection was proposed. A SVM was trained and tested using the data obtained from the simulation under the condition of incident and non-incident. Compared with the multi-layer feed forward neural network (MLF) algorithm trained with the same data, the simulation results showed that the SVM offers a lower misclassification rate, higher correct detection rate and lower false alarm, and it can improve the detection performance.

Osteoarthritis is a chronic joint disease that occurs when the protective cartilage that cushions the ends of bones wears down over time and fails to be repaired. The common form of the disease is knee osteoarthritis while it can affect all body parts with joints, such as hands, ankles, hips, and spine. The major cause of knee osteoarthritis is the continuous depletion of its cartilage. During the diagnosis, machine learning is used because early prevention is necessary for proper treatment. This study, therefore, considers classification methods of Support Vector Machine (SVM) and clustering methods using fuzzy clusterings such as Fuzzy C-Means (FCM), Fuzzy Possibilistic C-Means (FPCM), and Fuzzy Possibilistic C-Means based on kernel (FPCMK) to analyze of knee osteoarthritis. SVM is a machine learning technique that works based on the principle of structural risk minimization (SRM) to obtain the best hyperplane to separate two or more classes in input space. Otherwise, the fuzzy clustering is to determine the value of a distance and to know and measure the similarity of each object to be observed. FPCMK uses the kernel Radial Base Function (RBF) in the fuzzy clustering method. The kernel function is applicable for handling non-separable data problems. This method will be compared to the level of the measured parameter; their accuracy, recall, precision, and f1 score. The greatest level of accuracy is generated from SVM with an accuracy value of 86.7%, then followed by FPCMK with an accuracy value of 85.5%.

An efficient optimization technique, support vector regression (SVR) approach, is proposed for designing of Ka-band MMIC sub-harmonically pumped image rejection diode mixer. This SVR approach comes from the support vector machine (SVM) learning theory, which is based on the structural risk minimization (SRM) principle and leads good generalization ability. With this method, a Ka-band MMIC 4th harmonic image rejection diode mixer is designed using comercial United Monolithic Semiconductors process. Details of design approach and outcome of performance simulations is presented.

A combination classification algorithm, ER-SVM, is proposed to improve the generalization performance of support vector machine (SVM) by directly making full use of the empirical risk (ER) information of SVM in the paper. SVM classification is the implementation of structure risk minimization (SRM) principle. SVM may achieve SRM from the minimal summation of ER and VC confidence according to the theory of VC dimension. However, the ER is seldom zero for a trained SVM in practice. That is, though the minimal summation of ER and VC confidence can be achieved in theory, it is very time-consuming in parameters selection for a given task to make ER zero. In order to overcome such difficulty, a combination classification algorithm is proposed to improve the performance by utilizing ER information. The SR arising from the existing ER is reduced by using aided nearest neighbor method. In addition, the proposed algorithm is independent of training parameters in SVM. The experimental results verify the effectiveness of the proposed algorithm

One-class classification is an important problem with applications in several different areas such as novelty detection, anomaly detection, outlier detection and machine monitoring. In this paper, we propose two novel methods for one-class classification, referred to as NNDDSRM and kNNDDSRM. The methods are based on the principle of structural risk minimization and the nearest neighbor data description (NNDD) one-class classifier. Experiments carried out using both artificial and real-world datasets show that the proposed methods are able to significantly reduce the number of stored prototypes in comparison to NNDD. The experimental results also show that the proposed methods outperformed NNDD—in terms of the area under the receiver operating characteristic (ROC) curve—on four of the five datasets considered in the experiments and had a similar performance on the remaining one.

Chlorination has made water supply safe from bacteria and viruses, and has almost completely eliminated risks of waterborne diseases such as typhoid fever, cholera and dysentery. However, the health benefit of chlorination has introduced some possible risks from the by-products of the disinfection process. In this work, we propose the use of a learning algorithm ‘support vector machine (SVM)’ for the prediction of the water pollutant total exposure index (TEI) and cumulative risk (CR). The SVM is an artificial intelligent approach that could capture the input/output mapping from the given data. Support vector machines were developed based on the structural risk minimization principle from statistical learning theory in which the empirical risk and the machine complexity are optimized. The proposed algorithm was validated by using statistical measurements on experimental data different from those used for the training (out-of-sample testing) to evaluate the algorithm's ‘generalizability’. The experimental results demonstrate a good correlation between the SVM predicted values and those conventionally estimated using probabilistic (Monte Carlo) simulations. In both cases, the TEI and CR values were estimated using a set of data extracted from the national database IRIS. The SVM, once trained with representative data, could be used easily and effectively in comparison with other conventional estimation methods of TEI and CR.

The market demand for dental implants is growing at a significant pace. In practice, some dental implants do not succeed. Important questions in this regard concern whether machine learning techniques could be used to predict whether an implant will be successful and which are the best techniques for this problem. This paper presents a comparative study on machine learning techniques for prediction of success of dental implants. The techniques compared here are: (a) constructive RBF neural networks (RBF-DDA), (b) support vector machines (SVM), (c) k nearest neighbors (kNN), and (d) a recently proposed technique, called NNSRM, which is based on kNN and the principle of structural risk minimization. We present a number of simulations using real-world data. The simulations were carried out using 10-fold cross-validation and the results show that the methods achieve comparable performance, yet NNSRM and RBF-DDA produced smaller classifiers.

This paper studies the use of wavelet and support vector machine (SVM) in machinery condition prediction. SVM is based on the VC dimension theory of statistical learning and the principle of structural risk minimization, and has shown advantages in solving the problem with limited sample, nonlinear and high dimensional pattern recognition. The soft failure of mechanical equipment makes its performance drop gradually, which occupies a large proportion and has certain regularity. The performance can be evaluated and predicted through early state monitoring and data analysis. The paper models the vibration signal from the rear pad of a gas blower and analyzes the 1-step and multi-step forecasting of wavelet transformation and SVM (WT-SVM model) and SVM model.

This paper studies the use of wavelet and support vector machine (SVM) in machinery condition prediction. SVM is based on the VC dimension theory of statistical learning and the principle of structural risk minimization, and has shown advantages in solving the problem with limited sample, nonlinear and high dimensional pattern recognition. The soft failure of mechanical equipment makes its performance drop gradually, which occupies a large proportion and has certain regularity. The performance can be evaluated and predicted through early state monitoring and data analysis. The paper models the vibration signal from the rear pad of a gas blower and analyzes the 1-step and multi-step forecasting of wavelet transformation and SVM (WT-SVM model) and SVM model.

Traditional methods used in the information security risk management are mostly based on the statistics, their validity of application are limited to large sample situations, while the statistical learning theory is introduced to the innovation of information security management, its structural risk minimization principle and from which the support vector machine developed offer new theory basis for predicting the security risk and achieve the minimal risk. This paper not only detailedly discusses the structural risk minimization principle and from which the support vector machine developed offer new theory basis for predicting the security risk and achieve the minimal risk, but also puts forwards the idea of using structural risk minimization principle and the support vector machine in information security risk management, which provides the brand-new mentality to the information security risk management.

Lithology prediction by support vector classifiers using inverted seismic attributes data and petrophysical logs as a new approach and investigation of training data set size effect on its performance in a heterogeneous carbonate reservoir

Bond rating is regarded as an important event for measuring financial risk of companies and for determining the investment returns of investors. As a result, it has been a popular research topic for researchers to predict companies' credit ratings by applying statistical and machine learning techniques. The statistical techniques, including multiple regression, multiple discriminant analysis (MDA), logistic models (LOGIT), and probit analysis, have been traditionally used in bond rating. However, one major drawback is that it should be based on strict assumptions. Such strict assumptions include linearity, normality, independence among predictor variables and pre-existing functional forms relating the criterion variablesand the predictor variables. Those strict assumptions of traditional statistics have limited their application to the real world. Machine learning techniques also used in bond rating prediction models include decision trees (DT), neural networks (NN), and Support Vector Machine (SVM). Especially, SVM is recognized as a new and promising classification and regression analysis method. SVM learns a separating hyperplane that can maximize the margin between two categories. SVM is simple enough to be analyzed mathematical, and leads to high performance in practical applications. SVM implements the structuralrisk minimization principle and searches to minimize an upper bound of the generalization error. In addition, the solution of SVM may be a global optimum and thus, overfitting is unlikely to occur with SVM. In addition, SVM does not require too many data sample for training since it builds prediction models by only using some representative sample near the boundaries called support vectors. A number of experimental researches have indicated that SVM has been successfully applied in a variety of pattern recognition fields. However, there are three major drawbacks that can be potential causes for degrading SVM's performance. First, SVM is originally proposed for solving binary-class classification problems. Methods for combining SVMs for multi-class classification such as One-Against-One, One-Against-All have been proposed, but they do not improve the performance in multi-class classification problem as much as SVM for binary-class classification. Second, approximation algorithms (e.g. decomposition methods, sequential minimal optimization algorithm) could be used for effective multi-class computation to reduce computation time, but it could deteriorate classification performance. Third, the difficulty in multi-class prediction problems is in data imbalance problem that can occur when the number of instances in one class greatly outnumbers the number of instances in the other class. Such data sets often cause a default classifier to be built due to skewed boundary and thus the reduction in the classification accuracy of such a classifier. SVM ensemble learning is one of machine learning methods to cope with the above drawbacks. Ensemble learning is a method for improving the performance of classification and prediction algorithms. AdaBoost is one of the widely used ensemble learning techniques. It constructs a composite classifier by sequentially training classifiers while increasing weight on the misclassified observations through iterations. The observations that are incorrectly predicted by previous classifiers are chosen more often than examples that are correctly predicted. Thus Boosting attempts to produce new classifiers that are better able to predict examples for which the current ensemble's performance is poor. In this way, it can reinforce the training of the misclassified observations of the minority class. This paper proposes a multiclass Geometric Mean-based Boosting (MGM-Boost) to resolve multiclass prediction problem. Since MGM-Boost introduces the notion of geometric mean into AdaBoost, it can perform learning process considering the geometric mean-based accuracy and errors of multiclass. This study applies MGM-Boost to the real-world bond rating case for Korean companies to examine the feasibility of MGM-Boost. 10-fold cross validations for threetimes with different random seeds are performed in order to ensure that the comparison among three different classifiers does not happen by chance. For each of 10-fold cross validation, the entire data set is first partitioned into tenequal-sized sets, and then each set is in turn used as the test set while the classifier trains on the other nine sets. That is, cross-validated folds have been tested independently of each algorithm. Through these steps, we have obtained the results for classifiers on each of the 30 experiments. In the comparison of arithmetic mean-based prediction accuracy between individual classifiers, MGM-Boost (52.95%) shows higher prediction accuracy than both AdaBoost (51.69%) and SVM (49.47%). MGM-Boost (28.12%) also shows the higher prediction accuracy than AdaBoost (24.65%) and SVM (15.42%)in terms of geometric mean-based prediction accuracy. T-test is used to examine whether the performance of each classifiers for 30 folds is significantly different. The results indicate that performance of MGM-Boost is significantly different from AdaBoost and SVM classifiers at 1% level. These results mean that MGM-Boost can provide robust and stable solutions to multi-classproblems such as bond rating.

Support vector machines for aerobic fitness prediction of athletes