Maximum Robustness Criterion on Kernel Selection of Support Vector Machine

In many classification applications, Support Vector Machines (SVMs) have proven to be highly performing and easy to handle classifiers with very good generalization abilities. However, one drawback of the SVM is its rather high classification complexity which scales linearly with the number of Support Vectors (SVs). This is due to the fact that for the classification of one sample, the kernel function has to be evaluated for all SVs. To speed up classification, different approaches have been published, most which of try to reduce the number of SVs. In our work, which is especially suitable for very large datasets, we follow a different approach: as we showed in (Zapien et al. 2006), it is effectively possible to approximate large SVM problems by decomposing the original problem into linear subproblems, where each subproblem can be evaluated in Ω(1). This approach is especially successful, when the assumption holds that a large classification problem can be split into mainly easy and only a few hard subproblems. On standard benchmark datasets, this approach achieved great speedups while suffering only sightly in terms of classification accuracy and generalization ability. In this contribution, we extend the methods introduced in (Zapien et al. 2006) using not only linear, but also non-linear subproblems for the decomposition of the original problem which further increases the classification performance with only a little loss in terms of speed. An implementation of our method is available in (Ronneberger and et al.) Due to page limitations, we had to move some of theoretic details (e.g. proofs) and extensive experimental results to a technical report (Zapien et al. 2007).

To design a fuzzy rule-based classification system (fuzzy classifier) with good generalization ability in a high dimensional feature space has been an active research topic for a long time. As a powerful machine learning approach for pattern recognition problems, the support vector machine (SVM) is known to have good generalization ability. More importantly, an SVM can work very well on a high- (or even infinite) dimensional feature space. This paper investigates the connection between fuzzy classifiers and kernel machines, establishes a link between fuzzy rules and kernels, and proposes a learning algorithm for fuzzy classifiers. We first show that a fuzzy classifier implicitly defines a translation invariant kernel under the assumption that all membership functions associated with the same input variable are generated from location transformation of a reference function. Fuzzy inference on the IF-part of a fuzzy rule can be viewed as evaluating the kernel function. The kernel function is then proven to be a Mercer kernel if the reference functions meet a certain spectral requirement. The corresponding fuzzy classifier is named positive definite fuzzy classifier (PDFC). A PDFC can be built from the given training samples based on a support vector learning approach with the IF-part fuzzy rules given by the support vectors. Since the learning process minimizes an upper bound on the expected risk (expected prediction error) instead of the empirical risk (training error), the resulting PDFC usually has good generalization. Moreover, because of the sparsity properties of the SVMs, the number of fuzzy rules is irrelevant to the dimension of input space. In this sense, we avoid the "curse of dimensionality." Finally, PDFCs with different reference functions are constructed using the support vector learning approach. The performance of the PDFCs is illustrated by extensive experimental results. Comparisons with other methods are also provided.

Relative density based support vector machine

Support vector machine is a machine learning method. It takes on the good generalization ability and prediction accuracy. But the parameters of SVM model seriously affect the generalization ability and prediction accuracy of SVM model on the great extent. So an improved particle swarm optimization (PSO) algorithm based on chaotic search is introduced into the SVM model in propose a novel data classification (AMPSVM) method for processing the complex data. The first, the ergodicity, stochastic property, and regularity of chaos is used to chaotically search the current best individual, which randomly replaces the selected individual in the population in order to speed up evolution, improve the searching ability, convergence speed and accuracy. Then the improved PSO algorithm is used to select and optimize the parameters of the SVM (AMPSVM) model in order to improve the learning performance and generalization ability of the SVM model. In order to verify the effectiveness of the AMPSVM method, UCI data is selected in here. The experiment results show that the proposed AMPSVM method takes on the strong generalization ability, best sensitivity and higher classification accuracy.

State forecast of machine using support vector machines has good generalization ability in situation of rare samples. Appropriate parameter selection is very crucial to the learning results and generalization ability of support vector machines. In addition, embedding dimension influences the phase space reconstitution of nonlinear systems, as well as the precision of machine state forecasting. In this paper, an approach to optimize the parameters of SVM and the embedding dimension based on genetic algorithms was proposed. The proposed model is applied to the tendency forecasting of the vibration of shovel electric drive system. The results show that it can avoid blindness of manually selection of parameters and meanwhile improves the prediction performance greatly.

Beras Miskin (Raskin) Program is a program of social protection, as supporters of other programs such as nutrition improvement, healthy increase, education and productivity improvement of Poor Households. According to Badan Pusat Statistika, there were 14 criteria to determine a household is classified as poor households. Based on these criteria it will be classified of recipient households and non-recipient households of Beras Miskin (Raskin) Program by Support Vector Machine (SVM) method using LibSVM. The concept of classification by SVM is search for the best hyperplane which serves as a separator of two classes of data in the input space. Kernel function is used to convert the data into a higher dimensional space to allow a separation. LibSVM is a package program created by Chih-Chung Chang and Chih-Jen Lin from Department of Computer Science at National Taiwan University. The method used by LibSVM to obtain global solution of duality lagrange problem is decomposition method. To determine the best parameters of kernel function, used k-vold cross validation method and grid search algorithm. In this classification by SVM method using LibSVM, obtain the best accuracy value as 83,1933%, which is the kernel function Radial Basis Function (RBF). Keywords : Beras Miskin (Raskin) Program, Classification, Support Vector Machine (SVM), LibSVM, Kernel Function

Top-coal caving technology is a productive and efficient method in modern mechanized coal mining, the study of coal-rock recognition is key to realizing automation in comprehensive mechanized coal mining. In this paper we propose a new discriminant analysis framework for coal-rock recognition. In the framework, a data acquisition model with vibration and acoustic signals is designed and the caving dataset with 10 feature variables and three classes is got. And the perfect combination of feature variables can be automatically decided by using the multi-class F-score (MF-Score) feature selection. In terms of nonlinear mapping in real-world optimization problem, an effective minimum enclosing ball (MEB) algorithm plus Support vector machine (SVM) is proposed for rapid detection of coal-rock in the caving process. In particular, we illustrate how to construct MEB-SVM classifier in coal-rock recognition which exhibit inherently complex distribution data. The proposed method is examined on UCI data sets and the caving dataset, and compared with some new excellent SVM classifiers. We conduct experiments with accuracy and Friedman test for comparison of more classifiers over multiple on the UCI data sets. Experimental results demonstrate that the proposed algorithm has good robustness and generalization ability. The results of experiments on the caving dataset show the better performance which leads to a promising feature selection and multi-class recognition in coal-rock recognition.

Support vector machine (SVM) has been often used in binary classification. In order to seek the guidance principles of the kernel function selection, this paper analyzed a variety of kernel functions used to construct the SVM classifiers and carried out comparative studies on the 4 data sets for binary classification of UCI Machine Learning Repository. The experimental results show that, using the nu-SVC with radial basis kernel function (RBF) has the optimal classification accuracy, but using the C-SVC with RBF kernel function has the best generalization ability.

The excellent generalisation ability of the Support Vector Machine (SVM) algorithm has made it one of the most popular statistical learning theories in supervised machine learning. The classification accuracy and effectiveness of SVM is highly sensitive to the kernel function used during the training process. This paper compares linear, polynomial, and Gaussian kernel functions for evaluating their contribution to SVM for accurately and effectively classifying healthy and faulty status of rotating machinery. A three-phase induction motor and a four-stroke diesel engine were considered as the machinery for this study. Acoustic signals coming from these machines were collected using microphones and Fast Fourier Transform (FFT) was used to extract the magnitudes of the dominant frequency components of the signals. The extracted ominant frequency components are considered as acoustic signatures and their variations are taken as condition monitoring parameters. The results show that with the second-order polynomial kernel function, SVM achieved an accuracy of at least 2.4% greater than the other kernel functions with 1.2% less training time. Furthermore, the third-order polynomial kernel function found to be the second best choice.

Machine Learning is a modern and actively developing field of computer science, devoted to extracting and estimating dependencies from empirical data. It combines such fields as statistics, optimization theory and artificial intelligence. In practical tasks, the general aim of Machine Learning is to construct algorithms able to generalize and predict in previously unseen situations based on some set of examples. Given some finite information, Machine Learning provides ways to exract knowledge, describe, explain and predict from data. Kernel Methods are one of the most successful branches of Machine Learning. They allow applying linear algorithms with well-founded properties such as generalization ability, to non-linear real-life problems. Support Vector Machine is a well-known example of a kernel method, which has found a wide range of applications in data analysis nowadays. In many practical applications, some additional prior knowledge is often available. This can be the knowledge about the data domain, invariant transformations, inner geometrical structures in data, some properties of the underlying process, etc. If used smartly, this information can provide significant improvement to any data processing algorithm. Thus, it is important to develop methods for incorporating prior knowledge into data-dependent models. The main objective of this thesis is to investigate approaches towards learning with kernel methods using prior knowledge. Invariant learning with kernel methods is considered in more details. In the first part of the thesis, kernels are developed which incorporate prior knowledge on invariant transformations. They apply when the desired transformation produce an object around every example, assuming that all points in the given object share the same class. Different types of objects, including hard geometrical objects and distributions are considered. These kernels were then applied for images classification with Support Vector Machines. Next, algorithms which specifically include prior knowledge are considered. An algorithm which linearly classifies distributions by their domain was developed. It is constructed such that it allows to apply kernels to solve non-linear tasks. Thus, it combines the discriminative power of support vector machines and the well-developed framework of generative models. It can be applied to a number of real-life tasks which include data represented as distributions. In the last part of the thesis, the use of unlabelled data as a source of prior knowledge is considered. The technique of modelling the unlabelled data with a graph is taken as a baseline from semi-supervised manifold learning. For classification problems, we use this apporach for building graph models of invariant manifolds. For regression problems, we use unlabelled data to take into account the inner geometry of the input space. To conclude, in this thesis we developed a number of approaches for incorporating some prior knowledge into kernel methods. We proposed invariant kernels for existing algorithms, developed new algorithms and adapted a technique taken from semi-supervised learning for invariant learning. In all these cases, links with related state-of-the-art approaches were investigated. Several illustrative experiments were carried out on real data on optical character recognition, face image classification, brain-computer interfaces, and a number of benchmark and synthetic datasets.

The field of genomics has seen a glorious explosion of high-quality data, with tremendous strides having been made in genomic sequencing instruments and computational genomics applications meant to make sense of the data. A common use case for genomics data is to answer the question if a specific genetic signature is correlated with some disease manifestations. Support Vector Machine (SVM) is a widely used classifier in computational literature. Previous studies have shown success in using these SVMs for the above use case of genomics data. However, SVMs suffer from a widely-recognized scalability problem in both memory use and computational time. It is as yet an unanswered question if training such classifiers can scale to the massive sizes that characterize many of the genomics data sets. We answer that question here for a specific dataset, in order to decipher whether some regulatory module of a particular combinatorial epigenetic “pattern” will regulate the expression of a gene. However, the specifics of the dataset is likely of less relevance to the claims of our work. We take a proposed theoretical technique for efficient training of SVM, namely Cascade SVM, create our classifier called EP-SVM, and empirically evaluate how it scales to the large genomics dataset. We implement Cascade SVM on the Apache Spark platform and open source this implementation1. Through our evaluation, we bring out the computational cost on each application process, the way of distributing the overall workload among multiple processes, which can potentially execute on different cores or different machines, and the cost of data transfer to different cores or different machines. We believe we are the first to shed light on the computational and network costs of training an SVM on a multi-dimensional genomics dataset. We also evaluate the accuracy of the classifier result as a function of the parameters of the SVM model.

Support vector machines (SVMs) are a class of machine learning algorithms which use kernel functions to map data into feature space, where a separating hyperplane can be computed to classify the data. Generally, SVMs are reliant on the selection of kernel function and parameters, which can severely downgrade the performance of the classifier if mishandled. In this paper, we compare the performance of the Euclidean-SVM (ESVM), that have low dependency on kernel selection, with conventional SVM. The comparison are conducted with two different datasets. Results show that the ESVM has higher performance consistency as compared to SVM.

Long-term prediction model of rockburst in underground openings using heuristic algorithms and support vector machines

A novel measure of attribute significance with complexity weight

Because hyperspectral remote sensing images have high band dimension and limited training samples, it is hard to achieve acceptable classification results using conventional statistical pattern recognition methods. Nevertheless, support vector machine (SVM) is suitable for hyperspectral image classification owing to its good generalization ability and minimal structural risk. Aimed to address the problems of large subjectivity, excessive time consumption, and low optimization accuracy in traditional SVM parameter selection, we applied genetic algorithm to optimize the model parameters. Using EO-1 Hyperion data as an example, an SVM object classification algorithm was proposed based on genetic algorithm optimization. To determine the effectiveness and superiority of this algorithm, a comprehensive evaluation and comparative analysis were performed with the classification effects of the cross-validation and maximum likelihood methods, respectively. Experimental results demonstrated that the proposed algorithm can effectively avert the aimlessness of artificial parameter selection and automatically optimize the SVM model parameters. It achieved relatively high overall classification accuracy (91.23%), which was 16.42% and 4.48% higher than the maximum likelihood method and cross-validation method, respectively.