Nonlinear programming for classification problems in machine learning

As the scale and dimensionality of data continue to grow in many applications of data analytics (e.g., bioinformatics, finance, computer vision, medical informatics), it becomes critical to develop efficient and effective algorithms to solve numerous machine learning and data mining problems. This tutorial will focus on simple yet practically effective techniques and algorithms for big data analytics. In the first part, we plan to present the state-of-the-art large-scale optimization algorithms, including various stochastic gradient descent methods, stochastic coordinate descent methods and distributed optimization algorithms, for solving various machine learning problems. In the second part, we will focus on randomized approximation algorithms for learning from large-scale data. We will discuss i) randomized algorithms for low-rank matrix approximation; ii) approximation techniques for solving kernel learning problems; iii) randomized reduction methods for addressing the high-dimensional challenge. Along with the description of algorithms, we will also present some empirical results to facilitate understanding of different algorithms and comparison between them.

Using information technology such as search engine for collecting and monitoring of network public opinion is a practical and effective method. This paper puts forward an improved algorithm of SVM classifier based on incremental learning online, and then implements a topic crawler system for network public opinion and to grab the public opinion.

The problem of optimization in machine learning is well established but it entails several approximations. The theory of Hilbert spaces, which is principled and well established, helps solve the representation problem in machine learning by providing a rich (universal) class of functions where the optimization can be conducted. Working with functions is cumbersome, but for the class of reproducing kernel Hilbert spaces (RKHSs) it is still manageable provided the algorithm is restricted to inner products. The best example is the support vector machine (SVM), which is a batch mode algorithm that uses a very efficient (supralinear) optimization procedure. However, the problem of SVMs is that they display large memory and computational complexity. For the large-scale data limit, SVMs are restrictive because for fast operation the Gram matrix, which increases with the square of the number of samples, must fit in computer memory. The computation in this best-case scenario is also proportional to number of samples square. This is not specific to the SVM algorithm and is shared by kernel regression. There are also other relevant data processing scenarios such as streaming data (also called a time series) where the size of the data is unbounded and potentially nonstationary, therefore batch mode is not directly applicable and brings added difficulties.

This chapter explains the important concepts and basics requirements of support vector machine (SVM) and apply it to real-life models. It focuses on forecasting the outcome values of time series using an SVM. The objectives of statistical learning are understanding and forecasting. SVM is used as a popular tool for classification of problems in machine learning. SVMs simply separate the given data into different classes by finding the line or hyper-plane and separate the kernel supports to find the hyper-plane in the higher dimension. In regression problems, the support vector regression works on the same principle as SVM. To reduce computational complexity in SVM, J. A. K. Suykensand and J. Vandewalle introduced the LS-SVM classifier in 1999. Though LS-SVM is suitable for time series forecasting, to make it more convenient, it has been modified to DLS-SVM using a slightly improved computation technique.

Machine learning and pattern recognition recently become a hot topic in computing world. This is due to the fast-growing of resources as well as techniques that make it easier to solve machine learning and pattern recognition problems. Problems that require machine learning solutions may be very simple for humans but actually can be very complex for machines to solve them. Face recognition is amongst those problems. Almost all human can easily recognize others without require specific knowledge to do it, different from machines which require its. This paper discussed face recognition task using machine learning strategies which involved Kernel Principal Component Analysis (KPCA) and Support Vector Machine (SVM) to identify person. KPCA extracted features from 2D image input and produced the important features of an image input. The extracted face features are recognized by SVM by classifying human face according to their stored identity in a database. SVM, which was basically a binary classifier worked by using one-against-one strategy to compare the face feature vector of a single test image to the stored face image in a face image database. Experiment results on grayscale images with size 92x112 pixels gave 96.25% of accuracy rate. Hence, KPCA and SVM for face recognition is a robust machine learning method.

Support Vector Machines (SVMs) are a type of supervised machine learning algorithm widely used for classification tasks. In contrast to traditional methods that split the data into separate training and testing sets, here we propose an innovative approach where subsets of the original data are randomly selected to train the model multiple times. This iterative training process aims to identify a representative data subset, leading to improved inferences about the population. Additionally, we introduce a novel distance-based kernel specifically designed for binary-type features based on a similarity matrix that efficiently handles both binary and multi-class classification problems. Computational experiments on publicly available datasets of varying sizes demonstrate that our proposed method significantly outperforms existing approaches in terms of classification accuracy. Furthermore, the distance-based kernel achieves superior performance compared to other well-known kernels from the literature and those used in previous studies on the same datasets. These findings validate the effectiveness of our proposed classification method and distance-based kernel for SVMs. By leveraging random subset selection and a unique kernel design, we achieve notable improvements in classification accuracy. These results have significant implications for diverse classification problems in Machine Learning and data analysis.

Convex optimization methods are used for many machine learning models such as support vector machine. However, the requirement of a convex formulation can place limitations on machine learning models. In recent years, a number of machine learning methods not requiring convexity have emerged. In this paper, we study non-convex optimization problems on the Stiefel manifold in which the feasible set consists of a set of rectangular matrices with orthonormal column vectors. We present examples of non-convex optimization problems in machine learning and apply three nonlinear optimization methods for finding a local optimal solution; geometric gradient descent method, augmented Lagrangian method of multipliers, and alternating direction method of multipliers. Although the geometric gradient method is often used to solve non-convex optimization problems on the Stiefel manifold, we show that the alternating direction method of multipliers generally produces higher quality numerical solutions within a reasonable computation time.

A principled approach to machine learning (ML) problems because of its mathematical foundations in statistical learning theory, support vector machines (SVM), a non-parametric method, require all the data to be available during the training phase. However, once the model parameters are identified, SVM relies only, for future prediction, on a subset of these training instances, called support vectors (SV). The SVM model is mathematically written as a weighted sum of these SV whose number, rather than the dimensionality of the input space, defines SVM's complexity. Since the final number of these SV can be up to half the size of the training dataset, SVM becomes challenged to run on energy aware computing platforms. We propose in this work Knee-Cut SVM (KCSVM) and Knee-Cut Ordinal Optimization inspired SVM (KCOOSVM) that use a soft trick of ordered kernel values and uniform subsampling to reduce SVM's prediction computational complexity while maintaining an acceptable impact on its generalization capability. When tested on several databases from UCL KCSVM and KCOOSVM produced promising results, comparable to similar published algorithms.

The support vector machine (SVM) has been widely applied for classification problems in both machine learning and statistics. Despite its popularity, however, SVM has some drawbacks in certain situations. In particular, the SVM classifier can be very sensitive to outliers in the training sample. Moreover, the number of support vectors (SVs) can be very large in many applications. To circumvent these drawbacks, we propose the robust truncated hinge loss SVM (RSVM), which uses a truncated hinge loss. The RSVM is shown to be more robust to outliers and to deliver more accurate classifiers using a smaller set of SVs than the standard SVM. Our theoretical results show that the RSVM is Fisher-consistent, even when there is no dominating class, a scenario that is particularly challenging for multicategory classification. Similar results are obtained for a class of margin-based classifiers.

Decentralized optimization to minimize a finite sum of functions, distributed over a network of nodes, has been a significant area within control and signal-processing research due to its natural relevance to optimal control and signal estimation problems. More recently, the emergence of sophisticated computing and large-scale data science needs have led to a resurgence of activity in this area. In this article, we discuss decentralized first-order gradient methods, which have found tremendous success in control, signal processing, and machine learning problems, where such methods, due to their simplicity, serve as the first method of choice for many complex inference and training tasks. In particular, we provide a general framework of decentralized first-order methods that is applicable to directed and undirected communication networks alike and show that much of the existing work on optimization and consensus can be related explicitly to this framework. We further extend the discussion to decentralized stochastic first-order methods that rely on stochastic gradients at each node and describe how local variance reduction schemes, previously shown to have promise in the centralized settings, are able to improve the performance of decentralized methods when combined with what is known as gradient tracking. We motivate and demonstrate the effectiveness of the corresponding methods in the context of machine learning and signal-processing problems that arise in decentralized environments.

Multivariate discrimination or classification is one of the best-studied problem in machine learning, with a plethora of well-tested and well-performing algorithms. There are also several good general textbooks [1–9] on the subject written to an average engineering, computer science, or statistics graduate student; most of them are also accessible for an average physics student with some background on computer science and statistics. Hence, instead of writing a generic introduction, we concentrate here on relating the subject to a practitioner experimental physicist. After a short introduction on the basic setup (Section 1) we delve into the practical issues of complexity regularization, model selection, and hyperparameter optimization (Section 2), since it is this step that makes high-complexity non-parametric fitting so different from low-dimensional parametric fitting. To emphasize that this issue is not restricted to classification, we illustrate the concept on a low-dimensional but non-parametric regression example (Section 2.1). Section 3 describes the common algorithmic-statistical formal framework that unifies the main families of multivariate classification algorithms. We explain here the large-margin principle that partly explains why these algorithms work. Section 4 is devoted to the description of the three main (families of) classification algorithms, neural networks, the support vector machine, and AdaBoost. We do not go into the algorithmic details; the goal is to give an overview on the form of the functions these methods learn and on the objective functions they optimize. Besides their technical description, we also make an attempt to put these algorithm into a socio-historical context. We then briefly describe some rather heterogeneous applications to illustrate the pattern recognition pipeline and to show how widespread the use of these methods is (Section 5). We conclude the chapter with three essentially open research problems that are either relevant to or even motivated by certain unorthodox applications of multivariate discrimination in experimental physics.

Classification and high dimensional data are the challenging problems in machine learning. In this paper, we carry out an empirical study of model selection for high dimensional data. We select three single classifiers, i.e., decision tree, support vector machine, and naive bayes, and two ensemble methods, i.e., bagging and random forest, to compare the performance of these five methods through real high dimensional data. Six real gene datasets with high correlation are used in this empirical study. Experimental results have illustrated that ensemble methods are better than single classifiers in terms of accuracy metric. In which, decision tree model is the weakest one, and bagging is the best model. The performance of random forest is much more efficient in terms of computing time metrics compared with bagging. From this empirical study, we can conclude that ensemble methods are an effective solution for high dimensional data.KeywordsHigh dimensional dataSupervised learningClassification

Time series analysis is an important and complex problem in machine learning. In this paper, least squares support vector machine (LS-SVM) combined with particle swarm optimization (PSO) is used to time series prediction. The LS-SVM can overcome some shortcoming in the multilayer perceptron (MLP) and the PSO is used to tune the LS-SVM parameters automatically. A benchmark problem, Hénon map time series, has been used as an example for demonstration. It is showed this approach can escape from the blindness of man-made choice of the LS-SVM parameters. It enhances the efficiency and the capability of prediction.

Spike sorting of neural data from single electrode recordings is a hard problem in machine learning that relies on significant input by human experts. We approach the task of learning to detect and classify spike waveforms in additive noise using two stages of large margin kernel classification and probability regression. Controlled numerical experiments using spike and noise data extracted from neural recordings indicate significant improvements in detection and classification accuracy over linear amplitude- and template-based spike sorting techniques.

From the observation in 2014, cancer was the number one cause of deaths in Thailand and has been continuously increased until present. Cholangiocarcinoma is the subset of liver cancer, which is one of the top five cancers founded in Thailand. To reduce the risk of cholangiocarcinoma in patients, we need to find the factors of causing cancer and predict the probability of cancer earlier. However, the probability of cholangiocarcinoma is less than 1%. This causes imbalance classification problem in machine learning. In this paper, we propose an adaptive cancer prognosis framework for cholangiocarcinoma based on machine learning techniques, namely “CanWiser”. CanWiser is used to automatically learn the patient dataset (e.g., demographics and laboratory test results), pre-process data, oversample data to solve the imbalance problem using SMOTE, generate the prediction models using classification of machine learning techniques (i.e., support vector machine, decision tree, naive Bayes, and random forest), and predict the probability of cholangiocarcinoma. The proposed framework can generate the prediction model providing the sensitivity 75%, specificity 83.41%, and accuracy 83.34%. CanWiser also provides the personalized recommendation for patients to reduce the risk of cholangiocarcinoma. Moreover, our proposed framework can adaptively learn and generate the models, which can fit for the new dataset.