Cooperative Evolution Multiclass Support Matrix Machines

Neuroevolution is a powerful and general technique for evolving the structure and weights of artificial neural networks. Though neuroevolutionary approaches such as NeuroEvolution of Augmenting Topologies (NEAT) have been successfully applied to various problems including classification, regression, and reinforcement learning problems, little work has explored application of these techniques to larger-scale multiclass classification problems. In this paper, NEAT is evaluated in several multiclass classification problems, and then extended via two ensemble approaches: One-vs-All and One-vs-One. These approaches decompose multiclass classification problems into a set of binary classification problems, in which each binary problem is solved by an instance of NEAT. These ensemble models exhibit reduced variance and increasingly superior accuracy as the number of classes increases. Additionally, higher accuracy is achieved early in training, even when artificially constrained for the sake of fair comparison with standard NEAT. However, because the approach can be trivially distributed, it can be applied quickly at large scale to solve real problems. In fact, these approaches are incorporated into DarwinTM, an enterprise automatic machine learning solution that also incorporates various other algorithmic enhancements to NEAT. The resulting complete system has proven robust to a wide variety of client datasets.

Improved multiclass feature selection via list combination

We consider the classification problem when the input features are represented as matrices rather than vectors. To preserve the intrinsic structures for classification, a successful method is the Support Matrix Machine (SMM) in [19], which optimizes an objective function with a hinge loss plus a so-called spectral elastic net penalty. However, the issues of extending SMM to multicategory classification still remain. Moreover, in practice, it is common to see the training data contaminated by outlying observations, which can affect the robustness of existing matrix classification methods. In this paper, we address these issues by introducing a robust angle-based classifier, which boils down binary and multicategory problems to a unified framework. Benefitting from the use of truncated hinge loss functions, the proposed classifier achieves certain robustness to outliers. The underlying optimization model becomes nonconvex, but admits a natural DC (difference of two convex functions) representation. We develop a new and efficient algorithm by incorporating the DC algorithm and primal-dual first-order methods together. The proposed DC algorithm adaptively chooses the accuracy of the subproblem at each iteration while guaranteeing the overall convergence of the algorithm. The use of primal-dual methods removes a natural complexity of the linear operator in the subproblems and enables us to use the proximal operator of the objective functions, and matrix-vector operations. This advantage allows us to solve large-scale problems efficiently. Theoretical and numerical results indicate that for problems with potential outliers, our method can be highly competitive among existing methods.

All-in-one multicategory least squares nonparallel hyperplanes support vector machine

Imbalanced data are known to be notably difficult to deal with, as it needs a thorough understanding of the data to know how it should be done. However, it occurs in many fields, especially in finance, like banking. Many problems that use financial data can be summarized as binary problems. Some of them are fatal if not identified correctly. This research aims to find how to utilize machine learning models to deal with imbalanced data, specifically one that contains binary classification problems. In this paper, we use imbalanced insurance and credit card datasets. The research is conducted by starting from doing feature selection in the datasets by removing irrelevant columns, followed by using SMOTE algorithm variants (K-Means SMOTE and Borderline SMOTE) and pure SMOTE algorithm as the oversampling methods, and Near-Miss and All-KNN for undersampling methods. The algorithms are implemented by using scikit libraries. Lastly, PCA is used for dimensionality reduction and Logistic Regression as the machine learning model with cross-validations for deciding the best hyperparameter. The procedure produces five different Logistic Regression models that differ in how it handles the imbalances, which will be compared. The result shows that the oversampling methods work better than undersampling methods, with K-Means SMOTE and Borderline SMOTE performing better than the pure SMOTE, meaning that machine learning can be used as a solution to deal with binary classification problems in imbalanced financial data.

Internet and social media networks have become a defining factor for modern socialization, communication, and collaboration. At the same time, they have also become a massive platform for antisocial behavior such as cyberbullying, hate speech, and aggressive behavior, among others. This paper examines the automatic detection and classification of cyberbullying incidents in social networks. Most of the current work focuses on the detection of cyberbullying as a binary problem. However, some incidents are more severe than others and demand immediate attention and intervention from the stakeholders such as parents, government, and other relevant agencies. This paper presents the detection and classification of cyberbullying events based on their severity using various deep learning techniques – CNN, recurrent neural networks (RNN-BiLSTM), and HLSTM (HAN-BiLSTM). We curated three datasets from different social networks of varying balances to study the impact of class balance in a multi-class classification problem such as cyberbullying severity detection. Experimental results reveal higher efficiency of HLSTM and RNN-BiLSTM over CNN using all significant metrics such as k-score, MAP, MAR, and F1-Score. Similarly, results demonstrate that the deep learning techniques generally work well with moderately balanced datasets. The study reveals that deep learning techniques effectively detect severity in cyberbullying incidents, a multi-class classification problem.

Speech is among the most natural methods for us as human beings to express ourselves. Due to the relevance of emotions in today’s digital world of distant communication, their detection and analysis are crucial. Emotion recognition is extremely difficult since emotions are different for everyone. Speech emotion recognition is one area of application in which deep neural networks have excelled. Single learners have been used in the majority of the work done in this subject. We have developed a Speech Emotion Recognition (SER) system named SERNet that processes and classifies speech inputs to recognize emotions. As part of our research, we tried to explore emotional feelings in audio talks by looking at the acoustic characteristics of such recordings. This study proposes a novel approach using an ensemble of binary classifiers to simplify the multiclass classification problem into a binary classification problem, aiming to improve overall model performance. The binary classifiers are ensembled using a multilayer perceptron to obtain their final predictions on the multiclass classification problem. Utilizing a benchmark dataset designed specifically for the purpose of speech emotion recognition, the efficacy of this strategy has been proven. On the basis of the findings of the experiments, with an accuracy of 98.81%, this technique exceeds the most advanced models currently available.

In this paper, we address the problem of unbalanced training dataset for source camera identification, namely, there are fewer training examples for some camera models compared to other camera models.A new source camera identification approach is proposed to alleviate the influence of imbalanced training dataset.In the proposed approach, firstly, we treat source camera identification as a multi-class classification problem, and decompose it into binary classification problems.After decomposing, the problem of imbalanced training dataset for multiclass classification is transformed to the problem of imbalanced training dataset for binary classification.Then, we incorporate SMOTE and AdaBoost algorithms to construct SVM ensemble to address the issue of imbalanced training dataset for binary classification.A number of experiments show the proposed approach can deal with the imbalanced training dataset effectively.

Recognizing textual entailment comprises the task of determining semantic entailment relations between text fragments. A text fragment entails another text fragment if, from the meaning of the former, one can infer the meaning of the latter. If such relation is bidirectional, then we are in the presence of a paraphrase. Automatically recognizing textual entailment relations captures major semantic inference needs in several natural language processing (NLP) applications. As in many NLP tasks, textual entailment corpora for English abound, while the same is not true for more resource-scarce languages such as Portuguese. Exploiting what seems to be the only Portuguese corpus for textual entailment and paraphrases (the ASSIN corpus), in this paper, we address the task of automatically recognizing textual entailment (RTE) and paraphrases from text written in the Portuguese language, by employing supervised machine learning techniques. We employ lexical, syntactic and semantic features, and analyze the impact of using semantic-based approaches in the performance of the system. We then try to take advantage of the bi-dialect nature of ASSIN to compensate its limited size. With the same aim, we explore modeling the task of recognizing textual entailment and paraphrases as a binary classification problem by considering the bidirectional nature of paraphrases as entailment relationships. Addressing the task as a multi-class classification problem, we achieve results in line with the winner of the ASSIN Challenge. In addition, we conclude that semantic-based approaches are promising in this task, and that combining data from European and Brazilian Portuguese is less straightforward than it may initially seem. The binary classification modeling of the problem does not seem to bring advantages to the original multi-class model, despite the outstanding results obtained by the binary classifier for recognizing textual entailments.

Motor Imagery-based Brain-computer interface (MI-BCI) enables communications between the human brain and external machines, which is considered a valuable frontier technology. Motor imagery (MI) identification is a fundamental problem for these applications. However, because of the non-stationary and nonlinear properties of EEG motor imagery signals, it is very hard to get a high accuracy multi-class method. In this study, a multi-class classification method of MI EEG signals using wavelet packets and a one-dimensional convolutional neural network is presented. Data normalization and wavelet threshold denoising were performed for signal preprocessing. Then EEG signals were decomposed and reconstructed by wavelet packets transform. According to the features between different channels of EEG signal, a multilayer one-dimensional convolutional neural network model was adopted to get a multi-class classification model for MI. This model was validated on the public datasets BCI competition IV 2a. The result shows that the mean accuracy of this model is 93.05%. This study is beneficial to MI-BCI research applications in the future.

A multiclass classification problem can be reduced to a collection of binary problems using an error-correcting coding matrix that specifies the binary partitions of the classes. The final classifier is an ensemble of base classifiers learned on binary problems and its performance is affected by two major factors: the qualities of the base classifiers and the coding matrix. Previous studies either focus on one of these factors or consider two factors separately. In this paper, we propose a new multiclass boosting algorithm called AdaBoost.SIP that considers both two factors simultaneously. In this algorithm, informative patterns, which are shareable by different classes rather than only discriminative on specific single class, are generated at first. Then the binary partition preferred by each pattern is found by performing stage-wise functional gradient descent on a margin-based cost function. Finally, base classifiers and coding matrix are optimized simultaneously by maximizing the negative gradient of such cost function. The proposed algorithm is applied to scene and event recognition and experimental results show its effectiveness in multiclass classification.

Significant research has been done in the medical domain using machine learning and clinical data sets. Although there are many interesting and influential clinical research works in the fields of healthcare and health services using machine learning, there is a need to apply machine learning in the field of health human resource planning. This study uses physician billing data and machine learning to identify and classify family physicians with the goal of improving health human resource planning. This research is essential for policy makers because it is important to know the number of family physicians practicing in certain geographical regions for providing timely care. Additionally, this issue becomes particularly important when it comes to serving communities with fewer resources such as the rural areas of Northwestern Ontario, where family physicians need to work to their full scope of practice, provide more services than physicians working in urban areas, to meet the needs of patients. In this study, recursive feature elimination method is used to reduce the number of predictors for the classification problems. As the result of this process, the most important features include physician's rurality, full-time equivalent hours, age, and years of experience. Further, several machine learning models are used to solve binary and multi-class classification problems. Gradient boosting machine learning was the most accurate in predicting family physician practice, with a receiver operating characteristic value, ROC value, of 0.73 and 0.72 for binary and multi-class classification, respectively.

On Detecting and Classifying DGA Botnets and their Families

Electrocardiogram analysis using a combination of statistical, geometric, and nonlinear heart rate variability features

Bond rating is regarded as an important event for measuring financial risk of companies and for determining the investment returns of investors. As a result, it has been a popular research topic for researchers to predict companies' credit ratings by applying statistical and machine learning techniques. The statistical techniques, including multiple regression, multiple discriminant analysis (MDA), logistic models (LOGIT), and probit analysis, have been traditionally used in bond rating. However, one major drawback is that it should be based on strict assumptions. Such strict assumptions include linearity, normality, independence among predictor variables and pre-existing functional forms relating the criterion variablesand the predictor variables. Those strict assumptions of traditional statistics have limited their application to the real world. Machine learning techniques also used in bond rating prediction models include decision trees (DT), neural networks (NN), and Support Vector Machine (SVM). Especially, SVM is recognized as a new and promising classification and regression analysis method. SVM learns a separating hyperplane that can maximize the margin between two categories. SVM is simple enough to be analyzed mathematical, and leads to high performance in practical applications. SVM implements the structuralrisk minimization principle and searches to minimize an upper bound of the generalization error. In addition, the solution of SVM may be a global optimum and thus, overfitting is unlikely to occur with SVM. In addition, SVM does not require too many data sample for training since it builds prediction models by only using some representative sample near the boundaries called support vectors. A number of experimental researches have indicated that SVM has been successfully applied in a variety of pattern recognition fields. However, there are three major drawbacks that can be potential causes for degrading SVM's performance. First, SVM is originally proposed for solving binary-class classification problems. Methods for combining SVMs for multi-class classification such as One-Against-One, One-Against-All have been proposed, but they do not improve the performance in multi-class classification problem as much as SVM for binary-class classification. Second, approximation algorithms (e.g. decomposition methods, sequential minimal optimization algorithm) could be used for effective multi-class computation to reduce computation time, but it could deteriorate classification performance. Third, the difficulty in multi-class prediction problems is in data imbalance problem that can occur when the number of instances in one class greatly outnumbers the number of instances in the other class. Such data sets often cause a default classifier to be built due to skewed boundary and thus the reduction in the classification accuracy of such a classifier. SVM ensemble learning is one of machine learning methods to cope with the above drawbacks. Ensemble learning is a method for improving the performance of classification and prediction algorithms. AdaBoost is one of the widely used ensemble learning techniques. It constructs a composite classifier by sequentially training classifiers while increasing weight on the misclassified observations through iterations. The observations that are incorrectly predicted by previous classifiers are chosen more often than examples that are correctly predicted. Thus Boosting attempts to produce new classifiers that are better able to predict examples for which the current ensemble's performance is poor. In this way, it can reinforce the training of the misclassified observations of the minority class. This paper proposes a multiclass Geometric Mean-based Boosting (MGM-Boost) to resolve multiclass prediction problem. Since MGM-Boost introduces the notion of geometric mean into AdaBoost, it can perform learning process considering the geometric mean-based accuracy and errors of multiclass. This study applies MGM-Boost to the real-world bond rating case for Korean companies to examine the feasibility of MGM-Boost. 10-fold cross validations for threetimes with different random seeds are performed in order to ensure that the comparison among three different classifiers does not happen by chance. For each of 10-fold cross validation, the entire data set is first partitioned into tenequal-sized sets, and then each set is in turn used as the test set while the classifier trains on the other nine sets. That is, cross-validated folds have been tested independently of each algorithm. Through these steps, we have obtained the results for classifiers on each of the 30 experiments. In the comparison of arithmetic mean-based prediction accuracy between individual classifiers, MGM-Boost (52.95%) shows higher prediction accuracy than both AdaBoost (51.69%) and SVM (49.47%). MGM-Boost (28.12%) also shows the higher prediction accuracy than AdaBoost (24.65%) and SVM (15.42%)in terms of geometric mean-based prediction accuracy. T-test is used to examine whether the performance of each classifiers for 30 folds is significantly different. The results indicate that performance of MGM-Boost is significantly different from AdaBoost and SVM classifiers at 1% level. These results mean that MGM-Boost can provide robust and stable solutions to multi-classproblems such as bond rating.