A novel progressive multi-label classifier for class-incremental data

In previous work, we proposed a neural network approach to estimate the blocking probability of optical networks with fixed routing. The neural network was implemented by the extreme learning machine (ELM) framework, in which the training inputs were the optical network parameters, and the output was the overall blocking probability. The numerical results showed that the neural-network-based estimation was accurate and thousands of times faster than a computer simulation. In this paper, we apply the neural network approach to optical circuit switching (OCS) networks with fixed-alternate routing and improve the training method by using an enhancement of ELM framework. Unlike the previous ELM framework, the enhancement of ELM framework provides a random-search-based selection phase for the hidden nodes during the training step. As a result, similar performance can be achieved using fewer hidden nodes than the previous ELM framework. The numerical results show that the new enhancement of ELM training algorithm provides more accurate blocking probability estimates while reducing the required number of hidden nodes by a third compared with the previous ELM training algorithm. Furthermore, for some light traffic loading situations, our new training algorithm is hundreds of times more accurate than the existing well-known analytical approximation method.

The presented paper describes our model for the WISE 2014 challenge multi-label classification task. The goal of the challenge was to implement a multi-label text classification model which maximizes the mean $$F_1$$ score on a private test data. The described method involves a binary relevance scheme with linear classifiers trained using stochastic gradient descent. A novel method for determining the values of classifiers' meta-parameters was developed. In addition, our solution employs the semi-supervised learning which significantly improves the evaluation score. The presented solution won the third place in the challenge. The results are discussed and the supervised and semi-supervised approaches are compared.

A novel random fast multi-label deep forest classification algorithm

Multi-label classification methods aim to a class of application problems where each individual contains a single instance while associates with a set of labels simultaneously. In this paper, we formulate a novel multi-label classification method based on extreme learning machine framework, named ELM-MC algorithm. The essence of ELM-MC algorithm is to convert the multi-label classification problem into some single-label classifications, and fully considers the relationship among different labels. After the classification of one label, the associations with next label are applied to update the learning parameters in ELM-MC algorithm. In addition, we design a backup pool for the hidden nodes. It can help to select relatively suitable hidden nodes to the corresponding label classification case. In the simulation part, six famous databases are applied to demonstrate the satisfied classification accuracy of the proposed method.

Multi-label classification aims to associate multiple labels to a given data/object instance to better describe them. Multi-label data sets are common in a lot of emerging application areas like: Text/Multimedia classification, Bio-Informatics, Medical image annotations and Computer Vision to name a few. There is a growing interest in efficient and accurate multi-label classification. There are two major approaches to perform multi-label classification (i) problem transformation methods and (ii) algorithm adaptation methods. In algorithm adaptation, the traditional classification algorithms are modified to handle multi-label data sets. One classification algorithm which is often modified to do multi-label classification is k- nearest neighbor (kNN). k-nearest neighbor is popular due to its simplicity, easy to implement and seamlessly adaptability. Despite its merits it has several drawbacks like: sensitivity of noisy data, missing values and outliers; feature scaling and often becoming inaccurate for large overlapping solution space. In this paper, a modification to kNN method is suggested for multi-label classification with three improvement strategies (i) selection of local example w.r.t. unknown example – the motivation for this comes from the fact that local and relevant space is vital for the improvement in multi-label classification; (ii) Splitting the input space into multiple sub-spaces for optimal label estimation – the motivation is to estimate label accurately in the presence of noisy labels; And (iii) selection of labels using Mean Average Precision (MAP) estimates – here our motivation is to utilize the training data effectively to maximize the hidden distribution and optimal parameters for the method. The proposed method is implemented and compared with state-of-the-art approaches based on kNN or similar approaches that effectively select and optimize relevant spaces for multi-label classification. Evaluation based on multiple metrics like Hamming loss, Precision/Recall and F-measure are used for evaluation. The suggested approach performed much better than the state-of-the-art on the datasets with strong label cardinalities.

Due to an ever-increasing demand for analyzing the large volumes of information issuing from high-speed data streams, multi-label stream classification is replacing the traditional offline multi-label classification system and has thus become a focal point in recent years. In this paper, we propose a new algorithm for multi-label stream classification that relies on an Online Dual Memory (ODM). It uses a short-term memory in order to adapt quickly to changes within the data stream, and employs a long-term memory for recurring concepts. Furthermore, via a modification of Learning Vector Quantization, ODM is not only capable of classifying efficiently but also of maintaining a low computational complexity. Experiments were conducted so as to establish a comprehensive overview of its performance, which was achieved by comparing four methods on two different kinds of datasets (nine stationary datasets and four datasets with concept drift). Our results confirm the high accuracy and low time complexity of ODM, as well as its ability to cope with concept drift.

Most existing work on multi-label learning focused on supervised learning which requires manual annotation samples that is labor-intensive, time-consuming and costly. To address such a problem, we present a novel method that incorporates active learning into the semi-supervised learning for multi-label image classification. What’s more, aiming at the curse of dimensionality existing in high-dimensional data, we explore a dimensionality reduction technique with non-negative sparseness constraint to extract a group of features that can completely describe the data and hence make the learning model more efficiently. Experimental results on common data sets validate that the proposed algorithm is relatively effective to improve the performance of the learner in multi-label classification, and the obtained learner is with reliability and robustness after data dimensionality using NNS-DR (Non-Negative Sparseness for Dimensionality Reduction).

A nonsupervised learning algorithm for multicategory pattern classification is presented, and its steady-state behavior is investigated by means of analysis and simulation. The algorithm assigns each pattern in an input sequence x(t) to one of the M possible categories by comparing the M values wi(t)' x(t), i = 1,2,...,M, where wi(t)are variable weight vectors adjusted according to a differential-equation type of rule. The algorithm minimizes a certain form of criterion function. Analytical techniques are developed for finding the steady-state solutions of the differential equation and for determining the stability of each solution found. By applying the techniques, the steady-state behavior of the algorithm to three types of patterns is analyzed in detail. It is shown that although the algorithm does not necessarily yield a unique solution every solution yielded is associated with a reasonable Classification such that one cluster or some adjacent clusters in the pattern distribution correspond to one category. Even if the total number of clusters in the distribution is smaller than M, no cluster is divided between two categories. Converging to the origin, unnecessary weight vectors vanish in the steady state. These results are verified by some digital computer simulations.

BPSO-Adaboost-KNN ensemble learning algorithm for multi-class imbalanced data classification

Incremental learning has been used extensively for data stream classification. Most attention on the data stream classification paid on non-evolutionary methods. In this paper, we introduce new incremental learning algorithms based on harmony search. We first propose a new classification algorithm for the classification of batch data called harmony-based classifier and then give its incremental version for classification of data streams called incremental harmony-based classifier. Finally, we improve it to reduce its computational overhead in absence of drifts and increase its robustness in presence of noise. This improved version is called improved incremental harmony-based classifier. The proposed methods are evaluated on some real world and synthetic data sets. Experimental results show that the proposed batch classifier outperforms some batch classifiers and also the proposed incremental methods can effectively address the issues usually encountered in the data stream environments. Improved incremental harmony-based classifier has significantly better speed and accuracy on capturing concept drifts than the non-incremental harmony based method and its accuracy is comparable to non-evolutionary algorithms. The experimental results also show the robustness of improved incremental harmony-based classifier.

Cost-sensitive continuous ensemble kernel learning for imbalanced data streams with concept drift

Multi-label classification (MLC) tasks are encountered more and more frequently in machine learning applications. While MLC methods exist for the classical batch setting, only a few methods are available for streaming setting. In this paper, we propose a new methodology for MLC via multi-target regression in a streaming setting. Moreover, we develop a streaming multi-target regressor iSOUP-Tree that uses this approach. We experimentally compare two variants of the iSOUP-Tree method (building regression and model trees), as well as ensembles of iSOUP-Trees with state-of-the-art tree and ensemble methods for MLC on data streams. We evaluate these methods on a variety of measures of predictive performance (appropriate for the MLC task). The ensembles of iSOUP-Trees perform significantly better on some of these measures, especially the ones based on label ranking, and are not significantly worse than the competitors on any of the remaining measures. We identify the thresholding problem for the task of MLC on data streams as a key issue that needs to be addressed in order to obtain even better results in terms of predictive performance.

Classification is a widely researched area in the machine learning and fuzzy communities with several approaches proposed by both communities. Some of the most relevant rule-based approaches from the machine learning community might include decision trees and rule inducers. The fuzzy community has also proposed many rule-based approaches, such as fuzzy decision trees and genetic fuzzy systems. This paper aims at comparing the models generated by rule-based methods for classification from both communities in terms of accuracy, and the induced rule set in terms of the syntactic complexity, taking into account the number of rules and average number of conjunctions in those rules. In general, models with lower syntactic complexity also show better interpretability, which is an important issue in knowledge acquisition. Results, using 10 datasets, a fuzzy C4.5 algorithm and two classic machine learning algorithms (C4.5 and PART), show that the fuzzy approach is able to produce lower error rates. Regarding the syntactic complexity of the models, PART produces in most cases the simplest models, although learning from different sets of features selected by filters. However, these simple models do not necessarily show a low error rate. Nevertheless, the induced fuzzy models inherit, from the fuzzy logic, the embedded ability of processing uncertainty and imprecision, avoiding the creation of rules using unnatural divisions of the attributes as the classic algorithms might do.

The advances in computing software, hardware, connected devices and wireless communication infrastructure in recent years have led to the desire to work with streaming data sources. Yet the number of techniques, approaches and algorithms which can work with data from a streaming source is still very limited, compared with batched data. Although data mining techniques have been a well-studied topic of knowledge discovery for decades, many unique properties as well as challenges in learning from a data stream have not been considered properly due to the actual presence of and the real needs to mine information from streaming data sources. This thesis aims to contribute to the knowledge by developing a rule-based algorithm to specifically learn classification rules from data streams, with the learned rules are expressive so that a human user can easily interpret the concept and rationale behind the predictions of the created model. There are two main structures to represent a classification model; the ‘tree-based’ structure and the ‘rule-based’ structure. Even though both forms of representation are popular and well-known in traditional data mining, they are different when it comes to interpretability and quality of models in certain circumstances. The first part of this thesis analyses background work and relevant topics in learning classification rules from data streams. This study provides information about the essential requirements to produce high quality classification rules from data streams and how many systems, algorithms and techniques related to learn the classification of a static dataset are not applicable in a streaming environment. The second part of the thesis investigates at a new technique to improve the efficiency and accuracy in learning heuristics from numeric features from a streaming data source. The computational cost is one of the important factors to be considered for an effective and practical learning algorithm/system because of the needs to learn from continuous arrivals of data examples sequentially and discard the seen data examples. If the computing cost is too expensive, then one may not be able to keep pace with the arrival of high velocity and possibly unbound data streams. The proposed technique was first discussed in the context of the use of Gaussian distribution as heuristics for building rule terms on numeric features. Secondly, empirical evaluation shows the successful integration of the proposed technique into an existing rule-based algorithm for the data stream, eRules. Continuing on the topic of a rule-based algorithm for classification data streams, the use of Hoeffding’s Inequality addresses another problem in learning from a data stream, namely how much data should be seen from a data stream before starting learning and how to keep the model updated over time. By incorporating the theory from Hoeffding’s Inequality, this study presents the Hoeffding Rules algorithm, which can induce modular rules directly from a streaming data source with dynamic window sizes throughout the learning period to ensure the efficiency and robustness towards the concept drifts. Concept drift is another unique challenge in mining data streams which the underlying concept of the data can change either gradually or abruptly over time and the learner should adapt to these changes as quickly as possible. This research focuses on the development of a rule-based algorithm, Hoeffding Rules, for data stream which considers streaming environments as primary data sources and addresses several unique challenges in learning rules from data streams such as concept drifts and computational efficiency. This knowledge facilitates the need and the importance of an interpretable machine learning model; applying new studies to improve the ability to mine useful insights from potentially high velocity, high volume and unbounded data streams. More broadly, this research complements the study in learning classification rules from data streams to address some of the unique challenges in data streams compared with conventional batch data, with the knowledge necessary to systematically and effectively learn expressive and modular classification rules from data streams.

To solve the problem that traditional multi-label support vector machine (SVM) classification algorithm adopting nonlinear kernel has been severely restricted from being used on large-scale data sets, we propose fast multi-label low-rank-linearized SVM classification algorithm based on approximate extreme points (AEML-LLSVM). First, it adopts the approximate extreme points' method to obtain representative sets from the training data set. Then, the approximate extreme points' low-rank-linearized SVM (AELLSVM) is trained on the representative sets. The AELLSVM integrates the advantages of approximate extreme points' method and LLSVM. Experimental results on three large-scale multi-label data sets have proven that the training and the testing speed of AEML-LLSVM classification algorithm are greatly improved under the premise that its classification performance is similar to that of ML-LIBSVM classification algorithm and superior to that of other fast multi-label SVM classification algorithms.