Two birds with one stone: Classifying positive and unlabeled examples on uncertain data streams

Frequent itemsets (FIs) mining from uncertain database is a very popular research area nowadays. Many algorithms have been proposed to mine FI from uncertain database. But in typical FI mining process, all the FIs have to be mined individually, which needs a huge memory. Four trees are proposed in this paper which are: (i) maximal frequent itemset from uncertain database (MFU) tree which contains only the maximal frequent itemsets generated from uncertain database, (ii) closed frequent itemset from uncertain database (CFU) tree which contains only closed frequent itemsets generated from uncertain database, (iii) maximal frequent itemset from uncertain data stream (MFUS) tree which contains maximal frequent itemsets generated from uncertain data stream and (iv) closed frequent itemset from uncertain data stream (CFUS) tree which contains closed frequent itemsets generated from uncertain data stream. Experimental results are also presented which show that maximal and closed frequent itemsets mining requires less time and memory than typical frequent itemsets mining.

A bagging SVM to learn from positive and unlabeled examples

Uncertainty is inherent in data streams, and present new challenges to data streams mining. For continuous arriving and large size of data streams, modeling sequences of uncertain time series data streams require significantly more space. Therefore, it is important to construct compressed representation for storing uncertain time series data. Based on granules, sequential sketches are created to store hash-compressed granules. And based on sliding windows, a sketch update strategy is given to store most resent granules. As the sequential sketches may be saturated with the increasing of data streams, this paper presents an optimization strategy to delete the absolute sparse patterns. Based on the sequential sketches, a sequential pattern mining algorithm is proposed for mining uncertain data streams. The experimental results illustrate the effectiveness of the pattern mining algorithm.

Unlabeled training examples are readily available in many applications, but labeled examples are fairly expensive to obtain. For instance, in our previous works on classification of peer-to-peer (P2P) Internet traffics, we observed that only about 25% of examples can be labeled as “P2P”or “NonP2P” using a port-based heuristic rule. We also expect that even fewer examples can be labeled in the future as more and more P2P applications use dynamic ports. This fact motivates us to investigate the techniques which enhance the accuracy of P2P traffic classification by exploiting the unlabeled examples. In addition, the Internet data flows dynamically in large volumes (streaming data). In P2P applications, new communities of peers often join and old communities of peers often leave, requiring the classifiers to be capable of updating the model incrementally, and dealing with concept drift. Based on these requirements, this paper proposes an incremental Tri-Training (iTT) algorithm. We tested our approach on a real data stream with 7.2 Mega labeled examples and 20.4 Mega unlabeled examples. The results show that iTT algorithm can enhance accuracy of P2P traffic classification by exploiting unlabeled examples. In addition, it can effectively deal with dynamic nature of streaming data to detect the changes in communities of peers. We extracted attributes only from the IP layer, eliminating the privacy concern associated with the techniques that use deep packet inspection.

Positive and unlabeled learning (PU learning) is a significant binary classification task in machine learning; it focuses on training accurate classifiers using positive data and unlabeled data. Most of the works in this area are based on a two-step strategy: the first step is to identify reliable negative examples from unlabeled examples, and the second step is to construct the classifiers based on the positive examples and the identified reliable negative examples using supervised learning methods. However, these methods always underutilize the remaining unlabeled data, which limits the performance of PU learning. Furthermore, many methods require the iterative solution of the formulated quadratic programming problems to obtain the final classifier, resulting in a large computational cost. In this paper, we propose a new method called the absolute value inequality support vector machine, which applies the concept of eccentricity to select reliable negative examples from unlabeled data and then constructs a classifier based on the positive examples, the selected negative examples, and the remaining unlabeled data. In addition, we apply a hyperparameter optimization technique to automatically search and select the optimal parameter values in the proposed algorithm. Numerical experimental results on ten real-world datasets demonstrate that our method is better than the other three benchmark algorithms.

Long short-term memory self-adapting online random forests for evolving data stream regression

In evolving data stream, when its concept undergoes a change it is known as concept drift. Detecting concept drift and handling it is a challenging task in data stream mining. If an algorithm is not adapted to concept drift, then it directly affects its performance. A number of algorithms have been developed to handle concept drift, but they are not suited for both sudden concept drift and gradual concept drift. Thus, there is a demand for an algorithm that can react to both the types of concept drift as well as incur less computational cost. A new approach hybrid early drift detection method (HEDDM) has been proposed for drift detection, which works with an ensemble method to improve the performance.

Deceptive reviews detection has attracted significant attention from both business and research communities. However, due to the difficulty of human labeling needed for supervised learning, the problem remains to be highly challenging. This paper proposed a novel angle to the problem by modeling PU (positive unlabeled) learning. A semi-supervised model, called mixing population and individual property PU learning (MPIPUL), is proposed. Firstly, some reliable negative examples are identified from the unlabeled dataset. Secondly, some representative positive examples and negative examples are generated based on LDA (Latent Dirichlet Allocation). Thirdly, for the remaining unlabeled examples (we call them spy examples), which can not be explicitly identified as positive and negative, two similarity weights are assigned, by which the probability of a spy example belonging to the positive class and the negative class are displayed. Finally, spy examples and their similarity weights are incorporated into SVM (Support Vector Machine) to build an accurate classifier. Experiments on gold-standard dataset demonstrate the effectiveness of MPIPUL which outperforms the state-of-the-art baselines.

Outlier detection on data streams is an important task in data mining. The challenges become even larger when considering uncertain data. This paper studies the problem of outlier detection on uncertain data streams. We propose Continuous Uncertain Outlier Detection (CUOD), which can quickly determine the nature of the uncertain elements by pruning to improve the efficiency. Furthermore, we propose a pruning approach — Probability Pruning for Continuous Uncertain Outlier Detection (PCUOD) to reduce the detection cost. It is an estimated outlier probability method which can effectively reduce the amount of calculations. The cost of PCUOD incremental algorithm can satisfy the demand of uncertain data streams. Finally, a new method for parameter variable queries to CUOD is proposed, enabling the concurrent execution of different queries. To the best of our knowledge, this paper is the first work to perform outlier detection on uncertain data streams which can handle parameter variable queries simultaneously. Our methods are verified using both real data and synthetic data. The results show that they are able to reduce the required storage and running time.

Multi-domain spoken dialog system should be able to detect more than one domain from a user’s utterance. However, it is difficult to train an accurate binary classifier of a domain based on only positive and unlabeled examples. This paper improves hierarchical clustering algorithm to automatically identify reliable negative examples among unlabeled examples. This paper also verifies three linkage criteria that measure the distance between two clusters. In experiments, the proposed method resulted in the highest gain of F1 score compared to the existing methods.

Processing data streams gained much importance in recent years. Standard machine learning algorithms do not cope well with non-stationary streaming data, where decision models evolve and generate so-called concept drift. Online adaptive algorithms emerged to solve these issues. They learn incrementally and generally require explicit forgetting mechanisms to adapt to concept drift. In this paper, we propose the application of Kalman filtering to handle evolving data streams. This novel approach addresses data stream mining and concept drift management challenges from a new perspective, directly modelling a representation suitable for the data streams. First, we study a Kalman filter based learning a pproach and investigate its integration into the Naïve Bayes algorithm, namely KalmanNB. Additionally, we propose the Hoeffding Kalman Tree, a combination of the Hoeffding Tree with KalmanNB. Empirical results demonstrate that the Kalman filter based approach inherently manages concept drifts, and it adapts to the emerging concept more rapidly than the state-of-the-art algorithms. Moreover, it is an accurate and robust approach and requires less storage while still being faster.

For generalization to be functional, it must occur with a precision that results in acquired responses occurring under appropriate, nontrained conditions, and acquired responses not occurring under inappropriate, nontrained conditions. This study examines the effect of differing types of negative teaching examples on the precision with which generalized grocery item selection is learned. Within a split-multiple baseline design, six young adults identified as mildly, moderately, or severely mentally retarded were trained to select or to reject grocery items using picture cards as cues. The dependent variables were correct selection of 10 trained “positive” grocery items and the correct rejection of 20 nontrained “negative” grocery items in a nontrained grocery store. Participants were trained in a grocery store to select 10 positive grocery examples matching their picture cards and to reject either (a) a set of negative examples that were maximally different from the positive examples, or (b) a set of negative examples that were minimally different from the positive examples. Both training sets resulted in participants correctly selecting the 10 positive items in a nontrained store. Training with the “minimally different” negative examples was functionally related to improved rejection of nontrained negative items in the nontrained store. The implications of teaching with minimally different, negative examples are discussed.

Supervised learning methods have been recently exploited to learn gene regulatory networks from gene expression data. They consist of building a binary classifier from feature vectors composed by expression levels of a set of known regulatory connections, available in public databases (eg. RegulonDB, TRRD, Transfac, IPA), and using such a classifier to predict new unknown connections. The input to a binary supervised classifier consists normally of positive and negative examples, but usually the only available information are a partial set of gene regulations, i.e. positive examples, and unlabeled data which could include both positive and negative examples. A fundamental challenge is the choice of negative examples from such unlabeled data to make the classifier able to learn from data. We exploit the known topology of a gene network to select such negative examples and show whether such an assumption benefits the performance of a classifier.

Amongst all the methods of machine learning, learning from examples is considered as a key to acquire knowledge automatically. Learning from examples is to obtain a general cover through induction from a given set of positive and negative examples of a concept, which may describe all the positive examples, but reject all the negative examples of that concept. Extension matrix theory, a branch of learning from examples, presents the training examples as two matrices which consist of positive and negative examples respectively, and then finds a path satisfied with a positive example against the background of negative examples. Obviously, such a path is a conjunction of conditions that is satisfied by the positive example. However, most of all of pattern recognition problems, such as handwritten Chinese character recognition, have an overlay area. It is necessary to solve the problem of nonlinear boundaries among different classes when machine learning is applied to acquire the recognition rules. An advanced extension matrix algorithm is proposed in this paper, in which a heuristic search based on the average entropy is used to get the approximate solutions of the minimal complex. A potential function is used to estimate the probability density function of the overlay area between positive and negative examples, so that the nonlinear interfaces of the interclass areas may be obtained. This algorithm is applied to supervised learning for Chinese character recognition to acquire recognition rules, and the implementations upon 751,000 loosely-constrained handwritten Chinese characters indicate that these methodologies can be applied to a practical recognition system with promising results.

Learning in nonstationary environments, also called learning concept drift, has been receiving increasing attention due to increasingly large number of applications that generate data with drifting distributions. These applications are usually associated with streaming data, either online or in batches, and concept drift algorithms are trained to detect and track the drifting concepts. While concept drift itself is a significantly more complex problem than the traditional machine learning paradigm of data coming from a fixed distribution, the problem is further complicated when obtaining labeled data is expensive, and training must rely, in part, on unlabelled data. Independently from concept drift research, semi-supervised approaches have been developed for learning from (limited) labeled and (abundant) unlabeled data; however, such approaches have been largely absent in concept drift literature. In this contribution, we describe an ensemble of classifiers based approach that takes advantage of both labeled and unlabeled data in addressing concept drift: available labeled data are used to generate classifiers, whose voting weights are determined based on the distances between Gaussian mixture model components trained on both labeled and unlabeled data in a drifting environment.