Estimating classification accuracy in positive-unlabeled learning: characterization and correction strategies

Supervised learning methods have been recently exploited to learn gene regulatory networks from gene expression data. They consist of building a binary classifier from feature vectors composed by expression levels of a set of known regulatory connections, available in public databases (eg. RegulonDB, TRRD, Transfac, IPA), and using such a classifier to predict new unknown connections. The input to a binary supervised classifier consists normally of positive and negative examples, but usually the only available information are a partial set of gene regulations, i.e. positive examples, and unlabeled data which could include both positive and negative examples. A fundamental challenge is the choice of negative examples from such unlabeled data to make the classifier able to learn from data. We exploit the known topology of a gene network to select such negative examples and show whether such an assumption benefits the performance of a classifier.

Positive-unlabeled learning for open set domain adaptation

For generalization to be functional, it must occur with a precision that results in acquired responses occurring under appropriate, nontrained conditions, and acquired responses not occurring under inappropriate, nontrained conditions. This study examines the effect of differing types of negative teaching examples on the precision with which generalized grocery item selection is learned. Within a split-multiple baseline design, six young adults identified as mildly, moderately, or severely mentally retarded were trained to select or to reject grocery items using picture cards as cues. The dependent variables were correct selection of 10 trained “positive” grocery items and the correct rejection of 20 nontrained “negative” grocery items in a nontrained grocery store. Participants were trained in a grocery store to select 10 positive grocery examples matching their picture cards and to reject either (a) a set of negative examples that were maximally different from the positive examples, or (b) a set of negative examples that were minimally different from the positive examples. Both training sets resulted in participants correctly selecting the 10 positive items in a nontrained store. Training with the “minimally different” negative examples was functionally related to improved rejection of nontrained negative items in the nontrained store. The implications of teaching with minimally different, negative examples are discussed.

Deceptive reviews detection has attracted significant attention from both business and research communities. However, due to the difficulty of human labeling needed for supervised learning, the problem remains to be highly challenging. This paper proposed a novel angle to the problem by modeling PU (positive unlabeled) learning. A semi-supervised model, called mixing population and individual property PU learning (MPIPUL), is proposed. Firstly, some reliable negative examples are identified from the unlabeled dataset. Secondly, some representative positive examples and negative examples are generated based on LDA (Latent Dirichlet Allocation). Thirdly, for the remaining unlabeled examples (we call them spy examples), which can not be explicitly identified as positive and negative, two similarity weights are assigned, by which the probability of a spy example belonging to the positive class and the negative class are displayed. Finally, spy examples and their similarity weights are incorporated into SVM (Support Vector Machine) to build an accurate classifier. Experiments on gold-standard dataset demonstrate the effectiveness of MPIPUL which outperforms the state-of-the-art baselines.

Amongst all the methods of machine learning, learning from examples is considered as a key to acquire knowledge automatically. Learning from examples is to obtain a general cover through induction from a given set of positive and negative examples of a concept, which may describe all the positive examples, but reject all the negative examples of that concept. Extension matrix theory, a branch of learning from examples, presents the training examples as two matrices which consist of positive and negative examples respectively, and then finds a path satisfied with a positive example against the background of negative examples. Obviously, such a path is a conjunction of conditions that is satisfied by the positive example. However, most of all of pattern recognition problems, such as handwritten Chinese character recognition, have an overlay area. It is necessary to solve the problem of nonlinear boundaries among different classes when machine learning is applied to acquire the recognition rules. An advanced extension matrix algorithm is proposed in this paper, in which a heuristic search based on the average entropy is used to get the approximate solutions of the minimal complex. A potential function is used to estimate the probability density function of the overlay area between positive and negative examples, so that the nonlinear interfaces of the interclass areas may be obtained. This algorithm is applied to supervised learning for Chinese character recognition to acquire recognition rules, and the implementations upon 751,000 loosely-constrained handwritten Chinese characters indicate that these methodologies can be applied to a practical recognition system with promising results.

Positive-unlabeled (PU) learning deals with the binary classification problem when only positive (P) and unlabeled (U) data are available, without negative (N) data. Existing PU methods perform well on the balanced dataset. However, in real applications such as financial fraud detection or medical diagnosis, data are always imbalanced. It remains unclear whether existing PU methods can perform well on imbalanced data. In this paper, we explore this problem and propose a general learning objective for PU learning targeting specially at imbalanced data. By this general learning objective, state-of-the-art PU methods based on optimizing a consistent risk can be adapted to conquer the imbalance. We theoretically show that in expectation, optimizing our learning objective is equivalent to learning a classifier on the oversampled balanced data with both P and N data available, and further provide an estimation error bound. Finally, experimental results validate the effectiveness of our proposal compared to state-of-the-art PU methods.

This paper studies the problem of Positive Unlabeled learning (PU learning), where positive and unlabeled examples are used for training. Naive Bayes (NB) and Tree Augmented Naive Bayes (TAN) have been extended to PU learning algorithms (PNB and PTAN). However, they require user-specified parameter, which is difficult for the user to provide in practice. We estimate this parameter following [2] by taking the selected completely at random assumption and reformulate these two algorithms with this assumption. Furthermore, based on supervised algorithms Averaged One-Dependence Estimators (AODE), Hidden Naive Bayes (HNB) and Full Bayesian network Classifier (FBC), we extend these algorithms to PU learning algorithms (PAODE, PHNB and PFBC respectively). Experimental results on 20 UCI datasets show that the performance of the Bayesian algorithms for PU learning are comparable to corresponding supervised ones in most cases. Additionally, PNB and PFBC are more robust against unlabeled data, and PFBC generally performs the best.

Positive-unlabeled (PU) learning deals with the circumstances where only a portion of positive instances are labeled, while the rest and all negative instances are unlabeled, and due to this confusion, the class prior can not be directly available. Existing PU learning methods usually estimate the class prior by training a nontraditional probabilistic classifier, which is prone to give an overestimation. Moreover, these methods learn the decision boundary by optimizing the minimum margin, which is not suitable in PU learning due to its sensitivity to label noise. In this paper, we enhance PU learning methods from the above two aspects. More specifically, we first explicitly learn a transformation from unlabeled data to positive data by entropy regularized optimal transport to achieve a much more precise estimation for class prior. Then we switch to optimizing the margin distribution, rather than the minimum margin, to obtain a label noise insensitive classifier. Extensive empirical studies on both synthetic and real-world data sets demonstrate the superiority of our proposed method.

A bagging SVM to learn from positive and unlabeled examples

Two studies were conducted to determine the relationship between systematic stimulus variation and the formation of misconceptions. Experiment 1 compared five levels of stimulus variation between positive and negative examples, and Experiment 2 compared two levels of stimulus variation among positive examples. Misconceptions were fewer with minimal variation between positive and negative examples and maximal relevant variation among positive examples.

Two birds with one stone: Classifying positive and unlabeled examples on uncertain data streams

Epilepsy is a chronic disorder of the brain. Intracranial electroencephalogram (iEEG) recorded from cortex is the most popular measurement for not only the diagnosis of epilepsy, but also the focus localization that is crucial for the surgery. In recent years, the machine learning methods have been rapidly developed and applied successfully to various real world problems. Given sufficient number of samples, the powerful deep learning methods can achieve high performance for epileptic focus localization. However, it is a challenging task to obtain large amount of labeled iEEG regarding focal/non-focal channels, since the annotations must be performed by multiple clinical experts through visual judgment on the long term iEEG signals. In order to reduce the necessary number of labeled training samples, we introduce the positive unlabeled (PU) learning method for classification of focal and non-focal epileptic iEEG signals. The proposed method enables us to learn a binary classifier by using small amount of labeled data containing only one class (i.e., focal signals) and unlabeled data containing two classes (i.e., focal and non-focal signals), which greatly reduces the workload of clinical experts for annotations. Experimental results on Bern dataset and iEEG recorded from Juntendo University Hospital demonstrate the effectiveness of our method.

Research has been devoted in the past few years to relevance feedback as an effective solution to improve performance of content-based image retrieval (CBIR). In this paper, we propose a new feedback approach with progressive learning capability combined with a novel method for the feature subspace extraction. The proposed approach is based on a Bayesian classifier and treats positive and negative feedback examples with different strategies. Positive examples are used to estimate a Gaussian distribution that represents the desired images for a given query; while the negative examples are used to modify the ranking of the retrieved candidates. In addition, feature subspace is extracted and updated during the feedback process using a principal component analysis (PCA) technique and based on user's feedback. That is, in addition to reducing the dimensionality of feature spaces, a proper subspace for each type of features is obtained in the feedback process to further improve the retrieval accuracy. Experiments demonstrate that the proposed method increases the retrieval speed, reduces the required memory and improves the retrieval accuracy significantly.

Automatic concept learning from large scale imbalanced data sets is a key issue in video semantic analysis and retrieval, which means the number of negative examples is far more than that of positive examples for each concept in the training data. The existing methods adopt generally under-sampling for the majority negative examples or over-sampling for the minority positive examples to balance the class distribution on training data. The main drawbacks of these methods are: (1) As a key factor that affects greatly the performance, in most existing methods, the degree of re-sampling needs to be pre-fixed, which is not generally the optimal choice; (2) Many useful negative samples may be discarded in under-sampling. In addition, some works only focus on the improvement of the computational speed, rather than the accuracy. To address the above issues, we propose a new approach and algorithm named AdaOUBoost (Adaptive Over-sampling and Under-sampling Boost). The novelty of AdaOUBoost mainly lies in: adaptively over-sample the minority positive examples and under-sample the majority negative examples to form different sub-classifiers. And combine these sub-classifiers according to their accuracy to create a strong classifier, which aims to use fully the whole training data and improve the performance of the class-imbalance learning classifier. In AdaOUBoost, first, our clustering-based under-sampling method is employed to divide the majority negative examples into some disjoint subsets. Then, for each subset of negative examples, we utilize the borderline-SMOTE (synthetic minority over-sampling technique) algorithm to over-sample the positive examples with different size, train each sub-classifier using each of them, and get the classifier by fusing these sub-classifiers with different weights. Finally, we combine these classifiers in each subset of negative examples to create a strong classifier. We compare the performance between AdaOUBoost and the state-of-the-art methods on TRECVID 2008 benchmark with all 20 concepts, and the results show the AdaOUBoost can achieve the superior performance in large scale imbalanced data sets.

C-PUGP: A cluster-based positive unlabeled learning method for disease gene prediction and prioritization