Cost-sensitive active learning through statistical methods

Cost-sensitive active learning with a label uniform distribution model

Cost sensitive active learning using bidirectional gated recurrent neural networks for imbalanced fault diagnosis

Active learning queries labels from the oracle for the most valuable instances to reduce the labeling cost. In many active learning studies, informative and representative instances are preferred because they are expected to have higher potential value for improving the model. Recently, the results in self-paced learning show that training the model with easy examples first and then gradually with harder examples can improve the performance. While informative and representative instances could be easy or hard, querying valuable but hard examples at early stage may lead to waste of labeling cost. In this paper, we propose a self-paced active learning approach to simultaneously consider the potential value and easiness of an instance, and try to train the model with least cost by querying the right thing at the right time. Experimental results show that the proposed approach is superior to state-of-the-art batch mode active learning methods.

Supervised learning, e.g., classification, plays an important role in processing and organizing microblogging data. In microblogging, it is easy to mass vast quantities of unlabeled data, but would be costly to obtain labels, which are essential for supervised learning algorithms. In order to reduce the labeling cost, active learning is an effective way to select representative and informative instances to query for labels for improving the learned model. Different from traditional data in which the instances are assumed to be independent and identically distributed (i.i.d.), instances in microblogging are networked with each other. This presents both opportunities and challenges for applying active learning to microblogging data. Inspired by social correlation theories, we investigate whether social relations can help perform effective active learning on networked data. In this paper, we propose a novel Active learning framework for the classification of Networked Texts in microblogging (ActNeT). In particular, we study how to incorporate network information into text content modeling, and design strategies to select the most representative and informative instances from microblogging for labeling by taking advantage of social network structure. Experimental results on Twitter datasets show the benefit of incorporating network information in active learning and that the proposed framework outperforms existing state-of-the-art methods.

Keyphrase extraction has been a popular research topic in the field of natural language processing in recent years. But how to extract keyphrases precisely and effectively is still a challenge. The mainstream methods are supervised learning methods and graph-based methods. Generally, the effects of supervised methods are better than unsupervised methods. However, there are many problems in supervised methods such as the difficulty in obtaining training data, the cost of labeling and the limitation of the classification function trained by training data. In recent years, the development of the graph-based method has made great progress and its performance of extraction is getting closer and closer to the supervised method, so the graph-based method of keyphrase extraction has got a wide concern from researchers. In this paper, we propose a new model that applies the three-way decision theory to graph-based keyphrase extraction model. In our model, we propose algorithms dividing the set of candidate phrases into the positive domain, the boundary domain and the negative domain depending on graph-based attributes, and combining candidate phrases in the positive domain and the boundary domain qualified by graph-based attributes and non- graph-based attributes to get keyphrases. Experimental results show that our model can effectively improve the extraction precision compared with baseline methods.

An enormous amount of clinical free-text information, such as pathology reports, progress reports, clinical notes and discharge summaries have been collected at hospitals and medical care clinics. These data provide an opportunity of developing many useful machine learning applications if the data could be transferred into a learn-able structure with appropriate labels for supervised learning. The annotation of this data has to be performed by qualified clinical experts, hence, limiting the use of this data due to the high cost of annotation. An underutilised technique of machine learning that can label new data called active learning (AL) is a promising candidate to address the high cost of the label the data. AL has been successfully applied to labelling speech recognition and text classification, however, there is a lack of literature investigating its use for clinical purposes. We performed a comparative investigation of various AL techniques using ML and deep learning (DL)-based strategies on three unique biomedical datasets. We investigated random sampling (RS), least confidence (LC), informative diversity and density (IDD), margin and maximum representativeness-diversity (MRD) AL query strategies. Our experiments show that AL has the potential to significantly reducing the cost of manual labelling. Furthermore, pre-labelling performed using AL expediates the labelling process by reducing the time required for labelling.

We consider the problem of generating balanced training samples from an unlabeled data set with an unknown class distribution. While random sampling works well when the data are balanced, it is very ineffective for unbalanced data. Other approaches, such as active learning and cost-sensitive learning, are also suboptimal as they are classifier-dependent and require misclassification costs and labeled samples, respectively. We propose a new strategy for generating training samples, which is independent of the underlying class distribution of the data and the classifier that will be trained using the labeled data. Our methods are iterative and can be seen as variants of active learning, where we use semi-supervised clustering at each iteration to perform biased sampling from the clusters. We provide several strategies to estimate the underlying class distributions in the clusters and to increase the balancedness in the training samples. Experiments with both highly skewed and balanced data from the UCI repository and a private data set show that our algorithm produces much more balanced samples than random sampling or uncertainty sampling. Further, our sampling strategy is substantially more efficient than active learning methods. The experiments also validate that, with more balanced training data, classifiers trained with our samples outperform classifiers trained with random sampling or active learning.

Data-driven Fault Detection and Diagnostics (FDD) methods often assume that sufficient labeled samples are class-balanced and faulty classes in testing are precedent or seen previously during model training. When monitoring a large fleet of assets at scale, these assumptions may be violated: (I) only a limited number of samples can be manually labeled due to constraints of time and/or cost; (II) most of the samples collected in the engineering systems are under normal conditions, leading to a highly imbalanced class distribution and a biased prediction model. This work presents a robust and cost-effective FDD framework that integrates active learning and semi-supervised learning methods to detect both known and unknown failure modes iteratively. This framework allows to strategically select the samples to be annotated from a fully unlabeled dataset, while labeling cost is minimal. Specifically, a novel graph-based semi-supervised classifier with adaptive graph construction is developed to predict labels with imbalanced data and detect novel classes. We designed a multi-criteria active learning sampling strategy to select the most informative samples from unlabeled data in order to query minimal number of labels for classification. We tested the framework and algorithms in three synthetic datasets and one real-world dataset of vehicle air intake systems, and demonstrated the superior performance compared to the state-of-the-art methods for fleet-level FDD.

Cluster ensemble is a popular method in data mining to discover hidden structures in unlabeled datasets. Some studies have indicated that choosing base clusters with more significant distinction and higher quality can effectively improve the final clustering results. However, the existing base clustering optimization methods consider the diversity and the quality of base clustering separately. Inspired by the three-way decisions, we introduce a three-way optimization model that considers both quality and diversity (3W_CBQD). With two thresholds α and β where α>β, the model calculates the product of the diversity and quality among base clusters as the quality metric and evaluates the quality metric by the thresholds to make decisions. A base cluster will be retained if the quality metric is greater than or equal to the threshold α, removed if the quality metric is less than or equal to the threshold, and recalculated otherwise. This procedure will be repeated until either no base clusters are removed or the number of repeats reaches a specified threshold. Finally, by comparing with some traditional clustering algorithms and clustering ensemble methods, our model shows an effective improvement in the final clustering results with good scalability.

This letter proposes a novel strategy for reducing the cost of in situ sample labeling for the definition of training sets by active learning (AL) in the framework of supervised classification of remote sensing images. AL methods define a training set according to an iterative procedure that at each iteration requires the labeling of a set of new samples selected by the classifier. The proposed strategy can be embedded in any AL method to identify the most informative area on the ground where focusing each AL iteration to reduce the overall cost (in terms of time) of labeling. To this end, at each iteration, the most uncertain unlabeled samples are initially identified. Then, the area on the ground (having a size predefined by the user) that has the highest spatial density of informative (i.e., uncertain and diverse) unlabeled samples is selected by the proposed strategy, and the AL technique is applied only to the samples of that area. This results in a decrease of the overall labeling cost with respect to that required by the use of a given technique in a standard way. Experimental results obtained by embedding the presented strategy in different literature active learning methods confirm its effectiveness.

In recent data-driven approaches to material discovery, scenarios where target quantities are expensive to compute and measure are often overlooked. In such cases, it becomes imperative to construct a training set that includes the most diverse, representative, and informative samples. Here, a novel regression tree-based active learning algorithm is employed for such a purpose. It is applied to predict the band gap and adsorption properties of metal-organic frameworks (MOFs), a novel class of materials that results from the virtually infinite combinations of their building units. Simpler and low dimensional descriptors, such as those based on stoichiometric and geometric properties, are used to compute the feature space for this model owing to their ability to better represent MOFs in the low data regime. The partitions given by a regression tree constructed on the labeled part of the data set are used to select new samples to be added to the training set, thereby limiting its size while maximizing the prediction quality. Tests on the QMOF, hMOF, and dMOF data sets reveal that our method constructs small training data sets to learn regression models that predict the target properties more efficiently than existing active learning approaches, and with lower variance. Specifically, our active learning approach is highly beneficial when labels are unevenly distributed in the descriptor space and when the label distribution is imbalanced, which is often the case for real world data. The regions defined by the tree help in revealing patterns in the data, thereby offering a unique tool to efficiently analyze complex structure-property relationships in materials and accelerate materials discovery.

Due to technological advances, a massive amount of data is produced daily, presenting challenges for application areas where data needs to be labelled by a domain specialist or by expensive procedures, in order to be useful for supervised machine learning purposes. In order to select which data points will provide more information when labelled, one can make use of active learning methods. Active learning (AL) is a subfield of machine learning which addresses methods to build models with fewer, but more representative instances. Even though AL has been vastly studied, it has not been thoroughly investigated in hierarchical multi-label classification, a learning task where multiple class labels can be assigned to an instance and these labels are hierarchically structured. In this work, we provide a public framework containing baseline and state-of-the-art algorithms suitable for this task. Additionally, we also propose a new algorithm, namely Hierarchical Query-By-Committee (H-QBC), which is validated on datasets from different domains. Our results show that H-QBC is capable of providing superior predictive performance results compared to its competitors, while being computationally efficient and parameter free.

Machine learning indicates methods and algorithms which allow a model to learn a behavior thanks to examples. Active learning gathers methods which select examples used to build a training set for the predictive model. All the strategies aim to use the less examples as possible and to select the most informative examples. After having formalized the active learning problem and after having located it in the literature, this article synthesizes in the first part the main approaches of active learning. Taking into account emotions in Human-machine interactions can be helpful for intelligent systems designing. The main difficulty, for the conception of calls center's automatic shunting system, is the cost of data labeling. The last section of this paper propose to reduce this cost thanks to two active learning strategies. The study is based on real data resulting from the use of a vocal stock exchange server.

Hemmer, Patrick Kühl, Niklas Schöffer, Jakob Convolutional Neural Networks (CNNs) have proven to be state-of-the-art models for supervised computer vision tasks, such as image classification. However, large labeled datasets are generally needed for the training and validation of such models. In many domains, unlabeled data is available but labeling is expensive, for instance, when specific expert knowledge is required. Active Learning (AL) is one approach to mitigate the problem of limited labeled data. Through selecting the most informative and representative data instances for labeling, AL can contribute to more efficient learning of a model. Recent AL methods for CNNs propose different solutions for the selection of to-be-labeled instances. However, they do not perform consistently well and are often computationally expensive. In this paper, we propose a novel AL algorithm that efficiently learns from unlabeled data by capturing high prediction uncertainty. By replacing the softmax standard output of a CNN with the parameters of a Dirichlet density, the model learns to identify data instances that contribute efficiently to improving model performance during training. We demonstrate in several experiments with publicly available data that our method consistently outperforms other state-of-the-art AL approaches. It can be easily implemented and does not require extensive computational resources for training. Additionally, we are able to show the benefits of the approach on the real-world medical use case of pneumonia detection in chest X-ray images.

Active semi-supervised learning with multiple complementary information