A Neural Network Model for Online Multi-Task Multi-Label Pattern Recognition

The Antarctic Ice Sheet is the largest potential contributor to future global sea level, with a maximum contribution of approximately 58 m. The flow of grounded ice towards the ocean is largely restricted by floating ice shelves, which fringe ~75% of the Antarctic coastline. This buttressing effect can be diminished or lost entirely following partial or complete ice-shelf disintegration events, leading to an acceleration of ice discharge. The vulnerability of ice shelves to fracture and disintegration is influenced by a range of factors, including the ponding of surface and subsurface meltwater, which can induce flexural stresses and promote fracture through the ice-shelf column.While the widespread extent of surface meltwater systems across numerous Antarctic ice shelves during the austral summer is now well documented, meltwater storage within the ice-shelf subsurface remains poorly understood. Liquid water can persist perennially beneath the ice-shelf surface if sufficiently insulated by surrounding and overlying layers of firn, snow, and/or ice. Due to their year-round persistence, buried meltwater lakes introduce a potential mechanism for hydrofracture outside of the melt season, with important implications for ice-shelf stability.In situ surveys in Antarctica are logistically challenging and limited in spatial extent, rendering spaceborne remote sensing an indispensable tool for monitoring ice-shelf processes at a continental scale. However, the growing volume of satellite observation poses data-analysis challenges typical of the ‘Big Data’ era; remote-sensing datasets are often high-dimensional, unstructured, and large in sample size, with rapidly increasing spatiotemporal coverage and low-cost availability. These characteristics limit the scalability of manual interpretation or traditional thresholding approaches for pan-Antarctic applications. In contrast, machine-learning methods are ideal for extracting patterns and features from large databases of satellite imagery. In light of this, machine learning offers great potential for the detection of subsurface meltwater across Antarctic ice shelves at a continental scale, if challenges relating to the sampling and labelling of training data, as well as class imbalance, are addressed carefully. An active learning strategy can help to reduce data redundancy and labelling requirements in deep learning, while also improving model performance in the presence of class imbalance. Here, we present a systematic training-data sampling strategy applied to all major Antarctic ice shelves. We employ a stratified random sampling approach to mitigate strong regional imbalances in data availability and create a curated training data subset that combines an equal number of random and expert-selected samples. This dataset is used to initialise an active learning framework for training a deep-learning model to detect subsurface lakes in Antarctica. We evaluate model performance across multiple configurations and present fine-tuning of model hyperparameters.

In this work, we extend the sequential multitask learning model called Resource Allocating Network for Multi-Task Pattern Recognition (RAN-MTPR) by introducing the following new learning functions: multi-label recognition, semi-supervised task learning and active learning. The extended RAN-MTPR can learn a training data with multiple class labels, can handle a semi-supervised setting for task learning, and can actively request class labels for unsure inputs. We evaluate the performance of the extended RAN-MTPR, and we know that the above three functions work well to enhance the generalization performance for pattern recognition problems.

Multi-label learning draws great interests in many real world applications. It is a highly costly task to assign many labels by the oracle for one instance. Meanwhile, it is also hard to build a good model without diagnosing discriminative labels. Can we reduce the label costs and improve the ability to train a good model for multi-label learning simultaneously? Active learning addresses the less training samples problem by querying the most valuable samples to achieve a better performance with little costs. In multi-label active learning, some researches have been done for querying the relevant labels with less training samples or querying all labels without diagnosing the discriminative information. They all cannot effectively handle the outlier labels for the measurement of uncertainty. Since maximum correntropy criterion (MCC) provides a robust analysis for outliers in many machine learning and data mining algorithms, in this paper, we derive a robust multi-label active learning algorithm based on an MCC by merging uncertainty and representativeness, and propose an efficient alternating optimization method to solve it. With MCC, our method can eliminate the influence of outlier labels that are not discriminative to measure the uncertainty. To make further improvement on the ability of information measurement, we merge uncertainty and representativeness with the prediction labels of unknown data. It cannot only enhance the uncertainty but also improve the similarity measurement of multi-label data with labels information. Experiments on benchmark multi-label data sets have shown a superior performance than the state-of-the-art methods.

Active learning is well-motivated in many supervised learning tasks where unlabeled data may be abundant but labeled examples are expensive to obtain. The goal of active learning is to maximize the performance of a learning model using as few labeled training data as possible, thereby minimizing the cost of data annotation. So far, there is still very limited work on active learning for regression. In this paper, we propose a new active learning framework for regression called Expected Model Change Maximization (EMCM), which aims to choose the examples that lead to the largest change to the current model. The model change is measured as the difference between the current model parameters and the updated parameters after training with the enlarged training set. Inspired by the Stochastic Gradient Descent (SGD) update rule, the change is estimated as the gradient of the loss with respect to a candidate example for active learning. Under this framework, we derive novel active learning algorithms for both linear regression and nonlinear regression to select the most informative examples. Extensive experimental results on the benchmark data sets from UCI machine learning repository have demonstrated that the proposed algorithms are highly effective in choosing the most informative examples and robust to various types of data distributions.

Online services are interested in solutions to opinion mining, which is the problem of extracting aspects, opinions, and sentiments from text. One method to mine opinions is to leverage the recent success of pre-trained language models which can be fine-tuned to obtain high-quality extractions from reviews. However, fine-tuning language models still requires a non-trivial amount of training data. In this paper, we study the problem of how to significantly reduce the amount of labeled training data required in fine-tuning language models for opinion mining. We describe Snippext, an opinion mining system developed over a language model that is fine-tuned through semi-supervised learning with augmented data. A novelty of Snippext is its clever use of a two-prong approach to achieve state-of-the-art (SOTA) performance with little labeled training data through: (1) data augmentation to automatically generate more labeled training data from existing ones, and (2) a semi-supervised learning technique to leverage the massive amount of unlabeled data in addition to the (limited amount of) labeled data. We show with extensive experiments that Snippext performs comparably and can even exceed previous SOTA results on several opinion mining tasks with only half the training data required. Furthermore, it achieves new SOTA results when all training data are leveraged. By comparison to a baseline pipeline, we found that Snippext extracts significantly more fine-grained opinions which enable new opportunities of downstream applications.

Multimodal hashing has attracted much interest for cross-modal similarity search on large-scale multimedia data sets because of its efficiency and effectiveness. Recently, supervised multimodal hashing, which tries to preserve the semantic information obtained from the labels of training data, has received considerable attention for its higher search accuracy compared with unsupervised multimodal hashing. Although these algorithms are promising, they are mainly designed to preserve pairwise similarities. When semantic labels of training data are given, the algorithms often transform the labels into pairwise similarities, which gives rise to the following problems: (1) constructing pairwise similarity matrix requires enormous storage space and a large amount of calculation, making these methods unscalable to large-scale data sets; (2) transforming labels into pairwise similarities loses the category information of the training data. Therefore, these methods do not enable the hash codes to preserve the discriminative information reflected by labels and, hence, the retrieval accuracies of these methods are affected. To address these challenges, this paper introduces a simple yet effective supervised multimodal hashing method, called label consistent matrix factorization hashing (LCMFH), which focuses on directly utilizing semantic labels to guide the hashing learning procedure. Considering that relevant data from different modalities have semantic correlations, LCMFH transforms heterogeneous data into latent semantic spaces in which multimodal data from the same category share the same representation. Therefore, hash codes quantified by the obtained representations are consistent with the semantic labels of the original data and, thus, can have more discriminative power for cross-modal similarity search tasks. Thorough experiments on standard databases show that the proposed algorithm outperforms several state-of-the-art methods.

Active Learning is a machine learning and data mining technique that selects the most informative samples for labeling and uses them as training data; it is especially useful when there are large amount of unlabeled data and labeling them is expensive. Recently, batch-mode active learning, where a set of samples are selected concurrently for labeling, based on their collective merit, has attracted a lot of attention. The objective of batch-mode active learning is to select a set of informative samples so that a classifier learned on these samples has good generalization performance on the unlabeled data. Most of the existing batch-mode active learning methodologies try to achieve this by selecting samples based on certain criteria. In this article we propose a novel criterion which achieves good generalization performance of a classifier by specifically selecting a set of query samples that minimize the difference in distribution between the labeled and the unlabeled data, after annotation. We explicitly measure this difference based on all candidate subsets of the unlabeled data and select the best subset. The proposed objective is an NP-hard integer programming optimization problem. We provide two optimization techniques to solve this problem. In the first one, the problem is transformed into a convex quadratic programming problem and in the second method the problem is transformed into a linear programming problem. Our empirical studies using publicly available UCI datasets and two biomedical image databases demonstrate the effectiveness of the proposed approach in comparison with the state-of-the-art batch-mode active learning methods. We also present two extensions of the proposed approach, which incorporate uncertainty of the predicted labels of the unlabeled data and transfer learning in the proposed formulation. In addition, we present a joint optimization framework for performing both transfer and active learning simultaneously unlike the existing approaches of learning in two separate stages, that is, typically, transfer learning followed by active learning. We specifically minimize a common objective of reducing distribution difference between the domain adapted source, the queried and labeled samples and the rest of the unlabeled target domain data. Our empirical studies on two biomedical image databases and on a publicly available 20 Newsgroups dataset show that incorporation of uncertainty information and transfer learning further improves the performance of the proposed active learning based classifier. Our empirical studies also show that the proposed transfer-active method based on the joint optimization framework performs significantly better than a framework which implements transfer and active learning in two separate stages.

Machine learning techniques have shown superior predictive power, among which Bayesian network classifiers (BNCs) have remained of great interest due to its capacity to demonstrate complex dependence relationships. Most traditional BNCs tend to build only one model to fit training instances by analyzing independence between attributes using conditional mutual information. However, for different class labels, the conditional dependence relationships may be different rather than invariant when attributes take different values, which may result in classification bias. To address this issue, we propose a novel framework, called discriminatory target learning, which can be regarded as a tradeoff between probabilistic model learned from unlabeled instance at the uncertain end and that learned from labeled training data at the certain end. The final model can discriminately represent the dependence relationships hidden in unlabeled instance with respect to different possible class labels. Taking k-dependence Bayesian classifier as an example, experimental comparison on 42 publicly available datasets indicated that the final model achieved competitive classification performance compared to state-of-the-art learners such as Random forest and averaged one-dependence estimators.

The study of model bias and variance with respect to decision boundaries is critically important in supervised learning and artificial intelligence. There is generally a tradeoff between the two, as fine-tuning of the decision boundary of a classification model to accommodate more boundary training samples (i.e., higher model complexity) may improve training accuracy (i.e., lower bias) but hurt generalization against unseen data (i.e., higher variance). By focusing on just classification boundary fine-tuning and model complexity, it is difficult to reduce both bias and variance. To overcome this dilemma, we take a different perspective and investigate a new approach to handle inaccuracy and uncertainty in the training data labels, which are inevitable in many applications where labels are conceptual entities and labeling is performed by human annotators. The process of classification can be undermined by uncertainty in the labels of the training data; extending a boundary to accommodate an inaccurately labeled point will increase both bias and variance. Our novel method can reduce both bias and variance by estimating the pointwise label uncertainty of the training set and accordingly adjusting the training sample weights such that those samples with high uncertainty are weighted down and those with low uncertainty are weighted up. In this way, uncertain samples have a smaller contribution to the objective function of the model’s learning algorithm and exert less pull on the decision boundary. In a real-world physical activity recognition case study, the data present many labeling challenges, and we show that this new approach improves model performance and reduces model variance.

In this paper, we propose a joint machine learning and human learning design approach to make the training data labeling task in linear regression problems more efficient and robust to noise, modeling mismatch, and human labeling errors. Considering a sequential active learning scheme which relies on human learning to enlarge training data set, we integrate it with sparse outlier detection algorithms to mitigate the inevitable human errors during training data labeling. First, we assume sparse human errors and formulate the outlier detection as a sparse optimization problem within the sequential active learning procedure. Then, for non-sparse human errors, with the IRT (item response theory) to model the distribution of human errors, appropriate data are selected to reconstruct a training data set with sparse human errors. Simulations are conducted to verify the desirable performance of the proposed approach.

Social media is a largely untapped source of information on side effects of drugs. Twitter in particular is widely used to report on everyday events and personal ailments. However, labeling this noisy data is a difficult problem because labeled training data is sparse and automatic labeling is error-prone. Crowd sourcing can help in such a scenario to obtain more reliable labels, but is expensive in comparison because workers have to be paid. To remedy this, semi-supervised active learning may reduce the number of labeled data needed and focus the manual labeling process on important information. We extracted data from Twitter using the public API. We subsequently use Amazon Mechanical Turk in combination with a state-of-the-art semi-supervised active learning method to label tweets with their associated drugs and side effects in two stages. Our results show that our method is an effective way of discovering side effects in tweets with an improvement from 53% F-measure to 67% F-measure as compared to a one stage work flow. Additionally, we show the effectiveness of the active learning scheme in reducing the labeling cost in comparison to a non-active baseline. Code and data will be published on https://github.com/kramerlab.

Object detection models have made significant progress and achieved state-of-the-art performance, which can now be comparable to human experts in various domains. However, training such models often requires a large amount of labeled data, which can be challenging to obtain. To address this issue, Active Learning (AL) has emerged as a technique to enhance the efficiency of deep learning models by reducing the amount of data and time required to train models to a satisfactory level. In this paper, in the context of smart farming, we propose to study the impact of AL in object detection models trained with a small dataset of labelled images of whitefly-infested tomato leaves. We use YOLOv5 and fit the bounding box with confidence as a score function to select the most active relevant examples. The results show a trade-off between performance and cost suggesting AL outweigh the associated costs when labelled training data is limited.

In recent years, Convolutional Neural Networks (CNNs) have been widely successful for numerous computer vision tasks using 2D color and 3D point data, such as object detection and classification, and 3D pose estimation. Despite these successes, it comes with a price --- CNNs require a vast amount of labeled training data to be successful. Many training datasets are now available, however, they are mostly limited to general objects in the public domain such as people, cars, buildings, fruit, and more. Thus, not all fields have sufficient training data. Even if an adequate number of data samples can be built into a dataset, manually collecting and labeling the training data remains a laborious task. Several approaches to reduce or eradicate the need for training data have been proposed by researchers. The two most popular approaches are training with synthetic data and training on generalized object features. Training with synthetic data refers to the use of computer graphics to generate training data. While this approach has demonstrated success, it comes with its own challenges. In contrast, training a CNN on generalized object features identifies specific local 3D features or control points on objects. Local 3D features and their descriptors have proven in the past to be universally applicable and effective for general object detection, and thus, this method allows for a generalized CNN. A generalized CNN refers to a network architecture that, after initial training, can be specialized for a new task with a minimum amount or no new training data. This dissertation investigates an encoder-decoder architecture and determines the capabilities of it to generalize the CNN architecture for object detection tasks in point cloud data. A CNN is trained to recognize 3D features and demonstrates, without retraining, its successful applicability for point cloud object detection using disparate data. Additionally, the geometric consistency of the descriptors is evaluated to obtain further insight. The results demonstrate that the proposed 3D feature descriptors, and their increased geometric consistency contribute to the increased encoder-decoder architecture performance. In summary, this dissertation contributes a state-of-the-art, generalized, feature descriptor-based CNN architecture that can be transferred to different objects without retraining. Furthermore, it provides insight explaining the increased performance of the CNN compared to the state-of-the-art.

Cross-lingual text classification (CLTC) refers to the task of classifying documents in different languages into the same taxonomy of categories. An open challenge in CLTC is to classify documents for the languages where labeled training data are not available. Existing approaches rely on the availability of either high-quality machine translation of documents (to the languages where massively training data are available), or rich bilingual dictionaries for effective translation of trained classification models (to the languages where labeled training data are lacking). This paper studies the CLTC challenge under the assumption that neither condition is met. That is, we focus on the problem of translating classification models with highly incomplete bilingual dictionaries. Specifically, we propose two new approaches that combines unsupervised word embedding in different languages, supervised mapping of embedded words across languages, and probabilistic translation of classification models. The approaches show significant performance improvement in CLTC on a benchmark corpus of Reuters news stories (RCV1/RCV2) in English, Spanish, German, French and Chinese and an internal dataset in Uzbek, compared to representative baseline methods using conventional bilingual dictionaries or highly incomplete ones.

Anomaly intrusion detection normally has high false alarm rates, and a high volume of false alarms will prevent system administrators identifying the real attacks. Machine learning methods provide an effective way to decrease the false alarm rate and improve the detection rate of anomaly intrusion detection. In this research, we propose a novel approach using kernel methods and Support Vector Machine (SVM) for improving anomaly intrusion detectors' accuracy. Two kernels, STIDE kernel and Markov Chain kernel, are developed specially for intrusion detection applications. The experiments show the STIDE and Markov Chain kernel based two class SVM anomaly detectors have better accuracy rate than the original STIDE and Markov Chain anomaly detectors. Generally, anomaly intrusion detection approaches build normal profiles from labeled training data. However, labeled training data for intrusion detection is expensive and not easy to obtain. We propose an anomaly detection approach, using STIDE kernel and Markov Chain kernel based one class SVM, that does not need labeled training data. To further increase the detection rate and lower the false alarm rate, an approach of integrating specification based intrusion detection with anomaly intrusion detection is also proposed. This research also establish a platform which generates automatically both misuse and anomaly intrusion detection software agents. In our method, a SIFT representing an intrusion is automatically converted to a Colored Petri Net (CPNs) representing an intrusion detection template, subsequently, the CPN is compiled into code for misuse intrusion detection software agents using a compiler and dynamically loaded and launched for misuse intrusion detection. On the other hand, a model representing a normal profile is automatically generated from training data, subsequently, an anomaly intrusion detection agent which carries this model is generated and launched for anomaly intrusion detection. By engaging both misuse and anomaly intrusion detection agents, our system can detect known attacks as well as novel unknown attacks.