Privileged Semi-Supervised Learning

The privileged information that is available only for the training examples and not available for test examples, is a new concept proposed by Vapnik and Vashist (Neural Netw 22(5–6):544–557, 2009). With the help of the privileged information, learning using privileged information (LUPI) (Neural Netw 22(5–6):544–557, 2009) can significantly accelerate the speed of learning. However, LUPI is a standard supervised learning method. In fact, in many real-world problems, there are also a lot of unlabeled data. This drives us to solve problems under a semi-supervised learning framework. In this paper, we propose a semi-supervised learning using privileged information (called Semi-LUPI), which can exploit both the distribution information in unlabeled data and privileged information to improve the efficiency of the learning. Furthermore, we also compare the relative importance of both types of information for the learning model. All experiments verify the effectiveness of the proposed method, and simultaneously show that Semi-LUPI can obtain superior performances over traditional supervised and semi-supervised methods.

Alzheimer's disease (AD) is a progressive brain disorder. Machine learning models have been proposed for the diagnosis of AD at early stage. Recently, deep learning architectures have received quite a lot attention. Most of the deep learning architectures suffer from the issues of local minima, slow convergence and sensitivity to learning rate. To overcome these issues, non-iterative learning based deep randomized models especially random vector functional link network (RVFL) with direct links have proven to be successful. However, deep RVFL and its ensemble models are trained only on normal samples. In this paper, deep RVFL and its ensembles are enabled to incorporate privileged information, as the standard RVFL model and its deep models are unable to use privileged information. To fill this gap, we have incorporated learning using privileged information (LUPI) in deep RVFL model, and propose deep RVFL with LUPI framework (dRVFL+). Privileged information is available while training the models. As RVFL is an unstable classifier, we propose ensemble deep RVFL+ with LUPI framework (edRVFL+) which exploits the LUPI as well as the diversity among the base leaners for better classification. Unlike traditional ensemble approach wherein multiple base learners are trained, the proposed edRVFL+ model optimises a single network and generates an ensemble via optimization at different levels of random projections of the data. Both dRVFL+ and edRVFL+ efficiently utilise the privileged information which results in better generalization performance. In LUPI framework, half of the available features are used as normal features and rest as the privileged features. However, we propose a novel approach for generating the privileged information. We utilise different activation functions while processing the normal and privileged information in the proposed deep architectures. To the best of our knowledge, this is first time that a separate privileged information is generated. The proposed dRVFL+ and edRVFL+ models are employed for the diagnosis of Alzheimer's disease. Experimental results demonstrate the superiority of the proposed dRVFL+ and edRVFL+ models over baseline models. Thus, the proposed edRVFL+ model can be utilised in clinical setting for the diagnosis of AD.

Learning using privileged information with logistic regression on acute respiratory distress syndrome detection

Face verification is an important task to verify people’s identities, which has sample pairs and labels with only side information about whether the two images in a pair are from the same subject or not instead of a specific label for each sample. This paper is motivated by the limitations of previous efforts that the deep neural networks with only the RGB feature are not to intra-class noise (e.g., illumination), and the deep networks with both the RGB and depth feature are not to data availability of the depth images in applications. To overcome these limitations, we focus on developing a double robust face verification architecture using the Siamese convolutional neural network (SCNN). In particular, our goal is to propose an SCNN with privileged information (SCNN+), which is inspired by Learning Using Privileged Information (LUPI), through incorporating additional depth feature along with the RGB feature in the training stage to provide privileged information to restrain the RGB feature’s prediction error. We conduct experiments to evaluate the performance of the proposed SCNN+ architecture and compare it with different categories of state-of-the-art baselines on two real-world RGB-D face datasets. The evaluation results demonstrate that SCNN+ significantly outperforms all types of baselines.

While recent studies on semi-supervised learning have shown remarkable progress in leveraging both labeled and unlabeled data, most of them presume a basic setting of the model is randomly initialized. In this work, we consider semi-supervised learning and transfer learning jointly, leading to a more practical and competitive paradigm that can utilize both powerful pre-trained models from source domain as well as labeled/unlabeled data in the target domain. To better exploit the value of both pre-trained weights and unlabeled target examples, we introduce adaptive consistency regularization that consists of two complementary components: Adaptive Knowledge Consistency (AKC) on the examples between the source and target model, and Adaptive Representation Consistency (ARC) on the target model between labeled and unlabeled examples. Examples involved in the consistency regularization are adaptively selected according to their potential contributions to the target task. We conduct extensive experiments on popular benchmarks including CIFAR-10, CUB-200, and MURA, by fine-tuning the ImageNet pre-trained ResNet-50 model. Results show that our proposed adaptive consistency regularization outperforms state-of-the-art semi-supervised learning techniques such as Pseudo Label, Mean Teacher, and FixMatch. Moreover, our algorithm is orthogonal to existing methods and thus able to gain additional improvements on top of MixMatch and FixMatch. Our code is available at https://github.com/Walleclipse/Semi-Supervised-Transfer-Learning-Paddle.

The efficiency in breast cancer from imaging-based computer-aided diagnosis (CAD) has been revealed in recent years. As a fact, the methods grounded on a single modality constantly lack behind multimodal CAD imaging. However, owing to the restrictions of imaging devices, expressly in rural hospitals, single-modal imaging becomes a favorite in clinical practice for diagnosis. A fresh learning model trending nowadays known as learning using privileged information (LUPI) adopts additional privileged information (PI) modality to help during the training stage, but PI does not contribute in the testing stage. Meanwhile, the link exists between PI and training samples; the same is then reassigned to the learned model. We propose a LUPI-based CAD framework for breast cancer using privileged information in this work. The work offers both a classifier- or feature-level LUPI, in which the information is shifted from the additional PI modality to the diagnosis modality. A thorough comparison has been made among six classifier-level algorithms and six feature-level LUPI algorithms. The experimental results on both the acquired primary datasets show that all classifier-level and deep learning-based feature-level LUPI algorithms can enhance the performance of a single-modal imaging-based CAD for breast cancer by relocating PI.

Face anti-spoofing is an important part of the face recognition algorithm that aims to prevent face presentation attacks. To facilitate face anti-spoofing research, this paper introduces a novel method in which a simple model provides enhanced performance on the RGB modality images of the CASIA-SURF dataset. Initially, a simple model based on ResNet-50 architecture is proposed that uses three residual layers of ResNet-50 along with a single neuron at the end for binary classification. Meanwhile, to judge the baseline performance, the proposed Resnet-50-based model is trained on the RGB and Depth images, separately. Later, to increase the performance of the proposed ResNet-50-based model on the RGB modality images, a novel Learning Using Privileged Information (LUPI) algorithm is applied. The LUPI contains Depth images and helps to train the inferior model with the help of a pre-trained superior model. Detailed simulations on the CASIA-SURF dataset reveal that an intelligent combination of the ResNet50 based model with the LUPI paradigm works exceptionally well and significantly improves the performance on the RGB modality of the images. The proposed model is less complex and can be deployed in resource-constrained environments.

In recent years, non-iterative learning approaches for kernel have received quite an attention by researchers and kernel ridge regression (KRR) approach is one of them. Recently, KRR-based Auto-Encoder is developed for the one-class classification (OCC) task and named as AEKOC. OCC is generally used for outlier or novelty detection. The brain can detect outlier just by learning from only normal samples. Similarly, OCC also uses only normal samples to train the model, and trained model can be used for outlier detection. In this paper, AEKOC is enabled to utilize privileged information, which is generally ignored by AEKOC or any traditional machine learning technique but usually present in human learning. For this purpose, we have combined learning using privileged information (LUPI) framework with AEKOC, and proposed a classifier, which is referred to as AEKOC+. Privileged information is only available during training but not during testing. Therefore, AEKOC is unable to utilize this information for building the model. However, AEKOC+ can efficiently handle the privileged information due to the inclusion of the LUPI framework with AEKOC. Experiments have been conducted on MNIST dataset and on various other datasets from UCI machine learning repository, which demonstrates the superiority of AEKOC+ over AEKOC. Our formulation shows that AEKOC does not utilize the privileged features in learning; however, formulation of AEKOC+ helps it in learning from the privileged features differently from other available features and improved generalization performance of AEKOC. Moreover, AEKOC+ also outperformed two LUPI framework–based one-class classifiers (i.e., OCSVM+ and SSVDD+).

Learning the implicit strain reconstruction in ultrasound elastography using privileged information.

In many human activity recognition systems the size of the unlabeled training data may be significantly large due to expensive human effort required for data annotation. Moreover, the insufficient data collection process from heterogenous sources may cause dissimilarities between training and testing data. To address these limitations, a novel probabilistic approach that combines learning using privileged information (LUPI) and active learning is proposed. A pool-based privileged active learning approach is presented for semi-supervising learning of human activities from multimodal labeled and unlabeled data. Both uncertainty and distance from the decision boundary are used as query inference strategies for selecting an unlabeled observation and querying its label. Experimental results in four publicly available datasets demonstrate that the proposed method can identify complex human activities with high accuracy.

Multi-target regression is the problem where each instance is associated with multiple continuous target outputs simultaneously. Its major challenges arise with jointly exploring the complex input-output relationships and inter-target correlations. One representative approach is to build many independent single-target Support Vector Regression (SVR) models for each output target which can capture complex input-output relationships via the kernel trick. However, it does not involve inter-target correlations to improve the performance. Meanwhile, there are also many regularization-based methods which mainly explore the linear inter-target correlations, e.g., a low-rank constraint on the parameter matrix. However, in practice, it might be restrictive to assume the targets to be linearly related, and allowing for nonlinear relationships is a challenge. Motivated by Learning Using Privileged Information (LUPI), we propose a novel privileged multi-target support vector regression (MT-PSVR) model which can jointly explore the complex input-output relationships and nonlinear inter-target correlations. It explicitly explores inter-target correlations by viewing other targets as privileged information when training each target model. Besides, it can naturally use the kernel trick to explore both the complex input-output relationships and nonlinear inter-target correlations. Experimental results on many benchmark datasets validate the effectiveness of our approach.

Most of the facial expression recognition methods consider that both training and testing data are equally distributed. As facial image sequences may contain information for heterogeneous sources, facial data may be asymmetrically distributed between training and testing, as it may be difficult to maintain the same quality and quantity of information. In this work, we present a novel classification method based on the learning using privileged information (LUPI) paradigm to address the problem of facial expression recognition. We introduce a probabilistic classification approach based on conditional random fields (CRFs) to indirectly propagate knowledge from privileged to regular feature space. Each feature space owns specific parameter settings, which are combined together through a Gaussian prior, to train the proposed t-CRF+ model and allow the different tasks to share parameters and improve classification performance. The proposed method is validated on two challenging and publicly available benchmarks on facial expression recognition and improved the state-of-the-art methods in the LUPI framework.

Affective Computing (AC) has made significant progress with the advent of deep learning, yet a persistent challenge remains: the reliable transfer of affective models from controlled laboratory settings (in-vitro) to uncontrolled real-world environments (in-vivo). To address this challenge we introduce the Privileged Contrastive Pretraining (PriCon) framework according to which models are first pretrained via supervised contrastive learning (SCL) and then act as teacher models within a Learning Using Privileged Information (LUPI) framework. PriCon both leverages privileged information during training and enhances the robustness of derived affect models via SCL. Experiments conducted on two benchmark affective corpora, RECOLA and AGAIN, demonstrate that models trained using PriCon consistently outperform LUPI and end to end models. Remarkably, in many cases, PriCon models achieve performance comparable to models trained with access to all modalities during both training and testing. The findings underscore the potential of PriCon as a paradigm towards further bridging the gap between in-vitro and in-vivo affective modelling, offering a scalable and practical solution for real-world applications.

Traffic anomaly detection is primarily concerned with identifying malicious traffic patterns in a much larger stream of benign traffic. Traditionally, this is achieved by selecting a very specialized set of traffic-based features that are used for both training a model, as well as for detection at runtime. This paper introduces a novel method of anomaly detection that breaks the assumption that the same set of features should be used for both training and runtime detection. Building upon the concept of Learning Using Privileged Information (LUPI), this paper shows how to build a detection model that uses an extended set of features, some that are available only during training, and some also available at runtime, with results significantly superior to the traditional learning methods. We demonstrate the usage of the privileged features in detecting malicious botnet activities in a network, activities shown to be challenging to detect in past works.

A growing specter in the rise of machine learning is whether the decisions made by machine learning models are fair. While research is already underway to formalize a machine-learning concept of fairness and to design frameworks for building fair models with sacrifice in accuracy, most are geared toward either supervised or unsupervised learning. Yet two observations inspired us to wonder whether semi-supervised learning might be useful to solve discrimination problems. First, previous study showed that increasing the size of the training set may lead to a better trade-off between fairness and accuracy. Second, the most powerful models today require an enormous of data to train which, in practical terms, is likely possible from a combination of labeled and unlabeled data. Hence, in this paper, we present a framework of fair semi-supervised learning in the pre-processing phase, including pseudo labeling to predict labels for unlabeled data, a re-sampling method to obtain multiple fair datasets and lastly, ensemble learning to improve accuracy and decrease discrimination. A theoretical decomposition analysis of bias, variance and noise highlights the different sources of discrimination and the impact they have on fairness in semi-supervised learning. A set of experiments on real-world and synthetic datasets show that our method is able to use unlabeled data to achieve a better trade-off between accuracy and discrimination.