Semi-supervised classification with privileged information

Alzheimer's disease (AD) is a progressive brain disorder. Machine learning models have been proposed for the diagnosis of AD at early stage. Recently, deep learning architectures have received quite a lot attention. Most of the deep learning architectures suffer from the issues of local minima, slow convergence and sensitivity to learning rate. To overcome these issues, non-iterative learning based deep randomized models especially random vector functional link network (RVFL) with direct links have proven to be successful. However, deep RVFL and its ensemble models are trained only on normal samples. In this paper, deep RVFL and its ensembles are enabled to incorporate privileged information, as the standard RVFL model and its deep models are unable to use privileged information. To fill this gap, we have incorporated learning using privileged information (LUPI) in deep RVFL model, and propose deep RVFL with LUPI framework (dRVFL+). Privileged information is available while training the models. As RVFL is an unstable classifier, we propose ensemble deep RVFL+ with LUPI framework (edRVFL+) which exploits the LUPI as well as the diversity among the base leaners for better classification. Unlike traditional ensemble approach wherein multiple base learners are trained, the proposed edRVFL+ model optimises a single network and generates an ensemble via optimization at different levels of random projections of the data. Both dRVFL+ and edRVFL+ efficiently utilise the privileged information which results in better generalization performance. In LUPI framework, half of the available features are used as normal features and rest as the privileged features. However, we propose a novel approach for generating the privileged information. We utilise different activation functions while processing the normal and privileged information in the proposed deep architectures. To the best of our knowledge, this is first time that a separate privileged information is generated. The proposed dRVFL+ and edRVFL+ models are employed for the diagnosis of Alzheimer's disease. Experimental results demonstrate the superiority of the proposed dRVFL+ and edRVFL+ models over baseline models. Thus, the proposed edRVFL+ model can be utilised in clinical setting for the diagnosis of AD.

Learning using privileged information with logistic regression on acute respiratory distress syndrome detection

Learning the implicit strain reconstruction in ultrasound elastography using privileged information.

In many human activity recognition systems the size of the unlabeled training data may be significantly large due to expensive human effort required for data annotation. Moreover, the insufficient data collection process from heterogenous sources may cause dissimilarities between training and testing data. To address these limitations, a novel probabilistic approach that combines learning using privileged information (LUPI) and active learning is proposed. A pool-based privileged active learning approach is presented for semi-supervising learning of human activities from multimodal labeled and unlabeled data. Both uncertainty and distance from the decision boundary are used as query inference strategies for selecting an unlabeled observation and querying its label. Experimental results in four publicly available datasets demonstrate that the proposed method can identify complex human activities with high accuracy.

Semi-Supervised Learning (SSL) aims to leverage unlabeled data to improve performance. Due to the lack of supervised information, previous works mainly focus on how to utilize the available unlabeled data to improve the training quality. However, the estimation of the data distribution revealed by the unlabeled examples might not be accurate as their classes are unknown. Inspired by the framework of Learning Using Privileged Information (LUPI), we propose to introduce an intelligent “teacher” which can utilize the privileged information (i.e. more precise explanations of training set) to improve the performance of SSL. This developed method is named as Privileged Semi-Supervised Learning (PSSL). Moreover, our method can be efficiently solved with $\ell_{2}$ -loss. Experimentally, we compare our PSSL with several typical algorithms on three real-world datasets, and the results suggest that our approach is able to achieve the state-of-the-art performance.

The efficiency in breast cancer from imaging-based computer-aided diagnosis (CAD) has been revealed in recent years. As a fact, the methods grounded on a single modality constantly lack behind multimodal CAD imaging. However, owing to the restrictions of imaging devices, expressly in rural hospitals, single-modal imaging becomes a favorite in clinical practice for diagnosis. A fresh learning model trending nowadays known as learning using privileged information (LUPI) adopts additional privileged information (PI) modality to help during the training stage, but PI does not contribute in the testing stage. Meanwhile, the link exists between PI and training samples; the same is then reassigned to the learned model. We propose a LUPI-based CAD framework for breast cancer using privileged information in this work. The work offers both a classifier- or feature-level LUPI, in which the information is shifted from the additional PI modality to the diagnosis modality. A thorough comparison has been made among six classifier-level algorithms and six feature-level LUPI algorithms. The experimental results on both the acquired primary datasets show that all classifier-level and deep learning-based feature-level LUPI algorithms can enhance the performance of a single-modal imaging-based CAD for breast cancer by relocating PI.

In recent years, non-iterative learning approaches for kernel have received quite an attention by researchers and kernel ridge regression (KRR) approach is one of them. Recently, KRR-based Auto-Encoder is developed for the one-class classification (OCC) task and named as AEKOC. OCC is generally used for outlier or novelty detection. The brain can detect outlier just by learning from only normal samples. Similarly, OCC also uses only normal samples to train the model, and trained model can be used for outlier detection. In this paper, AEKOC is enabled to utilize privileged information, which is generally ignored by AEKOC or any traditional machine learning technique but usually present in human learning. For this purpose, we have combined learning using privileged information (LUPI) framework with AEKOC, and proposed a classifier, which is referred to as AEKOC+. Privileged information is only available during training but not during testing. Therefore, AEKOC is unable to utilize this information for building the model. However, AEKOC+ can efficiently handle the privileged information due to the inclusion of the LUPI framework with AEKOC. Experiments have been conducted on MNIST dataset and on various other datasets from UCI machine learning repository, which demonstrates the superiority of AEKOC+ over AEKOC. Our formulation shows that AEKOC does not utilize the privileged features in learning; however, formulation of AEKOC+ helps it in learning from the privileged features differently from other available features and improved generalization performance of AEKOC. Moreover, AEKOC+ also outperformed two LUPI framework–based one-class classifiers (i.e., OCSVM+ and SSVDD+).

Face anti-spoofing is an important part of the face recognition algorithm that aims to prevent face presentation attacks. To facilitate face anti-spoofing research, this paper introduces a novel method in which a simple model provides enhanced performance on the RGB modality images of the CASIA-SURF dataset. Initially, a simple model based on ResNet-50 architecture is proposed that uses three residual layers of ResNet-50 along with a single neuron at the end for binary classification. Meanwhile, to judge the baseline performance, the proposed Resnet-50-based model is trained on the RGB and Depth images, separately. Later, to increase the performance of the proposed ResNet-50-based model on the RGB modality images, a novel Learning Using Privileged Information (LUPI) algorithm is applied. The LUPI contains Depth images and helps to train the inferior model with the help of a pre-trained superior model. Detailed simulations on the CASIA-SURF dataset reveal that an intelligent combination of the ResNet50 based model with the LUPI paradigm works exceptionally well and significantly improves the performance on the RGB modality of the images. The proposed model is less complex and can be deployed in resource-constrained environments.

Face verification is an important task to verify people’s identities, which has sample pairs and labels with only side information about whether the two images in a pair are from the same subject or not instead of a specific label for each sample. This paper is motivated by the limitations of previous efforts that the deep neural networks with only the RGB feature are not to intra-class noise (e.g., illumination), and the deep networks with both the RGB and depth feature are not to data availability of the depth images in applications. To overcome these limitations, we focus on developing a double robust face verification architecture using the Siamese convolutional neural network (SCNN). In particular, our goal is to propose an SCNN with privileged information (SCNN+), which is inspired by Learning Using Privileged Information (LUPI), through incorporating additional depth feature along with the RGB feature in the training stage to provide privileged information to restrain the RGB feature’s prediction error. We conduct experiments to evaluate the performance of the proposed SCNN+ architecture and compare it with different categories of state-of-the-art baselines on two real-world RGB-D face datasets. The evaluation results demonstrate that SCNN+ significantly outperforms all types of baselines.

Multi-target regression is the problem where each instance is associated with multiple continuous target outputs simultaneously. Its major challenges arise with jointly exploring the complex input-output relationships and inter-target correlations. One representative approach is to build many independent single-target Support Vector Regression (SVR) models for each output target which can capture complex input-output relationships via the kernel trick. However, it does not involve inter-target correlations to improve the performance. Meanwhile, there are also many regularization-based methods which mainly explore the linear inter-target correlations, e.g., a low-rank constraint on the parameter matrix. However, in practice, it might be restrictive to assume the targets to be linearly related, and allowing for nonlinear relationships is a challenge. Motivated by Learning Using Privileged Information (LUPI), we propose a novel privileged multi-target support vector regression (MT-PSVR) model which can jointly explore the complex input-output relationships and nonlinear inter-target correlations. It explicitly explores inter-target correlations by viewing other targets as privileged information when training each target model. Besides, it can naturally use the kernel trick to explore both the complex input-output relationships and nonlinear inter-target correlations. Experimental results on many benchmark datasets validate the effectiveness of our approach.

Most of the facial expression recognition methods consider that both training and testing data are equally distributed. As facial image sequences may contain information for heterogeneous sources, facial data may be asymmetrically distributed between training and testing, as it may be difficult to maintain the same quality and quantity of information. In this work, we present a novel classification method based on the learning using privileged information (LUPI) paradigm to address the problem of facial expression recognition. We introduce a probabilistic classification approach based on conditional random fields (CRFs) to indirectly propagate knowledge from privileged to regular feature space. Each feature space owns specific parameter settings, which are combined together through a Gaussian prior, to train the proposed t-CRF+ model and allow the different tasks to share parameters and improve classification performance. The proposed method is validated on two challenging and publicly available benchmarks on facial expression recognition and improved the state-of-the-art methods in the LUPI framework.

Affective Computing (AC) has made significant progress with the advent of deep learning, yet a persistent challenge remains: the reliable transfer of affective models from controlled laboratory settings (in-vitro) to uncontrolled real-world environments (in-vivo). To address this challenge we introduce the Privileged Contrastive Pretraining (PriCon) framework according to which models are first pretrained via supervised contrastive learning (SCL) and then act as teacher models within a Learning Using Privileged Information (LUPI) framework. PriCon both leverages privileged information during training and enhances the robustness of derived affect models via SCL. Experiments conducted on two benchmark affective corpora, RECOLA and AGAIN, demonstrate that models trained using PriCon consistently outperform LUPI and end to end models. Remarkably, in many cases, PriCon models achieve performance comparable to models trained with access to all modalities during both training and testing. The findings underscore the potential of PriCon as a paradigm towards further bridging the gap between in-vitro and in-vivo affective modelling, offering a scalable and practical solution for real-world applications.

Traffic anomaly detection is primarily concerned with identifying malicious traffic patterns in a much larger stream of benign traffic. Traditionally, this is achieved by selecting a very specialized set of traffic-based features that are used for both training a model, as well as for detection at runtime. This paper introduces a novel method of anomaly detection that breaks the assumption that the same set of features should be used for both training and runtime detection. Building upon the concept of Learning Using Privileged Information (LUPI), this paper shows how to build a detection model that uses an extended set of features, some that are available only during training, and some also available at runtime, with results significantly superior to the traditional learning methods. We demonstrate the usage of the privileged features in detecting malicious botnet activities in a network, activities shown to be challenging to detect in past works.

Due to the large number of malicious software (malware) and the large variety among them, automated detection and analysis using machine learning techniques have become more and moreimportant for network and computer security.An often encountered scenario in these security applications is that training examples are scarce but unlabeled data are abundant. Semi-supervised learning where both labeled and unlabeled data are used to learn a good model quickly is a natural choice under such condition.We investigate semi-supervised classification for malware categorization.We observed that malware data have specific characteristics and that they are noisy. Off-the-shelf semi-supervised learning may not work well in this case. We proposed a semi supervised approach that addresses the problems with malware data and can provide better classification. We conducted a set of experiments to test and compare our method to others. The experimental results show that semi-supervised classification is a promising direction for malware classification. Our method achieved more than 90% accuracy when there were only a few number of training examples. The results also indicates that modifications are needed to make semi-supervised learning work with malware data. Otherwise, semi-supervised classification may perform worse than classifiers trained on only the labeled data.

Protein-protein interaction (PPI) plays an extremely important role in almost all life activities. The study of PPI has always been an important issue in the field of biomedicine. Extracting PPI information from the literature can provide important references for related research. In order to build an automated PPI extraction system, labeled corpora are required. However, annotated corpora are very limited, and annotating corpora is a time-consuming, labor-intensive, and costly task. The amount of unlabeled data is huge and easy to obtain, so it is of great significance to apply semi-supervised learning to PPI extraction. Existing semi-supervised methods cannot make good use of the information in unlabeled data. Therefore, a semi-supervised PPI extraction method based on knowledge distillation and virtual adversarial training is proposed in this work, which enhances the modeling ability of the target model for text features through knowledge distillation, and conducts semi-supervised learning through virtual adversarial training to better utilize a large number of unlabeled data. Experiments on public datasets for PPI extraction show that the proposed method can achieve competitive performance while using only a small amount of labeled data.