Application of Learning Using Privileged Information(LUPI): Botnet Detection

Alzheimer's disease (AD) is a progressive brain disorder. Machine learning models have been proposed for the diagnosis of AD at early stage. Recently, deep learning architectures have received quite a lot attention. Most of the deep learning architectures suffer from the issues of local minima, slow convergence and sensitivity to learning rate. To overcome these issues, non-iterative learning based deep randomized models especially random vector functional link network (RVFL) with direct links have proven to be successful. However, deep RVFL and its ensemble models are trained only on normal samples. In this paper, deep RVFL and its ensembles are enabled to incorporate privileged information, as the standard RVFL model and its deep models are unable to use privileged information. To fill this gap, we have incorporated learning using privileged information (LUPI) in deep RVFL model, and propose deep RVFL with LUPI framework (dRVFL+). Privileged information is available while training the models. As RVFL is an unstable classifier, we propose ensemble deep RVFL+ with LUPI framework (edRVFL+) which exploits the LUPI as well as the diversity among the base leaners for better classification. Unlike traditional ensemble approach wherein multiple base learners are trained, the proposed edRVFL+ model optimises a single network and generates an ensemble via optimization at different levels of random projections of the data. Both dRVFL+ and edRVFL+ efficiently utilise the privileged information which results in better generalization performance. In LUPI framework, half of the available features are used as normal features and rest as the privileged features. However, we propose a novel approach for generating the privileged information. We utilise different activation functions while processing the normal and privileged information in the proposed deep architectures. To the best of our knowledge, this is first time that a separate privileged information is generated. The proposed dRVFL+ and edRVFL+ models are employed for the diagnosis of Alzheimer's disease. Experimental results demonstrate the superiority of the proposed dRVFL+ and edRVFL+ models over baseline models. Thus, the proposed edRVFL+ model can be utilised in clinical setting for the diagnosis of AD.

The privileged information that is available only for the training examples and not available for test examples, is a new concept proposed by Vapnik and Vashist (Neural Netw 22(5–6):544–557, 2009). With the help of the privileged information, learning using privileged information (LUPI) (Neural Netw 22(5–6):544–557, 2009) can significantly accelerate the speed of learning. However, LUPI is a standard supervised learning method. In fact, in many real-world problems, there are also a lot of unlabeled data. This drives us to solve problems under a semi-supervised learning framework. In this paper, we propose a semi-supervised learning using privileged information (called Semi-LUPI), which can exploit both the distribution information in unlabeled data and privileged information to improve the efficiency of the learning. Furthermore, we also compare the relative importance of both types of information for the learning model. All experiments verify the effectiveness of the proposed method, and simultaneously show that Semi-LUPI can obtain superior performances over traditional supervised and semi-supervised methods.

Learning using privileged information with logistic regression on acute respiratory distress syndrome detection

In recent years, non-iterative learning approaches for kernel have received quite an attention by researchers and kernel ridge regression (KRR) approach is one of them. Recently, KRR-based Auto-Encoder is developed for the one-class classification (OCC) task and named as AEKOC. OCC is generally used for outlier or novelty detection. The brain can detect outlier just by learning from only normal samples. Similarly, OCC also uses only normal samples to train the model, and trained model can be used for outlier detection. In this paper, AEKOC is enabled to utilize privileged information, which is generally ignored by AEKOC or any traditional machine learning technique but usually present in human learning. For this purpose, we have combined learning using privileged information (LUPI) framework with AEKOC, and proposed a classifier, which is referred to as AEKOC+. Privileged information is only available during training but not during testing. Therefore, AEKOC is unable to utilize this information for building the model. However, AEKOC+ can efficiently handle the privileged information due to the inclusion of the LUPI framework with AEKOC. Experiments have been conducted on MNIST dataset and on various other datasets from UCI machine learning repository, which demonstrates the superiority of AEKOC+ over AEKOC. Our formulation shows that AEKOC does not utilize the privileged features in learning; however, formulation of AEKOC+ helps it in learning from the privileged features differently from other available features and improved generalization performance of AEKOC. Moreover, AEKOC+ also outperformed two LUPI framework–based one-class classifiers (i.e., OCSVM+ and SSVDD+).

The efficiency in breast cancer from imaging-based computer-aided diagnosis (CAD) has been revealed in recent years. As a fact, the methods grounded on a single modality constantly lack behind multimodal CAD imaging. However, owing to the restrictions of imaging devices, expressly in rural hospitals, single-modal imaging becomes a favorite in clinical practice for diagnosis. A fresh learning model trending nowadays known as learning using privileged information (LUPI) adopts additional privileged information (PI) modality to help during the training stage, but PI does not contribute in the testing stage. Meanwhile, the link exists between PI and training samples; the same is then reassigned to the learned model. We propose a LUPI-based CAD framework for breast cancer using privileged information in this work. The work offers both a classifier- or feature-level LUPI, in which the information is shifted from the additional PI modality to the diagnosis modality. A thorough comparison has been made among six classifier-level algorithms and six feature-level LUPI algorithms. The experimental results on both the acquired primary datasets show that all classifier-level and deep learning-based feature-level LUPI algorithms can enhance the performance of a single-modal imaging-based CAD for breast cancer by relocating PI.

Face anti-spoofing is an important part of the face recognition algorithm that aims to prevent face presentation attacks. To facilitate face anti-spoofing research, this paper introduces a novel method in which a simple model provides enhanced performance on the RGB modality images of the CASIA-SURF dataset. Initially, a simple model based on ResNet-50 architecture is proposed that uses three residual layers of ResNet-50 along with a single neuron at the end for binary classification. Meanwhile, to judge the baseline performance, the proposed Resnet-50-based model is trained on the RGB and Depth images, separately. Later, to increase the performance of the proposed ResNet-50-based model on the RGB modality images, a novel Learning Using Privileged Information (LUPI) algorithm is applied. The LUPI contains Depth images and helps to train the inferior model with the help of a pre-trained superior model. Detailed simulations on the CASIA-SURF dataset reveal that an intelligent combination of the ResNet50 based model with the LUPI paradigm works exceptionally well and significantly improves the performance on the RGB modality of the images. The proposed model is less complex and can be deployed in resource-constrained environments.

Learning the implicit strain reconstruction in ultrasound elastography using privileged information.

Face verification is an important task to verify people’s identities, which has sample pairs and labels with only side information about whether the two images in a pair are from the same subject or not instead of a specific label for each sample. This paper is motivated by the limitations of previous efforts that the deep neural networks with only the RGB feature are not to intra-class noise (e.g., illumination), and the deep networks with both the RGB and depth feature are not to data availability of the depth images in applications. To overcome these limitations, we focus on developing a double robust face verification architecture using the Siamese convolutional neural network (SCNN). In particular, our goal is to propose an SCNN with privileged information (SCNN+), which is inspired by Learning Using Privileged Information (LUPI), through incorporating additional depth feature along with the RGB feature in the training stage to provide privileged information to restrain the RGB feature’s prediction error. We conduct experiments to evaluate the performance of the proposed SCNN+ architecture and compare it with different categories of state-of-the-art baselines on two real-world RGB-D face datasets. The evaluation results demonstrate that SCNN+ significantly outperforms all types of baselines.

Multi-target regression is the problem where each instance is associated with multiple continuous target outputs simultaneously. Its major challenges arise with jointly exploring the complex input-output relationships and inter-target correlations. One representative approach is to build many independent single-target Support Vector Regression (SVR) models for each output target which can capture complex input-output relationships via the kernel trick. However, it does not involve inter-target correlations to improve the performance. Meanwhile, there are also many regularization-based methods which mainly explore the linear inter-target correlations, e.g., a low-rank constraint on the parameter matrix. However, in practice, it might be restrictive to assume the targets to be linearly related, and allowing for nonlinear relationships is a challenge. Motivated by Learning Using Privileged Information (LUPI), we propose a novel privileged multi-target support vector regression (MT-PSVR) model which can jointly explore the complex input-output relationships and nonlinear inter-target correlations. It explicitly explores inter-target correlations by viewing other targets as privileged information when training each target model. Besides, it can naturally use the kernel trick to explore both the complex input-output relationships and nonlinear inter-target correlations. Experimental results on many benchmark datasets validate the effectiveness of our approach.

Most of the facial expression recognition methods consider that both training and testing data are equally distributed. As facial image sequences may contain information for heterogeneous sources, facial data may be asymmetrically distributed between training and testing, as it may be difficult to maintain the same quality and quantity of information. In this work, we present a novel classification method based on the learning using privileged information (LUPI) paradigm to address the problem of facial expression recognition. We introduce a probabilistic classification approach based on conditional random fields (CRFs) to indirectly propagate knowledge from privileged to regular feature space. Each feature space owns specific parameter settings, which are combined together through a Gaussian prior, to train the proposed t-CRF+ model and allow the different tasks to share parameters and improve classification performance. The proposed method is validated on two challenging and publicly available benchmarks on facial expression recognition and improved the state-of-the-art methods in the LUPI framework.

In many human activity recognition systems the size of the unlabeled training data may be significantly large due to expensive human effort required for data annotation. Moreover, the insufficient data collection process from heterogenous sources may cause dissimilarities between training and testing data. To address these limitations, a novel probabilistic approach that combines learning using privileged information (LUPI) and active learning is proposed. A pool-based privileged active learning approach is presented for semi-supervising learning of human activities from multimodal labeled and unlabeled data. Both uncertainty and distance from the decision boundary are used as query inference strategies for selecting an unlabeled observation and querying its label. Experimental results in four publicly available datasets demonstrate that the proposed method can identify complex human activities with high accuracy.

Affective Computing (AC) has made significant progress with the advent of deep learning, yet a persistent challenge remains: the reliable transfer of affective models from controlled laboratory settings (in-vitro) to uncontrolled real-world environments (in-vivo). To address this challenge we introduce the Privileged Contrastive Pretraining (PriCon) framework according to which models are first pretrained via supervised contrastive learning (SCL) and then act as teacher models within a Learning Using Privileged Information (LUPI) framework. PriCon both leverages privileged information during training and enhances the robustness of derived affect models via SCL. Experiments conducted on two benchmark affective corpora, RECOLA and AGAIN, demonstrate that models trained using PriCon consistently outperform LUPI and end to end models. Remarkably, in many cases, PriCon models achieve performance comparable to models trained with access to all modalities during both training and testing. The findings underscore the potential of PriCon as a paradigm towards further bridging the gap between in-vitro and in-vivo affective modelling, offering a scalable and practical solution for real-world applications.

Semi-Supervised Learning (SSL) aims to leverage unlabeled data to improve performance. Due to the lack of supervised information, previous works mainly focus on how to utilize the available unlabeled data to improve the training quality. However, the estimation of the data distribution revealed by the unlabeled examples might not be accurate as their classes are unknown. Inspired by the framework of Learning Using Privileged Information (LUPI), we propose to introduce an intelligent “teacher” which can utilize the privileged information (i.e. more precise explanations of training set) to improve the performance of SSL. This developed method is named as Privileged Semi-Supervised Learning (PSSL). Moreover, our method can be efficiently solved with $\ell_{2}$ -loss. Experimentally, we compare our PSSL with several typical algorithms on three real-world datasets, and the results suggest that our approach is able to achieve the state-of-the-art performance.

This paper describes a hybrid design for intrusion detection that combines anomaly detection with misuse detection. The proposed method includes an ensemble feature selecting classifier and a data mining classifier. The former consists of four classifiers using different sets of features and each of them employs a machine learning algorithm named fuzzy belief k-NN classification algorithm. The latter applies data mining technique to automatically extract computer users' normal behavior from training network traffic data. The outputs of ensemble feature selecting classifier and data mining classifier are then fused together to get the final decision. The experimental results indicate that hybrid approach effectively generates a more accurate intrusion detection model on detecting both normal usages and malicious activities.

River water extent is essential for river hydrological surveys. Traditional methods for river water mapping often result in significant uncertainties. This paper proposes a support vector machine (SVM)-based river water mapping method that can quantify the extraction uncertainties simultaneously. Five specific bands of Landsat-8 Operational Land Imager (OLI) data were selected to construct the feature set. Considering the effect of terrain, a widely used terrain index called height above nearest drainage, calculated from the 1 arc-second Shuttle Radar Topography Mission (SRTM) digital elevation model (DEM), was also added into the feature set. With this feature set, a posterior probability SVM model was established to extract river water bodies and quantify the uncertainty with posterior probabilities. Three river sections in Northwestern China were selected as the case study areas, considering their different river characteristics and geographical environment. Then, the reliability and stability of the proposed method were evaluated through comparisons with the traditional Normalized Difference Water Index (NDWI) and modified NDWI (mNDWI) methods and validated with higher-resolution Sentinel-2 images. It was found that resultant probability maps obtained by the proposed SVM method achieved generally high accuracy with a weighted root mean square difference of less than 0.1. Other accuracy indices including the Kappa coefficient and critical success index also suggest that the proposed method outperformed the traditional water index methods in terms of river mapping accuracy and thresholding stability. Finally, the proposed method resulted in the ability to separate water bodies from hill shades more easily, ensuring more reliable river water mapping in mountainous regions.