Adversarial deep learning on digital media security and forensics

Learning an effective distance measurement between sample pairs plays an important role in visual analysis, where the training procedure largely relies on hard negative samples. However, hard negative samples usually account for the tiny minority in the training set, which may fail to fully describe the data distribution close to the decision boundary. In this paper, we present a deep adversarial metric learning (DAML) framework to generate synthetic hard negatives from the original negative samples, which is widely applicable to existing supervised deep metric learning algorithms. Different from existing sampling strategies which simply ignore numerous easy negatives, our DAML aim to exploit them by generating synthetic hard negatives adversarial to the learned metric as complements. We simultaneously train the feature embedding and hard negative generator in an adversarial manner, so that adequate and targeted synthetic hard negatives are created to learn more precise distance metrics. As a single transformation may not be powerful enough to describe the global input space under the attack of the hard negative generator, we further propose a deep adversarial multi-metric learning (DAMML) method by learning multiple local transformations for more complete description. We simultaneously exploit the collaborative and competitive relationships among multiple metrics, where the metrics display unity against the generator for effective distance measurement as well as compete for more training data through a metric discriminator to avoid overlapping. Extensive experimental results on five benchmark datasets show that our DAML and DAMML effectively boost the performance of existing deep metric learning approaches through adversarial learning.

Deep learning methods for person identification based on electroencephalographic (EEG) brain activity encounters the problem of exploiting the temporally correlated structures or recording session specific variability within EEG. Furthermore, recent methods have mostly trained and evaluated based on single session EEG data. We address this problem from an invariant representation learning perspective. We propose an adversarial inference approach to extend such deep learning models to learn session-invariant person-discriminative representations that can provide robustness in terms of longitudinal usability. Using adversarial learning within a deep convolutional network, we empirically assess and show improvements with our approach based on longitudinally collected EEG data for person identification from half-second EEG epochs.

Despite the great success, existing regression clustering methods based on shallow models are vulnerable due to: (1) They often pay no attention to the combination between learning representations and clustering, thus resulting in unsatisfactory clustering performance. (2) They ignore the relationship of data distribution and target distribution such that those methods are noise and illumination-change sensitive. (3) These nonlinear regression methods usually impose the hard constraint to minimize the mismatch between the discrete cluster assignment matrix and latent representations, which leads to over-fitting. In this paper, we utilize deep adversarial regression to tackle these problems and formulate regression based clustering by deep adversarial learning (RCDA). By seamlessly combining with the stacked autoencoder, the proposed model integrates learning deep nonlinear latent representation and clustering in a unified framework. Specifically, RCDA uses a kind of relax constraint between latent representations and continuous cluster assignment matrix to avoid over-fitting, and simultaneously utilizes the t-SNE algorithm and adversarial learning to analyze data distribution and target distribution so that improve representations learning. Experimental results on public benchmark datasets demonstrate that the proposed architecture achieves better performance than state-of-the-art clustering models in image clustering task.

The design of a new therapeutic agent is a time-consuming and expensive process. The rise of machine intelligence provides a grand opportunity of expeditiously discovering novel drug candidates through smart search in the vast molecular structural space. In this paper, we propose a new approach called adversarial deep evolutionary learning (ADEL) to search for novel molecules in the latent space of an adversarial generative model and keep improving the latent representation space. In ADEL, a custom-made adversarial autoencoder (AAE) model is developed and trained under a deep evolutionary learning (DEL) process. This involves an initial training of the AAE model, followed by an integration of multi-objective evolutionary optimization in the continuous latent representation space of the AAE rather than the discrete structural space of molecules. By using the AAE, an arbitrary distribution can be provided to the training of AAE such that the latent representation space is set to that distribution. This allows for a starting latent space from which new samples can be produced. Throughout the process of learning, new samples of high-quality are generated after each iteration of training and then added back into the full dataset. Therefore, allowing for a more comprehensive procedure of understanding the data structure. This combination of evolving data and continuous learning not only enables improvement in the generative model, but the data as well. By comparing ADEL to the previous work in DEL, we see that ADEL can obtain better property distributions.

Adversarial deep reinforcement learning based robust depth tracking control for underactuated autonomous underwater vehicle

Conversational agents are getting more popular and applied in a wide range of practical application areas. The main task of these agents is not only to generate context-appropriate responses to a given query but also to make the conversation human-like. Thanks to the ability of deep learning based models in natural language modeling, recent studies have made progress in designing conversational agents that can provide more semantically accurate responses. However, the naturalness in such conversation setting has not been given adequate attention in these studies. This paper aims to incorporate both important criteria of accuracy and naturalness of conversation in developing a new model for conversational agents. To this end, inspired by the idea of Turing test and the idea of adversarial learning strategy, we propose to design a model based on generative deep neural networks that interestingly allow to generate accurate responses optimized by the mechanics of imitating human-generated conversations. Experimental results demonstrate that the proposed models produce more natural and accurate responses, yielding significant gains in BLEU scores.

We study two important concepts in adversarial deep learning---adversarial training and generative adversarial network (GAN). Adversarial training is the technique used to improve the robustness of discriminator by combining adversarial attacker and discriminator in the training phase. GAN is commonly used for image generation by jointly optimizing discriminator and generator. We show these two concepts are indeed closely related and can be used to strengthen each other---adding a generator to the adversarial training procedure can improve the robustness of discriminators, and adding an adversarial attack to GAN training can improve the convergence speed and lead to better generators. Combining these two insights, we develop a framework called Rob-GAN to jointly optimize generator and discriminator in the presence of adversarial attacks---the generator generates fake images to fool discriminator; the adversarial attacker perturbs real images to fool the discriminator, and the discriminator wants to minimize loss under fake and adversarial images. Through this end-to-end training procedure, we are able to simultaneously improve the convergence speed of GAN training, the quality of synthetic images, and the robustness of discriminator under strong adversarial attacks. Experimental results demonstrate that the obtained classifier is more robust than the state-of-the-art adversarial training approach, and the generator outperforms SN-GAN on ImageNet-143.

The presence of defects in substation equipment is a major factor affecting the safety of power transmission. Therefore, timely and accurate detection of these defects is crucial. As intelligent inspection robots advance, using mainstream object detection models to diagnose surface defects in substation equipment has become a focal point of current research. However, the lack of defect image data is one of the main factors affecting the accuracy of supervised deep learning-based defect detection models. To address the issue of insufficient training data for defect images with complex backgrounds, such as rust and surface oil leakage in substation equipment, which leads to the poor performance of detection models, this paper proposes a novel adversarial deep learning model for substation defect image generation: the Abnormal Defect Detection Generative Adversarial Network (ADD-GAN). Unlike existing generative adversarial networks, this model generates defect images based on effectively segmented local areas of substation equipment images, avoiding image distortion caused by global style changes. Additionally, the model uses a joint discriminator for both overall images and defect images to address the issue of low attention to local defect areas, thereby reducing the loss of image features. This approach enhances the overall quality of generated images as well as locally generated defect images, ultimately improving image realism. Experimental results demonstrate that the YOLOV7 object detection model trained on the dataset generated using the ADD-GAN method achieves a mean average precision (mAP) of 81.5% on the test dataset, and outperforms other image data augmentation and generation methods. This confirms that the ADD-GAN method can generate a high-fidelity image dataset of substation equipment defects.

In this research, imaging techniques such as CT and X-ray are used to locate important muscles in the shoulders and legs. Athletes who participate in sports that require running, jumping, or throwing are more likely to get injuries such as sprains, strains, tendinitis, fractures, and dislocations. One proposed automated technique has the overarching goal of enhancing recognition. This study aims to determine how to recognize the major muscles in the shoulder and leg utilizing X-ray CT images as its primary diagnostic tool. Using a shape model, discovering landmarks, and generating a form model are the steps necessary to identify injuries in key shoulder and leg muscles. The method also involves identifying injuries in significant abdominal muscles. The use of adversarial deep learning, and more specifically Deep-Injury Region Identification, can improve the ability to identify damaged muscle in X-ray and CT images. Applying the proposed diagnostic model to 150 sets of CT images, the study results show that Jaccard similarity coefficient (JSC) rate for the procedure is 0.724, the repeatability is 0.678, and the accuracy is 94.9% respectively. The study results demonstrate feasibility of using adversarial deep learning and deep-injury region identification to automatically detect severe muscle injuries in the shoulder and leg, which can enhance the identification and diagnosis of injuries in athletes, especially for those who compete in sports that include running, jumping, and throwing.

BackgroundTreatment effect prediction (TEP) plays an important role in disease management by ensuring that the expected clinical outcomes are obtained after performing specialized and sophisticated treatments on patients given their personalized clinical status. In recent years, the wide adoption of electronic health records (EHRs) has provided a comprehensive data source for intelligent clinical applications including the TEP investigated in this study.MethodWe examined the problem of using a large volume of heterogeneous EHR data to predict treatment effects and developed an adversarial deep treatment effect prediction model to address the problem. Our model employed two auto-encoders for learning the representative and discriminative features of both patient characteristics and treatments from EHR data. The discriminative power of the learned features was further enhanced by decoding the correlational information between the patient characteristics and subsequent treatments by means of a generated adversarial learning strategy. Thereafter, a logistic regression layer was appended on the top of the resulting feature representation layer for TEP.ResultThe proposed model was evaluated on two real clinical datasets collected from the cardiology department of a Chinese hospital. In particular, on acute coronary syndrome (ACS) dataset, the proposed adversarial deep treatment effect prediction (ADTEP) (0.662) exhibited 1.4, 2.2, and 6.3% performance gains in terms of the area under the ROC curve (AUC) over deep treatment effect prediction (DTEP) (0.653), logistic regression (LR) (0.648), and support vector machine (SVM) (0.621), respectively. As for heart failure (HF) case study, the proposed ADTEP also outperformed all benchmarks. The experimental results demonstrated that our proposed model achieved competitive performance compared to state-of-the-art models in tackling the TEP problem.ConclusionIn this work, we propose a novel model to address the TEP problem by utilizing a large volume of observational data from EHR. With adversarial learning strategy, our proposed model can further explore the correlational information between patient statuses and treatments to extract more robust and discriminative representation of patient samples from their EHR data. Such representation finally benefits the model on TEP. The experimental results of two case studies demonstrate the superiority of our proposed method compared to state-of-the-art methods.

One-class classification (OCC) seeks to build a machine-learning model when the negative class is either absent, poorly sampled, or not well defined. In this paper, we present a deep adversarial learning based architecture for one-class classification. Our architecture is composed of two deep neural networks, a generator and a discriminator, that are competing while collaborating with each other since it is inspired by the success of Generative Adversarial Networks (GANs). The generator network contains a Deconvolutional Neural Networks (aka. decoder), which is trained using a zero-centered Gaussian noise as the feature representation of the pseudo-negative class to learn a good representation as well as the boundary for the classifiable distortion of the target (or positive) class with the assistance from the target class. The outputs produced by the generator network are aggregated with the real positive class data samples, which are then used to train the discriminator network, whose goal is to understand the underlying concept in the positive class, and then classify the negative testing samples. The proposed architecture applies to a variety of OCC problems such as novelty detection, anomaly detection, and mobile user authentication. The experiments on MNIST and Caltech-256 images demonstrate that our architecture achieves superior results over the recent state-of-the-art approaches.

The emerging Deep fake Detection System has been a major threat to digital media integrity since its introduction in 2017; early applications of the technology have been largely based on applications of face- swapping techniques using auto encoders and generative adversarial networks (GANs).Owing to the fast pace of development of these synthetic media generation techniques, these methods have outpaced more traditional detection methods, leading to an urgent need for more adaptive solutions. The system creates a solution to the problem by designing a new self-learning detection system that does not require pre-existing datasets. Compared to traditional methods, which have to be provided with large amounts of labelled data, our implementation uses incremental machine learning algorithms that are incrementally refined based on user feedback and confidence-based learning mechanisms. By leveraging transfer learning and online learning paradigms, the platform learns to adapt to emerging deep fake variations in real-time. Results of current implementation show promising results in detecting artifacts from eye blinking patterns, facial micro-expressions and audio-visual synchronization errors. Further work will be done to incorporate transformer-based architectures for better temporal analysis, add text deep fakes detection. The project will serve to build an open source, continually improving defence against synthetic media manipulation that will further increase the authenticity and security of digital media.

Mobile user profiles are a summary of characteristics of user-specific mobile activities. Mobile user profiling is to extract a user's interest and behavioral patterns from mobile behavioral data. While some efforts have been made for mobile user profiling, existing methods can be improved via representation learning with awareness of substructures in users' behavioral graphs. Specifically, in this paper, we study the problem of mobile users profiling with POI check-in data. To this end, we first construct a graph, where a vertex is a POI category and an edge is the transition frequency of a user between two POI categories, to represent each user. We then formulate mobile user profiling as a task of representation learning from user behavioral graphs. We later develop a deep adversarial substructured learning framework for the task. This framework has two mutually-enhanced components. The first component is to preserve the structure of the entire graph, which is formulated as an encoding-decoding paradigm. In particular, the structure of the entire graph is preserved by minimizing reconstruction loss between an original graph and a reconstructed graph. The second component is to preserve the structure of subgraphs, which is formulated as a substructure detector based adversarial training paradigm. In particular, this paradigm includes a substructure detector and an adversarial trainer. Instead of using non-differentiable substructure detection algorithms, we pre-train a differentiable convolutional neural network as the detector to approximate these detection algorithms. The adversarial trainer is to match the detected substructure of the reconstructed graph to the detected substructure of the original graph. Also, we provide an effective solution for the optimization problems. Moreover, we exploit the learned representations of users for the next activity type prediction. Finally, we present extensive experimental results to demonstrate the improved performances of the proposed method.

Endoscope size is a major design constraint that must be managed with the clinical demand for high-quality illumination and imaging. Existing commercial endoscopes most often use an arc lamp to produce bright, incoherent white light, requiring large-area fiber bundles to deliver sufficient illumination power to the sample. Moreover, the power instability of these light sources creates challenges for computer vision applications. We demonstrate an alternative illumination technique using red-green-blue laser light and a data-driven approach to combat the speckle noise that is a byproduct of coherent illumination. We frame the speckle artifact problem as an image-to-image translation task solved using conditional Generative Adversarial Networks (cGANs). To train the network, we acquire images illuminated with a coherent laser diode, with a laser diode source made partially- coherent using a laser speckle reducer, and with an incoherent LED light source as the target domain. We train networks using laser-illuminated endoscopic images of ex-vivo, porcine gastrointestinal tissues, augmented by images of laser-illuminated household and laboratory objects. The network is then benchmarked against state of-the-art optical and image processing speckle reduction methods, achieving an increased peak signal-to-noise ratio (PSNR) of 4.1 db, compared to 0.7 dB using optical speckle reduction, 0.6 dB using median filtering, and 0.5 dB using non-local means. This approach not only allows for endoscopes with smaller, more efficient light sources with extremely short triggering times, but it also enables imaging modalities that require both coherent and incoherent sources, such as combined widefield and speckle ow contrast imaging in a single image frame.

This book provided an overview of the field of adversarial machine learning. Important issues were clearly left out, some deliberately to simplify exposition, others perhaps unintentionally. The field has become quite active in recent years in no small measure due to the attention that attacks on deep learning methods have received. While we devote an entire chapter solely to adversarial deep learning, we emphasize that proper understanding of these necessitates a broader look at adversarial learning that the rest of the book provides.