Cross-subject generalization for EEG decoding: a survey of deep learning methods

Learning an effective distance measurement between sample pairs plays an important role in visual analysis, where the training procedure largely relies on hard negative samples. However, hard negative samples usually account for the tiny minority in the training set, which may fail to fully describe the data distribution close to the decision boundary. In this paper, we present a deep adversarial metric learning (DAML) framework to generate synthetic hard negatives from the original negative samples, which is widely applicable to existing supervised deep metric learning algorithms. Different from existing sampling strategies which simply ignore numerous easy negatives, our DAML aim to exploit them by generating synthetic hard negatives adversarial to the learned metric as complements. We simultaneously train the feature embedding and hard negative generator in an adversarial manner, so that adequate and targeted synthetic hard negatives are created to learn more precise distance metrics. As a single transformation may not be powerful enough to describe the global input space under the attack of the hard negative generator, we further propose a deep adversarial multi-metric learning (DAMML) method by learning multiple local transformations for more complete description. We simultaneously exploit the collaborative and competitive relationships among multiple metrics, where the metrics display unity against the generator for effective distance measurement as well as compete for more training data through a metric discriminator to avoid overlapping. Extensive experimental results on five benchmark datasets show that our DAML and DAMML effectively boost the performance of existing deep metric learning approaches through adversarial learning.

Machine learning (ML) provides effective means to learn from spectrum data and solve complex tasks involved in wireless communications. Supported by recent advances in computational resources and algorithmic designs, deep learning (DL) has found success in performing various wireless communication tasks such as signal recognition, spectrum sensing and waveform design. However, ML in general and DL in particular have been found vulnerable to manipulations thus giving rise to a field of study called adversarial machine learning (AML). Although AML has been extensively studied in other data domains such as computer vision and natural language processing, research for AML in the wireless communications domain is still in its early stage. This paper presents a comprehensive review of the latest research efforts focused on AML in wireless communications while accounting for the unique characteristics of wireless systems. First, the background of AML attacks on deep neural networks is discussed and a taxonomy of AML attack types is provided. Various methods of generating adversarial examples and attack mechanisms are also described. In addition, an holistic survey of existing research on AML attacks for various wireless communication problems as well as the corresponding defense mechanisms in the wireless domain are presented. Finally, as new attacks and defense techniques are developed, recent research trends and the overarching future outlook for AML in next-generation wireless communications are discussed.

This book provided an overview of the field of adversarial machine learning. Important issues were clearly left out, some deliberately to simplify exposition, others perhaps unintentionally. The field has become quite active in recent years in no small measure due to the attention that attacks on deep learning methods have received. While we devote an entire chapter solely to adversarial deep learning, we emphasize that proper understanding of these necessitates a broader look at adversarial learning that the rest of the book provides.

Deep learning methods for person identification based on electroencephalographic (EEG) brain activity encounters the problem of exploiting the temporally correlated structures or recording session specific variability within EEG. Furthermore, recent methods have mostly trained and evaluated based on single session EEG data. We address this problem from an invariant representation learning perspective. We propose an adversarial inference approach to extend such deep learning models to learn session-invariant person-discriminative representations that can provide robustness in terms of longitudinal usability. Using adversarial learning within a deep convolutional network, we empirically assess and show improvements with our approach based on longitudinally collected EEG data for person identification from half-second EEG epochs.

Path-based multi-hop reasoning over knowledge graph for answering questions via adversarial reinforcement learning

Deep understanding of urban mobility is of great significance for many real-world applications, such as urban traffic management and autonomous driving. This thesis develops deep learning methodologies to extract vehicle counts from streaming realtime video captured by multiple low resolution web cameras and construct maps of traffic density in a city environment; in particular, we focus on cameras installed in the Manhattan borough of NYC. The large-scale videos from these web cameras have low spatial and temporal resolution, high occlusion, large perspective, and variable environment conditions, making most existing methods to lose their efficacy. To overcome these challenges, the thesis develops several techniques: 1. a block-level regression model with a rank constraint to map the dense image feature into vehicle densities; 2. a deep multi-task learning framework based on fully convolutional neural networks to jointly learn vehicle density and vehicle count; 3. deep spatio-temporal networks for vehicle counting to incorporate temporal information of the traffic flow; and 4. multi-source domain adaptation mechanisms with adversarial learning to adapt the deep counting model to multiple cameras. To train and validate the proposed system, we have collected a largescale webcam traffic dataset CityCam that contains 60 million frames from 212 webcams installed in key intersections of NYC. Of there, 60; 000 frames have been annotated with rich information, leading to about 900; 000 annotated objects. To the best of our knowledge, it is the first and largest webcam traffic dataset with such large number of elaborate annotations. The proposed methods are integrated into the CityScapeEye system that has been extensively evaluated and compared to existing techniques on different counting tasks and datasets, with experimental results demonstrating the effectiveness and robustness of CityScapeEye.

Untargeted white-box adversarial attack with heuristic defence methods in real-time deep learning based network intrusion detection system

The lack of training samples is one of the main factors affecting the development of deep learning methods. Deep learning models often fail to learn useful features and have serious over-fitting problems when lacking of training data. In this work, we exploit two popular unsupervised learning techniques: adversarial learning and self-supervised learning, which is aimed at mine more useful representations and relieve over-fitting problems. Our training scheme is mainly divided into three steps. Firstly, we train a self-supervision network with unsupervised learning to extract obvious features from our liver lesion samples and these features will be transferred to next step. Secondly, we use the final output feature map generated by self-supervision network to train a discriminator by adversarial learning. Finally, the backbone network is trained under the constraint of discriminator and classifier. Our main idea is to train a discriminator with adversarial learning and self-supervised learning. Then, we use the discriminator to constrain the backbone network, which is aimed to reduce the backbone network solution search space. In particular, Different from generating data with GAN, we use GAN to feature adversarial learning for feature augmentation. Our experiments on liver lesion classification in CT show an average accuracy as 92.51% compared with the baseline training scheme, which demonstrates our proposed method can mime useful features and relieve over-fitting problem. It can assist physicians in the early detection and treatment of liver lesions.KeywordsSelf-supervised learningAdversarial learningLiver lesions

Human voices contain a significant number of sources of variability. This fact can result in speaker's intrinsic characteristics, such as gender, age, accent, and other extrinsic characteristics, such as the acoustic environment, severely interfering with the performance of different speech technology applications. Along the years, several proposals have been developed in the area of speech technologies to tackle the impact of these undesired sources of variability. There is clearly a need to build robust speech representations that avoid these undesired factors, which are one of the reasons for complexity when speech applications are developed. One of these approaches is domain adaptation which has been proposed to learn a mapping between different domains, reducing the gap that may exist between them. With the rapid advances in Deep Learning, new methods for robust speech representations are being proposed. The concept of adversarial learning necessitates the creation of an intermediate latent space representation, which is invariant to a specific domain. This method formulates a minimization-maximization problem, where a primary learning optimization task is combined with a secondary domain classification task, which is optimized to perform badly. The main objective of this Thesis is to contribute to the research on supervised adversarial learning within Deep Neural Networks, as a powerful way to obtain speech features discriminant to the application task, and invariant to undesired sources of variability. This goal was planned to attempt to solve the limitations of speech technologies in three different areas of application: i) the assessment of obstructive sleep apnea from speech, where the target is to model robust apnea-related representations by suppressing the undesirable effect of other patient characteristics, such age or obesity; ii) speech privacy preservation, by developing a state-of-the-art anonymization system to model speaker-invariant representations by removing personal identifiable information, such as gender or accent; and iii) automatic speech recognition to build robust speech representations invariant to challenging acoustic conditions in TV shows. The experimental results proved that adversarial learning contributes to improve the peri formance in the three analyzed speech applications: increasing the accuracy of the obstructive sleep apnea assessment after removing speaker's body mass index; obtaining high level of speech privacy and good intelligibility, being one of the best methods in the state-of-the art; and finally, achieving slight improvements in automatic speech recognition but yielding promising results to continue its exploration. Thus, the specific objectives of this Thesis can be considered fulfilled. Main contributions of this PhD Thesis resulted in several publications in Journals with JCR and peer-reviewed conferences. Three of them, each one related to the aforementioned speech applications, are advocated for the compendium presentation of this Thesis according to the requirements demanded by the Universidad Politécnica de Madrid. An additional, but no less significant, contribution is an in-depth review of the state-ofthe-art adversarial learning techniques, which leads us to study the evolution of this technique, helping us to determine that it is a competitive approach to develop domain-invariant features in speech applications. Asimismo, me gustaría agradecer a los revisores de esta tesis, su tiempo y predisposición, para mejorar la calidad de esta memoria, y por las valoraciones presentadas. Dr. Rubén San Segundo, Dra. Ascensión Gallardo, y Dr. Doroteo Torre, gracias. Mi agradecimiento a todos los miembros de Sigma Technologies y Tax Planning, especialmente al Dr. Daniel Tapias, quién apoyó el inicio de este camino dentro de la empresa. A Javier, eterno compañero, con el que hice los primeros pinitos en un congreso. A Richard, por su tiempo y ayuda revisando esta memoria escrita en su idioma natal. A Rafael, quien me dio la oportunidad de tutorizar su TFM y aprender juntos. Y a todos aquellos antiguos compañeros que se han quedado con un sitio de amigo/a. Siempre guardo una mención especial para mi amigo Fulgencio, un hermano, gracias por tu apoyo y tus consejos a lo largo de estos años. Gracias también a mis amigos David, Guille, Mario, Raúl, Ángel y Jorge, la mayoría de ellos han convertido mis viernes de regreso a Madrid, en una cita obligada. vii No pueden faltar los integrantes de la familia del "Alcobendas United": Marco, Isma, Robert, Nando, Diego, Rodri, Miguel, Jose, Raúl. Siempre dispuestos a repartir alegría en la jornada del domingo. A los de siempre: Álvaro, Rubén, Nacho y Sergio. Seguimos creciendo juntos. Y a María R., con quien la distancia no es un obstáculo para mantenerse actualizados. Tampoco quiero olvidarme de los que llegaron no hace mucho, pero han contribuido a que Tarragona se convierta en nuestra casa: Arantxa,

Generalized Wireless Adversarial Deep Learning

Recently, domain-adaptation based transfer learning has been extensively studied and successfully achieved promising results in addressing the domain drift in closed-set scenarios. However, in the bridge damage diagnosis field, the target data-sets collected from bridges frequently present samples of new damages that were not observed in the source domain, which is known as the open-set domain adaptation problem. To address this problem, this paper proposes a new open-set deep transfer learning algorithm based on joint weighted sub-domain adaptation. First, a joint weighting mechanism is proposed based on adversarial learning and fuzzy theory to represent the similarity of target-domain samples with source-domain classes, and explore the method of separating the known and unknown classes in the target domain to solve the negative transfer problem. Then, to capture the fine-grained transferable information, a sub-domain adaptation algorithm based on minimizing the multi-channel multi-kernel weighted local maximum mean discrepancy (MCMK-WLMMD) is proposed to align the corresponding sub-domains in the two domains. Finally, membership is introduced to build an unsupervised fuzzy clustering model with evaluation indicator to recognize multiple unknown damages. Extensive experiments on open-set transfer tasks between three bridges verify the effectiveness of the algorithm.

Machinery fault diagnosis is crucial to ensure production safety. Deep learning has been widely investigated in intelligent fault diagnosis. However, the mechanical equipment generally runs in a complex and polytropic condition. It is challenging to perform machinery fault diagnosis under polytopic conditions. In this article, a novel multisource domain adaptation method is proposed for machinery fault diagnosis under different working conditions. First, a deep negative correlation multisource domains adaptation network (DNC-MDAN) is proposed, where the information from multisource domains is transferred into target domain based on the multisource feature alignment and adversarial learning. Second, in order to make full of the multisource domains information, an ensemble classifier corresponding to multisource domains is developed, where a DNC learning is performed in the ensemble classifier. Third, an end-to-end feature generator, discrete cosine convolutional block, is proposed in DNC-MDAN, where discrete cosine transform is embedded in the network smoothly for noise reduction. In order to validate the effectiveness of DNC-MDAN for machinery fault diagnosis, three cases are considered in this article. In comparison with other transfer learning methods, DNC-MDAN has a better fault diagnosis performance.

Real-world entities' behaviors, associated with their side information, are often recorded over time as asynchronous event sequences. Such event sequences are the basis of many practical applications, neural spiking train study, earth quack prediction, crime analysis, infectious disease diffusion forecasting, condition-based preventative maintenance, information retrieval and behavior-based network analysis and services, etc. Temporal point process (TPP) is a principled mathematical tool for the modeling and learning of asynchronous event sequences, which captures the instantaneous happening rate of the events and the temporal dependency between historical and current events. TPP provides us with an interpretable model to describe the generative mechanism of event sequences, which is beneficial for event prediction and causality analysis. Recently, it has been shown that TPP has potentials to many machine learning and data science applications and can be combined with other cutting-edge machine learning techniques like deep learning, reinforcement learning, adversarial learning, and so on. We will start with an elementary introduction of TPP model, including the basic concepts of the model, the simulation method of event sequences; in the second part of the tutorial, we will introduce typical TPP models and their traditional learning methods; in the third part of the tutorial, we will discuss the recent progress on the modeling and learning of TPP, including neural network-based TPP models, generative adversarial networks (GANs) for TPP, and deep reinforcement learning of TPP. We will further talk about the practical application of TPP, including useful data augmentation methods for learning from imperfect observations, typical applications and examples like healthcare and industry maintenance, and existing open source toolboxes.

Recently, deep learning based approaches have emerged as indispensable tools to perform big data analytics. Normally, deep learning models are first trained with a supervised method and then deployed to execute various tasks. The supervised method involves extensive human efforts to collect and label the large-scale dataset, which becomes impractical in the big data era where raw data is largely un-labeled and uncategorized. Fortunately, the adversarial learning, represented by Generative Adversarial Network (GAN), enjoys a great success on the unsupervised learning. However, the distinct features of GAN, such as massive computing phases and non-traditional convolutions challenge the existing deep learning accelerator designs. In this work, we propose the first holistic solution for accelerating the unsupervised GAN-based Deep Learning. We overcome the above challenges with an algorithm and architecture co-design approach. First, we optimize the training procedure to reduce on-chip memory consumption. We then propose a novel time-multiplexed design to efficiently map the abundant computing phases to our microarchitecture. Moreover, we design high-efficiency dataflows to achieve high data reuse and skip the zero-operand multiplications in the non-traditional convolutions. Compared with traditional deep learning accelerators, our proposed design achieves the best performance (average 4.3X) with the same computing resource. Our design also has an average of 8.3X speedup over CPU and 6.2X energy-efficiency over NVIDIA GPU.

Health state estimation of cutting tools is a key component for maintaining the reliability and stability of the manufacturing process. Although deep learning (DL) based tool wear indirect monitoring methods have been presented, most of them require a substantial amount of data to avoid high misclassification rates. To this end, this article proposes a novel estimation approach based on data-enhanced deep transfer learning (TL) methodology, which integrates multivariate data enhancements, deep convolution network, and generative adversarial learning for state assessment of the tool wear under small sample size. The proposed method includes three innovative points: 1) a multi-step data enhancement method is employed to augment the wavelet scalogram data as model input; 2) a novel training process is adopted to obtain optimal hyperparameters of TL-AlexNet and TL-VGGNet-16 model; 3) a deep transfer learning model for the enhanced data is presented to adaptively estimate the health state of the cutting tools. The proposed approach only requires a small quantity of data to achieve satisfactory estimation performance compared with other DL methods through experimental validation, and the average prediction accuracy for each stage is greater than 95%, which demonstrates the effectiveness and superiority of the presented model.