Modeling Adversarial Learning as Nested Stackelberg Games

Human voices contain a significant number of sources of variability. This fact can result in speaker's intrinsic characteristics, such as gender, age, accent, and other extrinsic characteristics, such as the acoustic environment, severely interfering with the performance of different speech technology applications. Along the years, several proposals have been developed in the area of speech technologies to tackle the impact of these undesired sources of variability. There is clearly a need to build robust speech representations that avoid these undesired factors, which are one of the reasons for complexity when speech applications are developed. One of these approaches is domain adaptation which has been proposed to learn a mapping between different domains, reducing the gap that may exist between them. With the rapid advances in Deep Learning, new methods for robust speech representations are being proposed. The concept of adversarial learning necessitates the creation of an intermediate latent space representation, which is invariant to a specific domain. This method formulates a minimization-maximization problem, where a primary learning optimization task is combined with a secondary domain classification task, which is optimized to perform badly. The main objective of this Thesis is to contribute to the research on supervised adversarial learning within Deep Neural Networks, as a powerful way to obtain speech features discriminant to the application task, and invariant to undesired sources of variability. This goal was planned to attempt to solve the limitations of speech technologies in three different areas of application: i) the assessment of obstructive sleep apnea from speech, where the target is to model robust apnea-related representations by suppressing the undesirable effect of other patient characteristics, such age or obesity; ii) speech privacy preservation, by developing a state-of-the-art anonymization system to model speaker-invariant representations by removing personal identifiable information, such as gender or accent; and iii) automatic speech recognition to build robust speech representations invariant to challenging acoustic conditions in TV shows. The experimental results proved that adversarial learning contributes to improve the peri formance in the three analyzed speech applications: increasing the accuracy of the obstructive sleep apnea assessment after removing speaker's body mass index; obtaining high level of speech privacy and good intelligibility, being one of the best methods in the state-of-the art; and finally, achieving slight improvements in automatic speech recognition but yielding promising results to continue its exploration. Thus, the specific objectives of this Thesis can be considered fulfilled. Main contributions of this PhD Thesis resulted in several publications in Journals with JCR and peer-reviewed conferences. Three of them, each one related to the aforementioned speech applications, are advocated for the compendium presentation of this Thesis according to the requirements demanded by the Universidad Politécnica de Madrid. An additional, but no less significant, contribution is an in-depth review of the state-ofthe-art adversarial learning techniques, which leads us to study the evolution of this technique, helping us to determine that it is a competitive approach to develop domain-invariant features in speech applications. Asimismo, me gustaría agradecer a los revisores de esta tesis, su tiempo y predisposición, para mejorar la calidad de esta memoria, y por las valoraciones presentadas. Dr. Rubén San Segundo, Dra. Ascensión Gallardo, y Dr. Doroteo Torre, gracias. Mi agradecimiento a todos los miembros de Sigma Technologies y Tax Planning, especialmente al Dr. Daniel Tapias, quién apoyó el inicio de este camino dentro de la empresa. A Javier, eterno compañero, con el que hice los primeros pinitos en un congreso. A Richard, por su tiempo y ayuda revisando esta memoria escrita en su idioma natal. A Rafael, quien me dio la oportunidad de tutorizar su TFM y aprender juntos. Y a todos aquellos antiguos compañeros que se han quedado con un sitio de amigo/a. Siempre guardo una mención especial para mi amigo Fulgencio, un hermano, gracias por tu apoyo y tus consejos a lo largo de estos años. Gracias también a mis amigos David, Guille, Mario, Raúl, Ángel y Jorge, la mayoría de ellos han convertido mis viernes de regreso a Madrid, en una cita obligada. vii No pueden faltar los integrantes de la familia del "Alcobendas United": Marco, Isma, Robert, Nando, Diego, Rodri, Miguel, Jose, Raúl. Siempre dispuestos a repartir alegría en la jornada del domingo. A los de siempre: Álvaro, Rubén, Nacho y Sergio. Seguimos creciendo juntos. Y a María R., con quien la distancia no es un obstáculo para mantenerse actualizados. Tampoco quiero olvidarme de los que llegaron no hace mucho, pero han contribuido a que Tarragona se convierta en nuestra casa: Arantxa,

There is a strong need for synthetic yet realistic distribution system test data sets that are as diverse, large, and complex to solve as real systems. Such data sets can facilitate the development of advanced algorithms and the assessment of emerging distributed energy resources while avoiding the need to acquire proprietary critical infrastructure or private data. Such synthetic data sets, however, are useful only if they are realistic enough to look and behave similarly to actual systems. This paper presents a comprehensive framework for validating synthetic distribution data sets using a three-pronged statistical, operational, and expert validation approach. It also presents a set of statistical and operational metric targets for achieving realistic data sets based on detailed characterization of more than 10,000 real U.S. utility feeders. The paper demonstrates the use of the proposed validation approach to validate three large-scale synthetic data sets developed by the authors representing Santa Fe, New Mexico; Greensboro, North Carolina; and the San Francisco Bay Area, California.

Deep-Learning (DL) methods have been successfully applied in the bearing fault diagnosis field. However, previous methods mainly focus on two assumptions: 1) training (source) and testing (target) data are sufficient with labels; 2) the distribution of the training and testing data are identical, which are obviously easy to be violated in the industry. Recently, domain adaptation has been successfully adopted to tackle with problem of insufficient labeled data and distribution discrepancies. However, most of the previous methods only focus on minimizing global distribution discrepancies without considering the conditional distribution. To remedy the limitation, a novel hybrid method combing Simplified Convolutional vision transformer and Domain Adaptation (SCDA) is proposed for aligning the distribution across global and subdomains in source and target data. Firstly, a convolutional vision transformer is utilized for extracting bearing feature with better generalization ability. Then, the adversarial learning and local maximum mean discrepancy (LMMD) is utilized for the global domain adaptation and subdomain adaptation to extract domain invariant features. Finally, the classifier trained by the domain invariant features is used for the fault diagnosis of target data. The proposed method is evaluated through the Paderborn University (PU) bearing dataset with the result showing its effectiveness and transferability.

The matrix factorization (MF) technique has been widely adopted for solving the rating prediction problem in recommender systems. The MF technique utilizes the latent factor model to obtain static user preferences (user latent vectors) and item characteristics (item latent vectors) based on historical rating data. However, in the real world, user preferences are not static but full of dynamics. Though there are several previous works that addressed this time-varying issue of user preferences, it seems (to the best of our knowledge) that none of them are specifically designed for tracking concept drift in individual user preferences . Motivated by this, we develop a temporal MF approach for tracking concept drift in each individual user latent vector. There are two key innovative steps in our approach: 1) we develop a modified stochastic gradient descent method to learn an individual user latent vector at each time step and 2) by Lasso regression, we learn a linear model for the transition of the individual user latent vectors. We test our method on a synthetic data set and several real data sets. In comparison with the original MF, our experimental results show that our temporal method is able to achieve lower root mean square errors (RMSEs) for both the synthetic and real data sets. One interesting finding is that the performance gain in RMSE is mostly from those users who indeed have concept drift in their user latent vectors at the time of prediction. In particular, for the synthetic data set and the Ciao data set, there are quite a few users with that property and the performance gains for these two data sets are roughly 20% and 5%, respectively.

Learning an effective distance measurement between sample pairs plays an important role in visual analysis, where the training procedure largely relies on hard negative samples. However, hard negative samples usually account for the tiny minority in the training set, which may fail to fully describe the data distribution close to the decision boundary. In this paper, we present a deep adversarial metric learning (DAML) framework to generate synthetic hard negatives from the original negative samples, which is widely applicable to existing supervised deep metric learning algorithms. Different from existing sampling strategies which simply ignore numerous easy negatives, our DAML aim to exploit them by generating synthetic hard negatives adversarial to the learned metric as complements. We simultaneously train the feature embedding and hard negative generator in an adversarial manner, so that adequate and targeted synthetic hard negatives are created to learn more precise distance metrics. As a single transformation may not be powerful enough to describe the global input space under the attack of the hard negative generator, we further propose a deep adversarial multi-metric learning (DAMML) method by learning multiple local transformations for more complete description. We simultaneously exploit the collaborative and competitive relationships among multiple metrics, where the metrics display unity against the generator for effective distance measurement as well as compete for more training data through a metric discriminator to avoid overlapping. Extensive experimental results on five benchmark datasets show that our DAML and DAMML effectively boost the performance of existing deep metric learning approaches through adversarial learning.

Generation of synthetic full-scale burst test data for corroded pipelines using the tabular generative adversarial network

Analyzing the dynamics of information diffusion cascades and accurately predicting their behavior holds significant importance in various applications. In this paper, we concentrate specifically on a recently introduced contrastive cascade graph learning framework, for the task of predicting cascade popularity. This framework follows a pre-training and fine-tuning paradigm to address cascade prediction tasks. In a previous study, the transferability of pre-trained models within the contrastive cascade graph learning framework was examined solely between two social media datasets. However, in our present study, we comprehensively evaluate the transferability of pre-trained models across 13 real datasets and six synthetic datasets. We construct several pre-trained models using real cascades and synthetic cascades generated by the independent cascade model and the Profile model. Then, we fine-tune these pre-trained models on real cascade datasets and evaluate their prediction accuracy based on the mean squared logarithmic error. The main findings derived from our results are as follows. (1) The pre-trained models exhibit transferability across diverse types of real datasets in different domains, encompassing different languages, social media platforms, and diffusion time scales. (2) Synthetic cascade data prove effective for pre-training purposes. The pre-trained models constructed with synthetic cascade data demonstrate comparable effectiveness to those constructed using real data. (3) Synthetic cascade data prove beneficial for fine-tuning the contrastive cascade graph learning models and training other state-of-the-art popularity prediction models. Models trained using a combination of real and synthetic cascades yield significantly lower mean squared logarithmic error compared to those trained solely on real cascades. Our findings affirm the effectiveness of synthetic cascade data in enhancing the accuracy of cascade popularity prediction.

Learning distance metrics is a fundamental problem in machine learning. Previous distance-metric learning research assumes that the training and test data are drawn from the same distribution, which may be violated in practical applications. When the distributions differ, a situation referred to as covariate shift, the metric learned from training data may not work well on the test data. In this case the metric is said to be inconsistent. In this paper, we address this problem by proposing a novel metric learning framework known as consistent distance metric learning (CDML), which solves the problem under covariate shift situations. We theoretically analyze the conditions when the metrics learned under covariate shift are consistent. Based on the analysis, a convex optimization problem is proposed to deal with the CDML problem. An importance sampling method is proposed for metric learning and two importance weighting strategies are proposed and compared in this work. Experiments are carried out on synthetic and real world datasets to show the effectiveness of the proposed method.

Domain adaptation techniques such as importance weighting modify the training data to better represent a different test data distribution, a process that may be particularly vulnerable to a malicious attack in an adversarial machine learning scenario. In this work, we explore the level of such vulnerability of importance weighting to poisoning attacks. Importance weighting, like other domain adaptation approaches, assumes that the distributions of training and test data are different but related. An intelligent adversary, having full or partial access to the training data, can take advantage of the expected difference between the distributions, and can inject well crafted malicious samples into the training data, resulting in an incorrect estimation of the importance ratio. In this work, we demonstrate the vulnerability of one of the simplest yet most effective approaches for directly estimating the importance ratio, namely, modifying the training distribution using a discriminative classifier such as the logistic regression. We test the robustness of the importance weighting process using well-controlled synthetic datasets, with an increasing number of attack points in the training data. Under the worst case perfect knowledge scenario, where the attacker has full access to the training data, we demonstrate that importance weighting can be dramatically compromised with the insertion of even a single attack point. We then show that even under limited knowledge scenario, where the attacker has limited access to the training data, the estimation process can still be significantly compromised.

Clustering is a useful exploratory technique for the analysis of gene expression data. Many different heuristic clustering algorithms have been proposed in this context. Clustering algorithms based on probability models offer a principled alternative to heuristic algorithms. In particular, model-based clustering assumes that the data is generated by a finite mixture of underlying probability distributions such as multivariate normal distributions. The issues of selecting a 'good' clustering method and determining the 'correct' number of clusters are reduced to model selection problems in the probability framework. Gaussian mixture models have been shown to be a powerful tool for clustering in many applications. We benchmarked the performance of model-based clustering on several synthetic and real gene expression data sets for which external evaluation criteria were available. The model-based approach has superior performance on our synthetic data sets, consistently selecting the correct model and the number of clusters. On real expression data, the model-based approach produced clusters of quality comparable to a leading heuristic clustering algorithm, but with the key advantage of suggesting the number of clusters and an appropriate model. We also explored the validity of the Gaussian mixture assumption on different transformations of real data. We also assessed the degree to which these real gene expression data sets fit multivariate Gaussian distributions both before and after subjecting them to commonly used data transformations. Suitably chosen transformations seem to result in reasonable fits. MCLUST is available at http://www.stat.washington.edu/fraley/mclust. The software for the diagonal model is under development. kayee@cs.washington.edu. http://www.cs.washington.edu/homes/kayee/model.

Extracting traffic information from images plays an increasingly significant role in Internet of vehicle. However, due to the high-speed movement and bumps of the vehicle, the image will be blurred during image acquisition. In addition, in rainy days, as a result of the rain attached to the lens, the target will be blocked by rain, and the image will be distorted. These problems have caused great obstacles for extracting key information from transportation images, which will affect the real-time judgment of vehicle control system on road conditions, and further cause decision-making errors of the system and even have a bearing on traffic accidents. In this paper, we propose a motion-blurred restoration and rain removal algorithm for IoV based on generative adversarial network and transfer learning. Dynamic scene deblurring and image de-raining are both among the challenging classical research directions in low-level vision tasks. For both tasks, firstly, instead of using ReLU in a conventional residual block, we designed a residual block containing three 256-channel convolutional layers, and we used the Leaky-ReLU activation function. Secondly, we used generative adversarial networks for the image deblurring task with our Resblocks, as well as the image de-raining task. Thirdly, experimental results on the synthetic blur dataset GOPRO and the real blur dataset RealBlur confirm the effectiveness of our model for image deblurring. Finally, as an image de-raining task based on transfer learning, we can fine-tune the pre-trained model with less training data and show good results on several datasets used for image rain removal.

With the development of the information age, the awareness of archives informatization of governments, organizations and groups at all levels is constantly increasing. The key content extraction of scanned archive images has been widely concerned by researchers. Based on the deep learning method, we extract text and key elements (ID photo, seal, date of birth) by analyzing the archive images. Based on the real archive images data set and synthetic archive images data set, we use the PP-OCR [1] model to finish the text structured extraction and YOLOv5 [2] to train a target detection model of ID photos, seals, and dates of birth to achieve the extraction of key elements. Since the real data set cannot be made public, we use ordinary archive data for explanation.KeywordsArchive imagesSynthetic archive imagesPP-OCRYOLOv5Text structured extractionKey elements extraction

We present a robust active learning technique for situations where there are weak and adversarial oracles. Our work falls under the general umbrella of active learning in which training data is insufficient and oracles are queried to supply labels for the most informative samples to expand the training set. On top of that, we consider problems where a large percentage of oracles may be strategically lying, as in adversarial settings. We present an adversarial active learning technique that explores the duality between oracle modeling and data modeling. We demonstrate on real datasets that our adversarial active learning technique is superior to not only the heuristic majority-voting technique but one of the state-of-the-art adversarial crowdsourcing technique—Generative model of Labels, Abilities, and Difficulties (GLAD), when genuine oracles are outnumbered by weak oracles and malicious oracles, and even in the extreme cases where all the oracles are either weak or malicious. To put our technique under more rigorous tests, we compare our adversarial active learner to the ideal active learner that always receives correct labels. We demonstrate that our technique is as effective as the ideal active learner when only one third of the oracles are genuine.

Machine learning is now playing important roles in daily lives, however, the privacy leakages are increasingly getting serious in the meantime. Current solutions to the privacy issues in machine learning, like differential privacy or homomorphic encryption either could only be applied to some specific scenarios or bring huge modification to the model construction, not to mention massive efficiency loss. In this paper, we consider addressing the privacy issue in machine learning from another perspective, without modification to models or severe efficiency loss. We proposed a straightforward privacy preserving machine learning scheme, training machine learning models directly over encrypted data. Ideally, this scheme could provide privacy protection to both training data and test data. We gave it a try by applying order preserving encryption (OPE) to the scheme. We discussed the possibility of using OPE to reveal the order information confidentially for model training. Several OPE algorithms were chosen to utilize the proposed method. Finally, comprehensive experiments were deployed on both synthetic and real datasets. The experiments on real datasets show that the learning performance of several well-known classifiers on before and after encryption changes slightly. The experiments on synthetic datasets show the classifier performance could be ranked according to fidelity and reliability.

Synthetic, i.e., computer-generated imagery (CGI) data is a key component for training and validating deep-learning-based perceptive functions due to its ability to simulate rare cases, avoidance of privacy issues, and generation of pixel-accurate ground truth data. Today, physical-based rendering (PBR) engines simulate already a wealth of realistic optical effects but are mainly focused on the human perception system. Whereas the perceptive functions require realistic images modeled with sensor artifacts as close as possible toward the sensor, the training data has been recorded. This chapter proposes a way to improve the data synthesis process by application of realistic sensor artifacts. To do this, one has to overcome the domain distance between real-world imagery and the synthetic imagery. Therefore, we propose a measure which captures the generalization distance of two distinct datasets which have been trained on the same model. With this measure the data synthesis pipeline can be improved to produce realistic sensor-simulated images which are closer to the real-world domain. The proposed measure is based on the Wasserstein distance (earth mover’s distance, EMD) over the performance metric mean intersection-over-union (mIoU) on a per-image basis, comparing synthetic and real datasets using deep neural networks (DNNs) for semantic segmentation. This measure is subsequently used to match the characteristic of a real-world camera for the image synthesis pipeline which considers realistic sensor noise and lens artifacts. Comparing the measure with the well-established Fréchet inception distance (FID) on real and artificial datasets demonstrates the ability to interpret the generalization distance which is inherent asymmetric and more informative than just a simple distance measure. Furthermore, we use the metric as an optimization criterion to adapt a synthetic dataset to a real dataset, decreasing the EMD distance between a synthetic and the Cityscapes dataset from 32.67 to 27.48 and increasing the mIoU of our test algorithm () from 40.36 to$$47.63\%$$47.63%.