Hybrid quantum-classical generative adversarial networks with transfer learning

Deep neural networks (DNNs), e.g., convolutional neural network (CNN), are able to learn effective features from wafer maps for dimensional reduction and feature extraction. However, very large image data are needed to train DNNs to obtain high generalization performance. It is still a difficult task due to the lack of sufficient labeled images with various defects. This article proposes a semisupervised deep transfer learning algorithm called joint feature and label adversarial network (JFLAN). JFLAN uses CNNs to extract transferable features of wafer maps and then introduces a multilayer domain adaptation and pseudolabel learning block based on the generative adversarial network (GAN). This effectively reduces the distribution discrepancy and the among-class distance of the transferable features. Finally, JFLAN transfers knowledge from wafer image source data collected offline and then achieves the goal of significantly improved accuracy of wafer defect recognition and realizes online adaptive defect recognition. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —The defect recognition on wafer maps plays a key role to recognize fault sources of the semiconductor manufacturing processes. Transfer learning is able to use the existing labeled data to assist in the classification of unlabeled data and is very effective to solve the problem of small samples and nonstationary generalization errors. In particular, the infusion of adversarial learning in transfer learning will provide a new idea for deep feature learning. This article provides a novel method based on transfer learning to implement wafer map defect recognition (WMDR) to quickly identify defect root causes for yield enhancement. This article provides a novel way for quality control of semiconductor manufacturing processes based on transfer and adversarial learning.

Recent progress in machine learning has been mainly due to the availability of large amounts of annotated data used for training complex models with deep architectures. Annotating this training data becomes burdensome and creates a major bottleneck in maintaining machine-learning databases. Moreover, these trained models fail to generalize to new categories or new varieties of the same categories. This is because new categories or new varieties have data distribution different from the training data distribution. To tackle these problems, this thesis proposes to develop a family of transfer-learning techniques that can deal with different training (source) and testing (target) distributions with the assumption that the availability of annotated data is limited in the testing domain. This is done by using the auxiliary data-abundant source domain from which useful knowledge is transferred that can be applied to data-scarce target domain. This transferable knowledge serves as a prior that biases target-domain predictions and prevents the target-domain model from overfitting. Specifically, we explore structural priors that encode relational knowledge between different data entities, which provides more informative bias than traditional priors. The choice of the structural prior depends on the information availability and the similarity between the two domains. Depending on the domain similarity and the information availability, we divide the transfer learning problem into four major categories and propose different structural priors to solve each of these sub-problems. This thesis first focuses on the unsupervised-domain-adaptation problem, where we propose to minimize domain discrepancy by transforming labeled source-domain data to be close to unlabeled target-domain data. For this problem, the categories remain the same across the two domains and hence we assume that the structural relationship between the source-domain samples is carried over to the target domain. Thus, graph or hyper-graph is constructed as the structural prior from both domains and a graph/hyper-graph matching formulation is used to transform samples in the source domain to be closer to samples in the target domain. An efficient optimization scheme is then proposed to tackle the time and memory inefficiencies associated with the matching problem. The few-shot learning problem is studied next, where we propose to transfer knowledge from source-domain categories containing abundantly labeled data to novel categories in the target domain that contains only few labeled data. The knowledge transfer biases the novel category predictions and prevents the model from overfitting. The knowledge is encoded using a neural-network-based prior that transforms a data sample to its corresponding class prototype. This neural network is trained from the source-domain data and applied to the target-domain data, where it transforms the few-shot samples to the novel-class prototypes for better recognition performance. The few-shot learning problem is then extended to the situation, where we do not have access to the source-domain data but only have access to the source-domain class prototypes. In this limited information setting, parametric neural-network-based priors would overfit to the source-class prototypes and hence we seek a non-parametric-based prior using manifolds. A piecewise linear manifold is used as a structural prior to fit the source-domain-class prototypes. This structure is extended to the target domain, where the novel-class prototypes are found by projecting the few-shot samples onto the manifold. Finally, the zero-shot learning problem is addressed, which is an extreme case of the few-shot learning problem where we do not have any labeled data in the target domain. However, we have high-level information for both the source and target domain categories in the form of semantic descriptors. We learn the relation between the sample space and the semantic space, using a regularized neural network so that classification of the novel categories can be carried out in a common representation space. This same neural network is then used in the target domain to relate the two spaces. In case we want to generate data for the novel categories in the target domain, we can use a constrained generative adversarial network instead of a traditional neural network. Thus, we use structural priors like graphs, neural networks and manifolds to relate various data entities like samples, prototypes and semantics for these different transfer learning sub-problems. We explore additional post-processing steps like pseudo-labeling, domain adaptation and calibration and enforce algorithmic and architectural constraints to further improve recognition performance. Experimental results on standard transfer learning image recognition datasets produced competitive results with respect to previous work. Further experimentation and analyses of these methods provided better understanding of machine learning as well.

We address the problem of segmenting cell contours from microscopy images of human induced pluripotent Retinal Pigment Epithelial stem cells (iRPE) using Convolutional Neural Networks (CNN). Our goal is to compare the accuracy gains of CNN-based segmentation by using (1) un-annotated images via Generative Adversarial Networks (GAN), (2) annotated out-of-bio-domain images via transfer learning, and (3) a priori knowledge about microscope imaging mapped into geometric augmentations of a small collection of annotated images. First, the GAN learns an abstract representation of cell objects. Next, this unsupervised learned representation is transferred to the CNN segmentation models which are further fine-tuned on a small number of manually segmented iRPE cell images. Second, transfer learning is applied by pre-training a part of the CNN segmentation model with the COCO dataset containing semantic segmentation labels. The CNN model is then adapted to the iRPE cell domain using a small set of annotated iRPE cell images. Third, augmentations based on geometrical transformations are applied to a small collection of annotated images. All these approaches to training CNN-based segmentation model are compared to a baseline CNN model trained on a small collection of annotated images. For very small annotation counts, the results show accuracy improvements up to 20 % by the best approach in comparison to the accuracy achieved using a baseline U-Net model. For larger annotation counts these approaches asymptotically approach the same accuracy.

Automatic image annotation (AIA) has been adopted in different applications such as image retrieval and classification. Deep Learning is used in AIA to extract image features and then convert these features into text descriptions and labels. However, conventional AIA models that employ deep learning methods suffer from various shortcomings, such as poor annotation performance. This work proposes an AIA model based on convolutional neural networks (CNNs), generative adversarial networks (GANs), and transfer learning. GANs have attracted a lot of interest because of its ability to generate data without explicitly using probability density. Thus, it has proven its usefulness in image annotation and image augmentation. In this work, an Auxiliary classifier-GAN (ACGAN) has been used, where the discriminator predicts the class of an image rather than taking it as a given input; therefore, the stabilization of the training stage is ensured, and the generation of high-quality images is provided. Transfer learning is also used to enhance the performance of the classification. The proposed model outperforms the best state-of-the-art models in terms of MiAP, F-measure and error rate using ImageClef, ESPGame and IAPR-TC12 datasets.

Extracting traffic information from images plays an increasingly significant role in Internet of vehicle. However, due to the high-speed movement and bumps of the vehicle, the image will be blurred during image acquisition. In addition, in rainy days, as a result of the rain attached to the lens, the target will be blocked by rain, and the image will be distorted. These problems have caused great obstacles for extracting key information from transportation images, which will affect the real-time judgment of vehicle control system on road conditions, and further cause decision-making errors of the system and even have a bearing on traffic accidents. In this paper, we propose a motion-blurred restoration and rain removal algorithm for IoV based on generative adversarial network and transfer learning. Dynamic scene deblurring and image de-raining are both among the challenging classical research directions in low-level vision tasks. For both tasks, firstly, instead of using ReLU in a conventional residual block, we designed a residual block containing three 256-channel convolutional layers, and we used the Leaky-ReLU activation function. Secondly, we used generative adversarial networks for the image deblurring task with our Resblocks, as well as the image de-raining task. Thirdly, experimental results on the synthetic blur dataset GOPRO and the real blur dataset RealBlur confirm the effectiveness of our model for image deblurring. Finally, as an image de-raining task based on transfer learning, we can fine-tune the pre-trained model with less training data and show good results on several datasets used for image rain removal.

The adoption of machine learning in materials science has rapidly transformed materials property prediction. Hurdles limiting full capitalization of recent advancements in machine learning include the limited development of methods to learn the underlying interactions of multiple elements as well as the relationships among multiple properties to facilitate property prediction in new composition spaces. To address these issues, we introduce the Hierarchical Correlation Learning for Multi-property Prediction (H-CLMP) framework that seamlessly integrates: (i) prediction using only a material's composition, (ii) learning and exploitation of correlations among target properties in multi-target regression, and (iii) leveraging training data from tangential domains via generative transfer learning. The model is demonstrated for prediction of spectral optical absorption of complex metal oxides spanning 69 three-cation metal oxide composition spaces. H-CLMP accurately predicts non-linear composition-property relationships in composition spaces for which no training data are available, which broadens the purview of machine learning to the discovery of materials with exceptional properties. This achievement results from the principled integration of latent embedding learning, property correlation learning, generative transfer learning, and attention models. The best performance is obtained using H-CLMP with transfer learning [H-CLMP(T)] wherein a generative adversarial network is trained on computational density of states data and deployed in the target domain to augment prediction of optical absorption from composition. H-CLMP(T) aggregates multiple knowledge sources with a framework that is well suited for multi-target regression across the physical sciences.

Images created using the Super Resolution method can generate more information compared to their low resolution counterparts. A super-resolved image, which is created using an original image captured by an imaging source is not only more meaningful to human perception but also has advantages on downstream tasks such as object detection and pattern recognition. In this work, we aim to apply the Super Resolution method to the Aerial Images captured for surveillance to enable more information about the original scenes. To achieve this Super Resolution Generative Adversarial Network (SRGAN), which is based on the Generative Adversarial Networks architecture is used. We also applied transfer learning methodology to achieve better image quality. Public xView and DOTA datasets which contain images mostly captured by satellites around the world are used to train a generative model via SRGAN architecture. Furthermore, DIV2K dataset is used to pre-train a generative model, and then the transfer learning technique is used to train separate models on xView and DOTA validation datasets. Perceptual Index (PI) and Root Mean Squared Error (RMSE) which are used on European Conference on Computer Vision -Perceptual Image Restoration and Manipulation Workshop 2018 are computed as the performance metrics. We have seen that the model which gives the best PI results, i.e. better perceptual quality, on xView and DOTA validation datasets is the one trained using the DIV2K dataset and the model which gives the best RMSE results, i.e. better reconstruction quality, is the one trained using the transfer learning technique.

Computer vision has been transformed by deep learning, which has made it possible for robots to comprehend and interpret visual data at previously unheard-of levels. This study explores deep learning for computer vision systems: future trends and upcoming technology. Looking ahead, several significant advancements are reshaping this profession. First off, a viable approach is to combine deep learning architectures with generative models like Variational Autoencoders (VAEs) and Generative Adversarial Networks (GANs). This synergy makes activities like picture generation, style transfer, and data augmentation easier and enables a more nuanced interpretation of visual data. Second, a key component of deep learning for computer vision is now recognized to be attention mechanisms. These techniques, which are based on human visual attention, allow models to concentrate on particular areas of a picture, improving efficiency and interpretability. Tasks such as object detection, image segmentation, and image captioning demonstrate the critical role of attention mechanisms. Moreover, computer vision is reaching new heights thanks to the combination of deep learning and other cutting-edge technologies. Real-time processing on devices is made possible by the combination of deep learning and edge computing, which lowers latency and improves privacy. Furthermore, new opportunities for applications in augmented reality, robotics, and immersive experiences are being created by the combination of deep learning and 3D vision technologies. In the deep learning space for computer vision, transfer learning and meta-learning are likewise becoming more and more popular [1]. These methods increase performance on a different but related job by utilizing pre-trained models or knowledge from one task. This method minimizes the need for large datasets for training in situations where labeled data is rare. Deep learning in computer vision raises ethical questions that are becoming more and more significant. Concerns about decision interpretability, algorithmic bias, and privacy are becoming more prevalent. Fairness, accountability, and transparency must be given top priority in this field's future developments in order to guarantee the appropriate use of these technologies. In conclusion, there are a lot of promising things that lie ahead for deep learning in computer vision. Advances in transfer learning, attention mechanisms, and the integration of generative models are all leading to more complex and effective vision systems. As these technologies advance, it is critical to address ethical issues in order to build trust and guarantee that deep learning has a good social impact. This study offers a thorough analysis of these recent developments, providing insight into the future direction of deep learning for computer vision.

Terahertz communications are envisioned as a promising technology for the sixth generation and beyond wireless systems, which can support wireless links with Terabits-per-second (Tbps) data rates. As the foundation of designing terahertz communications, channel modeling and characterization are crucial to scrutinize the potential of this spectrum. However, current channel modeling in the terahertz band heavily relies on time-consuming and costly measurements. Here, we propose a transfer learning enabled transformer based generative adversarial network to mitigate this problem in terahertz channel modeling. Specifically, as a fundamental building block, a generative adversarial network is exploited to generate channel parameters. To improve the accuracy, a transformer structure with a self-attention mechanism is incorporated in generative adversarial network. Still incurring errors compared with ground-truth measurement, a transfer learning is designed to solve the mismatch between the formulated network and measurement. The proposed method can achieve high accuracy in channel modeling, while requiring only rather limited amount of measurement, which is a promising complement of current channel modeling techniques.

Transfer learning can solve some problems that the lack of data and labeled data is unavailable in deep learning. The domain adaptation method in transfer learning can solve the problem of domain-shift, which is the main direction of current research. The generative adversarial network is popular in the generative model now, So the combination of them will obtain more respectable performance. While previous approaches only use it as a data augmentation tool. Even though the overall distribution is aligned, some samples will still be misclassified, and the corresponding classifier is trained on the source samples and cannot perfectly generalize to the target. We propose a novel generative discriminative domain adaptation method based on the reverse gradient by combining the generative adversarial method to reverse gradient. The method uses unlabeled target domain data to make the target and source domains distributions get closer in the feature space. At the same time, the auxiliary class's generation adversarial network is used to generate consistent-class representation, and the corresponding class's distribution is aligned across domains. Besides, we use circle loss as the classification function. Circle loss proposed a unified formula for deep feature learning features. It is useful for our classification task in our method. Our proposed method achieves better performance than others in most experimental settings and has shown to work well across different datasets such as digits and Office.

Low wind output process (LWOP), which often occur under extreme weather conditions, exhibit low frequency and few-sample characteristics, making challenges of direct modeling with historical data. To address this, sample augmentation is required to obtain sufficient high-quality data for model training. This study proposes a LWOP forecasting method based on the integrating of Diffusion Model and Generative Adversarial Network (GAN). Firstly, the historical LWOP is identified based on indicators such as duration, extremal, and average value. Then, a sample generation model of LWOP integrating Diffusion Model and GAN is developed, where the generator of GAN is replaced by Diffusion Model. The discriminator evaluates the authenticity of samples generated by Diffusion Model, while generator aims to deceive the discriminator through adversarial training. Finally, a forecasting model of LWOP based on transfer learning is built, where the generation samples are used as input for model pre-training, and historical samples are used as input for model fine-tuning. Operation data of ten wind farms in a certain province of China is used for case study; BiLSTM, DNN, and TCN are employed to verify the effectiveness and robustness of the proposed method. The results show that, the proposed method can improve the forecasting accuracy of LWOP for 4.24% to 81.08% compared with traditional methods, when root mean square error is used as evaluation index.

Singing voice conversion (SVC) is a task of converting the perception of the source speaker's identity to the target speaker without changing lyrics and rhythm. Recent approaches in traditional voice conversion involve the use of the generative models, such as Variational Autoencoders (VAE), and Generative Adversarial Networks (GANs). However, in the case of SVC, GANs are not explored much. The only system that has been proposed in the literature uses traditional GAN on the parallel data. The parallel data collection for real scenarios (with the same background music) is not feasible. Moreover, in the presence of background music, SVC is one of the most challenging tasks as it involves the source separation of vocals from the inputs, which will have some noise. Therefore, in this paper, we propose transfer learning, and fine-tuning-based Cycle consistent GAN (CycleGAN) model for non-parallel SVC, where music source separation is done using Deep Attractor Network (DANet). We designed seven different possible systems to identify the best possible combination of transfer learning and fine-tuning. Here, we use a more challenging database, MUSDB18, as our primary dataset, and we also use the NUS-48E database to pre-train CycleGAN. We perform extensive analysis via objective and subjective measures and report that with a 4.14 MOS score out of 5 for naturalness, the CycleGAN model pre-trained on NUS-48E corpus performs the best compared to the other systems described in the paper.

The increasing complexity of power systems as well as the rapid expansion of renewable energy resources and deregulation of electricity industry lead to continual demand for operating power systems at higher efficiencies. The modern power systems are pushed to operate close to their secure limits. When subjected to a severe contingency, the stressed power systems may lose its dynamic security, resulting in catastrophic consequences such as cascading failure and even wide-spread blackout. Under this new reality, online and real-time power system stability assessments have great impacts on avoiding blackout events. The traditional stability methods are mainly based on time-domain simulation, which are not fast enough to satisfy the online and real-time application requirements. In recent years, with the development of advanced real-time system monitoring and communication techniques, the data-driven methods have been identified as powerful tools to help power systems counteract large disturbance and avoid instability. To solve above threats, this Ph.D. research aims to develop data-analytics methodologies from power grid measurement data to recognize the system stability, covering both pre-fault dynamic security assessment (DSA) and post-fault real-time short-term voltage stability (STVS) assessment areas under some extreme scenarios and practical issues. Specific objectives of this Ph.D. research are to develop an intelligent system for the future smart grid. Based on several kinds of promising advanced data-analytics technologies, including randomized learning, incremental broad learning (BL), transfer learning (TL), adversarial machine learning (ML), and generative adversarial network (GAN), etc, an intelligent system will be developed to interpret the power grid stability characteristics, and to support stability-constrained operation control actions. Hence, it can make power systems achieve seven goals, including extensibility, efficiency, robustness, security, personalization, reliability, and effectiveness. The outcomes of this Ph.D. research will enable a more secure, intelligent, and knowledge-transparent future of the next-generation smart grid. The main contributions through this Ph.D. research are as follows: Firstly, the temporal-adaptive data-driven methods are proposed based on optimized hybrid ensemble learning mechanism for online STVS assessment. Compared to the traditional data-driven stability methods, the proposed methods can balance the tradeoff between the stability accuracy and speed and select the most suitable Pareto set. Secondly, in the area of DSA, an online updating DSA method is developed by using incremental BL technique. The incremental BL method only needs to calculate the pseudoinverse of new state matrix without the computations of the whole output weights matrix. In this way, it does not need re-training the whole model to continuously maintain and improve the online DSA accuracy. Besides, the proposed method can achieve under three different increment scenarios, including increment of enhancement hidden nodes, features, and new training instances. Thirdly, in the area of missing phasor measurement unit (PMU) data, two different fully robust data-driven stability methods are developed, respectively for pre-fault DSA and post-fault STVS assessment. The former one is based on GAN without observability-constrained feature subsets to ensure its availability under any PMU missing scenario. The latter one is based on super-resolution perception and incremental BL, which can achieve both satisfactory stability accuracy and speed. Fourthly, in order to accurately quantify the robustness of the ML-based model under adversarial perturbations on safety-critical power system stability problem, an adversarial robustness verification method for data-driven stability models is proposed. Besides, a model-free and attack-independent robust index are defined to evaluate the ability of the ML-based stability models against adversarial examples, which can be used to select the candidate model and provide formal robustness guarantee for stability application in practice. Rigorous mathematic proof is provided for the robust index under both linear and non-linear binary scenarios. Moreover, the proposed robust index can be used to mitigate the adversarial examples via adversarial training. Finally, three different TL-based DSA methods are proposed to use one known fault to evaluate other related but different unknown faults. The first two methods aim to reduce the distribution differences between the trained data and unknown data with the complete unlearned faults. The third one considers the different faults with incomplete data scenario. Through applying the adversarial training process, the DSA model trained by one fault can work for a different but related fault by confusing domain discriminator. With the property of feature extractor network, the proposed integrated TL-based method can solve the missing data issue for the unknown faults. All the methodologies have been tested on benchmark power systems to verify their effectiveness, and comparative and comparative studies with existing methods in the literature are conducted where applicable.

Rotating machinery is a crucial component of industrial equipment, and the fault diagnosis of bearings and gears, as vital elements of rotating machinery, is essential since they often fail under harsh working conditions, leading to significant property losses and serious personal safety problems. However, fault data for gears and bearings are often sparse in actual condition, and it is a challenge to ensure the reliability and stability of fault diagnosis results by extracting the features of a single data. To solve the above problems, this paper proposes a fault diagnosis method that combines Transfer Learning and data fusion techniques. Firstly, in this method, two kinds of fault signals are transformed into Gramian Angular Difference Fields and Recurrence Plot. Next, a U-shaped feature fusion dual discriminator generative adversarial network is used to fuse two-dimensional images from multiple sensor data. Its feature fusion module deeply integrates the features of the two images, thereby solving the impact of single data on the reliability and stability of fault diagnosis. Moreover, open-source datasets are used for Transfer Learning training to tackle the small sample problem. Finally, a decision-level information fusion classifier, the Dual-Branch Dempster-Shafer Classifier (DB-DSC), classifies the fused images. This classifier incorporates an improved soft threshold function and D-S evidence theory to achieve adaptive gradient changes and improve the robustness and accuracy of classification results. The experimental results show the effectiveness and stability of the proposed method, and the generated images get high score in several metrics. The average classification accuracy of the classification network reaches 93% and 92.5% on the two datasets, Therefore, the proposed method exhibits strong fault diagnosis capabilities under the small sample conditions of bearings and gears.

In the field of high-end manufacturing, it is valuable to study few-shot health condition estimation. Although transfer learning and other methods have effectively improved the ability of few-shot learning, they still cannot solve the lack of prior knowledge. In this paper, by combining data enhancement, knowledge reasoning, and transfer learning, a generative knowledge-based transfer learning model is proposed to achieve few-shot health condition estimation. First, with the effectiveness of data enhancement on machine learning, a novel batch monotonic generative adversarial network (BM-GAN) is designed for few-shot health condition data generation, which can solve the problem of insufficient data and generate simulated training data. Second, a generative knowledge-based transfer learning model is proposed with the performance advantages of the belief rule base (BRB) method on few-shot learning, which combines expert knowledge and simulated training data to obtain a generalized BRB model and then fine-tunes the generalized model with real data to obtain a dedicated BRB model. Third, through uniform sampling of NASA lithium battery data and simulating few-shot conditions, the generative transfer-belief rule base (GT-BRB) method proposed in this paper is verified to be feasible for few-shot health condition estimation and improves the estimation accuracy of the BRB method by approximately 17.3%.