Materials representation and transfer learning for multi-property prediction

Extracting traffic information from images plays an increasingly significant role in Internet of vehicle. However, due to the high-speed movement and bumps of the vehicle, the image will be blurred during image acquisition. In addition, in rainy days, as a result of the rain attached to the lens, the target will be blocked by rain, and the image will be distorted. These problems have caused great obstacles for extracting key information from transportation images, which will affect the real-time judgment of vehicle control system on road conditions, and further cause decision-making errors of the system and even have a bearing on traffic accidents. In this paper, we propose a motion-blurred restoration and rain removal algorithm for IoV based on generative adversarial network and transfer learning. Dynamic scene deblurring and image de-raining are both among the challenging classical research directions in low-level vision tasks. For both tasks, firstly, instead of using ReLU in a conventional residual block, we designed a residual block containing three 256-channel convolutional layers, and we used the Leaky-ReLU activation function. Secondly, we used generative adversarial networks for the image deblurring task with our Resblocks, as well as the image de-raining task. Thirdly, experimental results on the synthetic blur dataset GOPRO and the real blur dataset RealBlur confirm the effectiveness of our model for image deblurring. Finally, as an image de-raining task based on transfer learning, we can fine-tune the pre-trained model with less training data and show good results on several datasets used for image rain removal.

Recent progress in machine learning has been mainly due to the availability of large amounts of annotated data used for training complex models with deep architectures. Annotating this training data becomes burdensome and creates a major bottleneck in maintaining machine-learning databases. Moreover, these trained models fail to generalize to new categories or new varieties of the same categories. This is because new categories or new varieties have data distribution different from the training data distribution. To tackle these problems, this thesis proposes to develop a family of transfer-learning techniques that can deal with different training (source) and testing (target) distributions with the assumption that the availability of annotated data is limited in the testing domain. This is done by using the auxiliary data-abundant source domain from which useful knowledge is transferred that can be applied to data-scarce target domain. This transferable knowledge serves as a prior that biases target-domain predictions and prevents the target-domain model from overfitting. Specifically, we explore structural priors that encode relational knowledge between different data entities, which provides more informative bias than traditional priors. The choice of the structural prior depends on the information availability and the similarity between the two domains. Depending on the domain similarity and the information availability, we divide the transfer learning problem into four major categories and propose different structural priors to solve each of these sub-problems. This thesis first focuses on the unsupervised-domain-adaptation problem, where we propose to minimize domain discrepancy by transforming labeled source-domain data to be close to unlabeled target-domain data. For this problem, the categories remain the same across the two domains and hence we assume that the structural relationship between the source-domain samples is carried over to the target domain. Thus, graph or hyper-graph is constructed as the structural prior from both domains and a graph/hyper-graph matching formulation is used to transform samples in the source domain to be closer to samples in the target domain. An efficient optimization scheme is then proposed to tackle the time and memory inefficiencies associated with the matching problem. The few-shot learning problem is studied next, where we propose to transfer knowledge from source-domain categories containing abundantly labeled data to novel categories in the target domain that contains only few labeled data. The knowledge transfer biases the novel category predictions and prevents the model from overfitting. The knowledge is encoded using a neural-network-based prior that transforms a data sample to its corresponding class prototype. This neural network is trained from the source-domain data and applied to the target-domain data, where it transforms the few-shot samples to the novel-class prototypes for better recognition performance. The few-shot learning problem is then extended to the situation, where we do not have access to the source-domain data but only have access to the source-domain class prototypes. In this limited information setting, parametric neural-network-based priors would overfit to the source-class prototypes and hence we seek a non-parametric-based prior using manifolds. A piecewise linear manifold is used as a structural prior to fit the source-domain-class prototypes. This structure is extended to the target domain, where the novel-class prototypes are found by projecting the few-shot samples onto the manifold. Finally, the zero-shot learning problem is addressed, which is an extreme case of the few-shot learning problem where we do not have any labeled data in the target domain. However, we have high-level information for both the source and target domain categories in the form of semantic descriptors. We learn the relation between the sample space and the semantic space, using a regularized neural network so that classification of the novel categories can be carried out in a common representation space. This same neural network is then used in the target domain to relate the two spaces. In case we want to generate data for the novel categories in the target domain, we can use a constrained generative adversarial network instead of a traditional neural network. Thus, we use structural priors like graphs, neural networks and manifolds to relate various data entities like samples, prototypes and semantics for these different transfer learning sub-problems. We explore additional post-processing steps like pseudo-labeling, domain adaptation and calibration and enforce algorithmic and architectural constraints to further improve recognition performance. Experimental results on standard transfer learning image recognition datasets produced competitive results with respect to previous work. Further experimentation and analyses of these methods provided better understanding of machine learning as well.

Deep neural networks (DNNs), e.g., convolutional neural network (CNN), are able to learn effective features from wafer maps for dimensional reduction and feature extraction. However, very large image data are needed to train DNNs to obtain high generalization performance. It is still a difficult task due to the lack of sufficient labeled images with various defects. This article proposes a semisupervised deep transfer learning algorithm called joint feature and label adversarial network (JFLAN). JFLAN uses CNNs to extract transferable features of wafer maps and then introduces a multilayer domain adaptation and pseudolabel learning block based on the generative adversarial network (GAN). This effectively reduces the distribution discrepancy and the among-class distance of the transferable features. Finally, JFLAN transfers knowledge from wafer image source data collected offline and then achieves the goal of significantly improved accuracy of wafer defect recognition and realizes online adaptive defect recognition. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —The defect recognition on wafer maps plays a key role to recognize fault sources of the semiconductor manufacturing processes. Transfer learning is able to use the existing labeled data to assist in the classification of unlabeled data and is very effective to solve the problem of small samples and nonstationary generalization errors. In particular, the infusion of adversarial learning in transfer learning will provide a new idea for deep feature learning. This article provides a novel method based on transfer learning to implement wafer map defect recognition (WMDR) to quickly identify defect root causes for yield enhancement. This article provides a novel way for quality control of semiconductor manufacturing processes based on transfer and adversarial learning.

Active Learning is a machine learning and data mining technique that selects the most informative samples for labeling and uses them as training data; it is especially useful when there are large amount of unlabeled data and labeling them is expensive. Recently, batch-mode active learning, where a set of samples are selected concurrently for labeling, based on their collective merit, has attracted a lot of attention. The objective of batch-mode active learning is to select a set of informative samples so that a classifier learned on these samples has good generalization performance on the unlabeled data. Most of the existing batch-mode active learning methodologies try to achieve this by selecting samples based on certain criteria. In this article we propose a novel criterion which achieves good generalization performance of a classifier by specifically selecting a set of query samples that minimize the difference in distribution between the labeled and the unlabeled data, after annotation. We explicitly measure this difference based on all candidate subsets of the unlabeled data and select the best subset. The proposed objective is an NP-hard integer programming optimization problem. We provide two optimization techniques to solve this problem. In the first one, the problem is transformed into a convex quadratic programming problem and in the second method the problem is transformed into a linear programming problem. Our empirical studies using publicly available UCI datasets and two biomedical image databases demonstrate the effectiveness of the proposed approach in comparison with the state-of-the-art batch-mode active learning methods. We also present two extensions of the proposed approach, which incorporate uncertainty of the predicted labels of the unlabeled data and transfer learning in the proposed formulation. In addition, we present a joint optimization framework for performing both transfer and active learning simultaneously unlike the existing approaches of learning in two separate stages, that is, typically, transfer learning followed by active learning. We specifically minimize a common objective of reducing distribution difference between the domain adapted source, the queried and labeled samples and the rest of the unlabeled target domain data. Our empirical studies on two biomedical image databases and on a publicly available 20 Newsgroups dataset show that incorporation of uncertainty information and transfer learning further improves the performance of the proposed active learning based classifier. Our empirical studies also show that the proposed transfer-active method based on the joint optimization framework performs significantly better than a framework which implements transfer and active learning in two separate stages.

The increasing complexity of power systems as well as the rapid expansion of renewable energy resources and deregulation of electricity industry lead to continual demand for operating power systems at higher efficiencies. The modern power systems are pushed to operate close to their secure limits. When subjected to a severe contingency, the stressed power systems may lose its dynamic security, resulting in catastrophic consequences such as cascading failure and even wide-spread blackout. Under this new reality, online and real-time power system stability assessments have great impacts on avoiding blackout events. The traditional stability methods are mainly based on time-domain simulation, which are not fast enough to satisfy the online and real-time application requirements. In recent years, with the development of advanced real-time system monitoring and communication techniques, the data-driven methods have been identified as powerful tools to help power systems counteract large disturbance and avoid instability. To solve above threats, this Ph.D. research aims to develop data-analytics methodologies from power grid measurement data to recognize the system stability, covering both pre-fault dynamic security assessment (DSA) and post-fault real-time short-term voltage stability (STVS) assessment areas under some extreme scenarios and practical issues. Specific objectives of this Ph.D. research are to develop an intelligent system for the future smart grid. Based on several kinds of promising advanced data-analytics technologies, including randomized learning, incremental broad learning (BL), transfer learning (TL), adversarial machine learning (ML), and generative adversarial network (GAN), etc, an intelligent system will be developed to interpret the power grid stability characteristics, and to support stability-constrained operation control actions. Hence, it can make power systems achieve seven goals, including extensibility, efficiency, robustness, security, personalization, reliability, and effectiveness. The outcomes of this Ph.D. research will enable a more secure, intelligent, and knowledge-transparent future of the next-generation smart grid. The main contributions through this Ph.D. research are as follows: Firstly, the temporal-adaptive data-driven methods are proposed based on optimized hybrid ensemble learning mechanism for online STVS assessment. Compared to the traditional data-driven stability methods, the proposed methods can balance the tradeoff between the stability accuracy and speed and select the most suitable Pareto set. Secondly, in the area of DSA, an online updating DSA method is developed by using incremental BL technique. The incremental BL method only needs to calculate the pseudoinverse of new state matrix without the computations of the whole output weights matrix. In this way, it does not need re-training the whole model to continuously maintain and improve the online DSA accuracy. Besides, the proposed method can achieve under three different increment scenarios, including increment of enhancement hidden nodes, features, and new training instances. Thirdly, in the area of missing phasor measurement unit (PMU) data, two different fully robust data-driven stability methods are developed, respectively for pre-fault DSA and post-fault STVS assessment. The former one is based on GAN without observability-constrained feature subsets to ensure its availability under any PMU missing scenario. The latter one is based on super-resolution perception and incremental BL, which can achieve both satisfactory stability accuracy and speed. Fourthly, in order to accurately quantify the robustness of the ML-based model under adversarial perturbations on safety-critical power system stability problem, an adversarial robustness verification method for data-driven stability models is proposed. Besides, a model-free and attack-independent robust index are defined to evaluate the ability of the ML-based stability models against adversarial examples, which can be used to select the candidate model and provide formal robustness guarantee for stability application in practice. Rigorous mathematic proof is provided for the robust index under both linear and non-linear binary scenarios. Moreover, the proposed robust index can be used to mitigate the adversarial examples via adversarial training. Finally, three different TL-based DSA methods are proposed to use one known fault to evaluate other related but different unknown faults. The first two methods aim to reduce the distribution differences between the trained data and unknown data with the complete unlearned faults. The third one considers the different faults with incomplete data scenario. Through applying the adversarial training process, the DSA model trained by one fault can work for a different but related fault by confusing domain discriminator. With the property of feature extractor network, the proposed integrated TL-based method can solve the missing data issue for the unknown faults. All the methodologies have been tested on benchmark power systems to verify their effectiveness, and comparative and comparative studies with existing methods in the literature are conducted where applicable.

We address the problem of segmenting cell contours from microscopy images of human induced pluripotent Retinal Pigment Epithelial stem cells (iRPE) using Convolutional Neural Networks (CNN). Our goal is to compare the accuracy gains of CNN-based segmentation by using (1) un-annotated images via Generative Adversarial Networks (GAN), (2) annotated out-of-bio-domain images via transfer learning, and (3) a priori knowledge about microscope imaging mapped into geometric augmentations of a small collection of annotated images. First, the GAN learns an abstract representation of cell objects. Next, this unsupervised learned representation is transferred to the CNN segmentation models which are further fine-tuned on a small number of manually segmented iRPE cell images. Second, transfer learning is applied by pre-training a part of the CNN segmentation model with the COCO dataset containing semantic segmentation labels. The CNN model is then adapted to the iRPE cell domain using a small set of annotated iRPE cell images. Third, augmentations based on geometrical transformations are applied to a small collection of annotated images. All these approaches to training CNN-based segmentation model are compared to a baseline CNN model trained on a small collection of annotated images. For very small annotation counts, the results show accuracy improvements up to 20 % by the best approach in comparison to the accuracy achieved using a baseline U-Net model. For larger annotation counts these approaches asymptotically approach the same accuracy.

Automatic image annotation (AIA) has been adopted in different applications such as image retrieval and classification. Deep Learning is used in AIA to extract image features and then convert these features into text descriptions and labels. However, conventional AIA models that employ deep learning methods suffer from various shortcomings, such as poor annotation performance. This work proposes an AIA model based on convolutional neural networks (CNNs), generative adversarial networks (GANs), and transfer learning. GANs have attracted a lot of interest because of its ability to generate data without explicitly using probability density. Thus, it has proven its usefulness in image annotation and image augmentation. In this work, an Auxiliary classifier-GAN (ACGAN) has been used, where the discriminator predicts the class of an image rather than taking it as a given input; therefore, the stabilization of the training stage is ensured, and the generation of high-quality images is provided. Transfer learning is also used to enhance the performance of the classification. The proposed model outperforms the best state-of-the-art models in terms of MiAP, F-measure and error rate using ImageClef, ESPGame and IAPR-TC12 datasets.

Dynamic multiobjective optimization problems (DMOPs) aim to optimize multiple (often conflicting) objectives that are changing over time. Recently, there are a number of promising algorithms proposed based on transfer learning methods to solve DMOPs. However, it is very challenging to reduce the negative effect in transfer learning and find more effective transferred solutions. To fill this research gap, this article proposes a clustering-based transfer (CBT) learning method to solve DMOPs. When the environment changes, two novel operations (clustering-based selection (CBS) and CBT) are used to guide knowledge transfer. Specifically, CBS aims to find a population with nondominated solutions and dominated solutions as the training data for the new environment. Then, CBT further collects the previous Pareto-optimal solutions and some noise solutions as the training data for the previous environment. Two training data sets from different environments are, respectively, divided into multiple clusters and transfer learning is conducted on two similar clusters with high probability to reduce the negative effect, which can train an accurate prediction model to identify the promising solutions for the new environment. Empirical studies have been conducted on 14 benchmark DMOPs and one real-life path planning problem of unmanned air/ground vehicles, which validate the effectiveness of our proposed method. Especially, our method can significantly reduce negative transfer on 12 out of 14 cases when compared with direct transfer learning.

Low wind output process (LWOP), which often occur under extreme weather conditions, exhibit low frequency and few-sample characteristics, making challenges of direct modeling with historical data. To address this, sample augmentation is required to obtain sufficient high-quality data for model training. This study proposes a LWOP forecasting method based on the integrating of Diffusion Model and Generative Adversarial Network (GAN). Firstly, the historical LWOP is identified based on indicators such as duration, extremal, and average value. Then, a sample generation model of LWOP integrating Diffusion Model and GAN is developed, where the generator of GAN is replaced by Diffusion Model. The discriminator evaluates the authenticity of samples generated by Diffusion Model, while generator aims to deceive the discriminator through adversarial training. Finally, a forecasting model of LWOP based on transfer learning is built, where the generation samples are used as input for model pre-training, and historical samples are used as input for model fine-tuning. Operation data of ten wind farms in a certain province of China is used for case study; BiLSTM, DNN, and TCN are employed to verify the effectiveness and robustness of the proposed method. The results show that, the proposed method can improve the forecasting accuracy of LWOP for 4.24% to 81.08% compared with traditional methods, when root mean square error is used as evaluation index.

We present a mathematically well-founded approach for the synthetic modeling of turbulent flows using generative adversarial networks (GAN). Based on the analysis of chaotic, deterministic systems in terms of ergodicity, we outline a mathematical proof that GAN can actually learn to sample state snapshots from the invariant measure of the chaotic system. Based on this analysis, we study a hierarchy of chaotic systems starting with the Lorenz attractor and then carry on to the modeling of turbulent flows with GAN. As training data, we use fields of velocity fluctuations obtained from large-eddy simulations (LES). Two architectures are investigated in detail: we use a deep, convolutional GAN (DCGAN) to synthesize the turbulent flow around a cylinder. We furthermore simulate the flow around a low-pressure turbine stator using the pix2pixHD architecture for a conditional DCGAN being conditioned on the position of a rotating wake in front of the stator. The settings of adversarial training and the effects of using specific GAN architectures are explained. We thereby show that GAN are efficient in simulating turbulence in technically challenging flow problems on the basis of a moderate amount of training data. GAN training and inference times significantly fall short when compared with classical numerical methods, in particular, LES, while still providing turbulent flows in high resolution. We furthermore analyze the statistical properties of the synthesized and LES flow fields, which agree excellently. We also show the ability of the conditional GAN to generalize over changes of geometry by generating turbulent flow fields for positions of the wake that are not included in the training data.

Generative Adversarial Networks (GANs) have been used as a deep learning approach to solving physics and engineering problems. Using deep learning for these problems is attractive in that reasonably accurate models can be inferred from only raw data, eliminating the need to define the exact physical equations governing a problem. We expand on previous work [Farimani et al. 2017] using GANs to generate steady-state solutions to the two-dimensional heat equation. Using a basic conditional GAN (cGAN), we generate accurate solutions for rectangular domains conditioned on four edge boundary conditions (MAE%3C0.5%). For finding steady-state solutions over arbitrary two-dimensional domains (not constrained to rectangles), we use a cGAN designed for image-to-image translation. We train this GAN on various types of geometric domains (circles, squares, triangles, shapes with one circular or rectangular hole), achieving accurate results on test data made up of geometries similar to those in training (MAE%3C1%). For both of these GANs, we experiment with different loss function terms, showing that a term using the gradients of solution images significantly improves the basic cGAN but not the image-to-image GAN. Lastly, we show that the image-to-image GAN performs poorly when applied to two-dimensional geometries that vary in structure from training data (MAE%3C8% for shapes with multiple holes or different shaped holes). This demonstrates the cGAN's lack of generalizability. While the cGAN is an accurate and computationally efficient method when trained and tested on similarly structured data, it is a much less reliable method when applied to data that is slightly different in structure from the training data.

The Intelligent Fault Diagnosis of rotating machinery calls for a substantial amount of training data, posing challenges in acquiring such data for damaged industrial machinery. This paper presents a novel approach for generating synthetic data using a Generative Adversarial Network (GAN) with cycle consistency loss function known as cycleGAN. The proposed method aims to generate synthetic data that could effectively replace real experimental data. The generative model is trained to transform wavelet images of simulated vibrational signals into authentic data obtained from machinery with damaged bearings. The utilization of Maximum Mean Discrepancy (MMD) and Fréchet Inception Distance (FID) demonstrates a noteworthy resemblance between synthetic and real experimental data. Also, the generative model enables the synthesis of data that may have been entirely lacking from the experimental observation, indicating generative zero-shot learning capabilities. The efficacy of synthetic data in training diagnosis algorithms by means of Transfer Learning (TL) on Convolutional Neural Networks (CNNs) has been demonstrated to be comparable to that of real data. The study has been validated by means of the test rig for medium-sized industrial bearings accessible at the Politecnico di Torino.

Terahertz communications are envisioned as a promising technology for the sixth generation and beyond wireless systems, which can support wireless links with Terabits-per-second (Tbps) data rates. As the foundation of designing terahertz communications, channel modeling and characterization are crucial to scrutinize the potential of this spectrum. However, current channel modeling in the terahertz band heavily relies on time-consuming and costly measurements. Here, we propose a transfer learning enabled transformer based generative adversarial network to mitigate this problem in terahertz channel modeling. Specifically, as a fundamental building block, a generative adversarial network is exploited to generate channel parameters. To improve the accuracy, a transformer structure with a self-attention mechanism is incorporated in generative adversarial network. Still incurring errors compared with ground-truth measurement, a transfer learning is designed to solve the mismatch between the formulated network and measurement. The proposed method can achieve high accuracy in channel modeling, while requiring only rather limited amount of measurement, which is a promising complement of current channel modeling techniques.

The development of structural materials with outstanding mechanical response has long been sought for innumerable industrial, technological, and even biomedical applications. However, these compounds tend to derive their fascinating properties from a myriad of interactions spanning multiple scales, from localized chemical bonding to macroscopic interactions between grains. This diversity has limited the ability of researchers to develop new materials on a reasonable timeline. Fortunately, the advent of machine learning in materials science has provided a new approach to analyze high-dimensional space and identify correlations among the structure-composition-property-processing relationships that may have been previously missed. In this review, we examine some successful examples of using data science to improve known structural materials by analyzing fatigue and failure, and we discuss approaches to develop entirely new classes of structural materials in complex composition spaces including high-entropy alloys and bulk metallic glasses. Highlighting the recent advancement in this field demonstrates the power of data-driven methodologies that will hopefully lead to the production of market-ready structural materials.

Currently, numerous machine learning (ML) techniques are being applied in the field of renewable energy (RE). These techniques may not perform well if they do not have enough training data. Additionally, the main assumption in most of the ML algorithms is that the training and testing data are from the same feature space and have similar distributions. However, in many practical applications, this assumption is false. Recently, transfer learning (TL) has been introduced as a promising machine-learning framework to mitigate these issues by preparing extra-domain data so that knowledge may be transferred across domains. This learning technique improves performance and avoids the resource expensive collection and labeling of domain-centric datasets; furthermore, it saves computing resources that are needed for re-training new ML models from scratch. Lately, TL has drawn the attention of researchers in the field of RE in terms of forecasting and fault diagnosis tasks. Owing to the rapid progress of this technique, a comprehensive survey of the related advances in RE is needed to show the critical issues that have been solved and the challenges that remain unsolved. To the best of our knowledge, few or no comprehensive surveys have reviewed the applications of TL in the RE field, especially those pertaining to forecasting solar and wind power, load forecasting, and predicting failures in power systems. This survey fills this gap in RE classification and forecasting problems, and helps researchers and practitioners better understand the state of the art technology in the field while identifying areas for more focused study. In addition, this survey identifies the main issues and challenges of using TL for REs, and concludes with a discussion of future perspectives.