A new class of fault recognition method for wind turbine systems based on deep learning

The problem of fault classification in industry has been studied extensively. Most classification algorithms are modeled on the premise of data balance. However, the difficulty of collecting industrial data in different modes is quite different. This inevitably leads to data imbalance, which will adversely affect the fault classification performance. This article proposes a novel data augmentation classifier (DAC) for imbalanced fault classification. Data augmentation based on generative adversarial networks (GANs) is an effective way to solve the problem of unbalanced classification. However, the randomness of the GAN generation process restricts the effect of data enhancement. DAC proposes a data selection strategy based on data filtering and data purification in model training to solve this problem. In addition, DAC combines supervised learning and data generation processes to obtain an end-to-end model. Meanwhile, multigenerator structure of DAC (MDAC) is proposed to solve the problem of incomplete learning of a single generator when data imbalances get complicated. The proposed DAC and MDAC are applied in two fault classification cases of the Tennessee Eastman (TE) benchmark process, results of which show superiority of DAC and MDAC compared to existing methods. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —Data imbalances are common in fault classification and affect the effectiveness of modeling in industry. As a generative model, generative adversarial networks (GANs) provide new ideas for small-class data augmentation. However, the instability of its training process and the randomness of data generation affect the results of data augmentation. In this article, the GAN generation process is analyzed in detail. The results of the visualization indicate that no data generation was perfect at any one time. Based on the rules of GAN data generation, we propose a data selection strategy during training. High-quality data are selected for data augmentation through data filtering and data purification. Apart from this, we combine the training process of GAN and classification model for imbalanced data to reduce modeling time. Through industrial examples, we have evaluated the effectiveness of this method.

Detection and classification are two significant tasks for industrial fault diagnosis. However, conventional methods typically treat these tasks as separate and independent problems, and necessitate a retraining process when new fault samples or classes are collected. Therefore, an incremental support vector data description scheme using Gaussian kernel function is pro-posed for industrial process fault diagnosis in a unified frame-work. In this framework, the decision boundary is updated incrementally only based on the specific original support vectors and newly collected samples. An adaptive threshold and a restructured radius are proposed to promote accuracy in the fault detection. In the classification procedure, the hyperspheres for all known classes are constructed by decision tree. The new sample that does not belong to any known class is identified as an unknown class. Without a time-consuming retraining process, the proposed diagnosis method with the incremental learning capability can synchronously achieve the fault detection and classification task. Experimental results demonstrate the effectiveness and superiority in terms of diagnosis performance. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —In practical industrial processes, new fault samples are collected and new fault classes emerge continually. Under such scenarios, fault diagnosis with incremental learning capability is becoming increasingly important. This work proposes a unified incremental framework for fault diagnosis based on support vector data description, which is able to achieve fault detection and classification synchronously. For fault detection, an adaptive threshold and a restructured radius are developed. For fault classification, hypersphere-shaped boundaries of all fault classes are given via the decision tree-based strategy. The proposed method is updated to include new fault samples and fault classes without dimensionality reduction or any distributional assumptions.

Surface defect recognition (SDC) is essential in intelligent manufacturing. Deep learning (DL) is a research hotspot in SDC. Limited defective samples are available in most real-world cases, which poses challenges for DL methods. Given such circumstances, generating defective samples by generative adversarial networks (GANs) is applied. However, insufficient samples and high-frequency texture details in defects make GANs very hard to train, yield mode collapse, and poor image quality, which can further impact SDC. To solve these problems, this article proposes a new GAN called contrastive GAN, which can be trained to generate diverse defects with only extremely limited samples. Specifically, a shared data augmentation (SDA) module is proposed for avoiding overfitting. Then, a feature attention matching (FAM) module is proposed to align features for improving the quality of generated images. Finally, a contrastive loss based on hypersphere is employed to constrain GANs to generate images that differ from the traditional transform. Experiments show that the proposed GAN generates defective images with higher quality and lower variance between real defects compared to other GANs. Synthetic images contribute to pretrained DL networks with accuracies of up to 95.00%–99.56% for Northeastern University (NEU) datasets of different sizes and 91.84% for printed circuit board (PCB) cases, which proves the effectiveness of the proposed method.

Deep neural networks (DNNs), e.g., convolutional neural network (CNN), are able to learn effective features from wafer maps for dimensional reduction and feature extraction. However, very large image data are needed to train DNNs to obtain high generalization performance. It is still a difficult task due to the lack of sufficient labeled images with various defects. This article proposes a semisupervised deep transfer learning algorithm called joint feature and label adversarial network (JFLAN). JFLAN uses CNNs to extract transferable features of wafer maps and then introduces a multilayer domain adaptation and pseudolabel learning block based on the generative adversarial network (GAN). This effectively reduces the distribution discrepancy and the among-class distance of the transferable features. Finally, JFLAN transfers knowledge from wafer image source data collected offline and then achieves the goal of significantly improved accuracy of wafer defect recognition and realizes online adaptive defect recognition. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —The defect recognition on wafer maps plays a key role to recognize fault sources of the semiconductor manufacturing processes. Transfer learning is able to use the existing labeled data to assist in the classification of unlabeled data and is very effective to solve the problem of small samples and nonstationary generalization errors. In particular, the infusion of adversarial learning in transfer learning will provide a new idea for deep feature learning. This article provides a novel method based on transfer learning to implement wafer map defect recognition (WMDR) to quickly identify defect root causes for yield enhancement. This article provides a novel way for quality control of semiconductor manufacturing processes based on transfer and adversarial learning.

Deep learning methods have made remarkable strides in surface defect detection. But, they heavily rely on large amount of training data, which can be a costly endeavor, especially for specific applications like steel strip surface defect detection, where acquiring and labeling large-scale data is impractical due to the rarity of certain defective categories in production environment. Hence, realistic defect image synthesis can greatly alleviate this issue. However, training image generation networks also demand substantial data, making image data augmentation merely an auxiliary effort. In this work, we propose a Generative Adversarial Network (GAN)-based image synthesis framework. We selectively extract the defect edges of the original image as well as the background texture information, and use them as network input through the spatially-adaptive (de)normalization (SPADE) module. This enriches the input information, thus significantly reducing the amount of training data for GAN network in image generation, and enhancing the background details as well as the defect boundaries in the generated images. Additionally, we introduce a novel generator loss term that balances the similarity and perceptual fidelity between synthetic and real images by constraining high-level features at different feature levels. This provides more valuable information for data augmentation in training object detection models using synthetic images. Our experimental results demonstrate the sophistication of the proposed image synthesis method and its effectiveness in data augmentation for steel strip surface defect detection tasks.

Deep Learning algorithms have almost become a key standard for majority of vision and machine learning problems. Despite its common usage and high performance for many applications, they have certain disadvantages. One major problem with deep learning methods is the size of the dataset to be used for training. The methods require a large dataset for an adequate training. However, a large dataset may not be available for all problems. In such a case, researchers apply data augmentation methods to obtain a larger dataset from a given dataset. For the image classification problem, conventional method for data augmentation is the application of transformation based methods; such as flipping, rotation, blurring etc. Recently, generative models which apply deep learning methods are also commonly used for data augmentation. On the other hand, in case of a too large dataset the classifiers may overfit and end up with a lack of generalization. In this study, we explore the usage of generative adversarial networks for data augmentation in the image classification problem. We evaluate the classification performance with three types of augmentation methods. Dataset is first augmented by two conventional methods; Gaussian blurring and dropout of regions, then by generative adversarial networks. Meanwhile, we observe the behavior of the classifier for various sized datasets with and without data augmentation. We observe that especially in datasets of certain sizes generative adversarial networks can be effectively used for data augmentation.

Recently, in the building and infrastructure fields, studies on defect detection methods using deep learning have been widely implemented. For robust automatic recognition of defects in buildings, a sufficiently large training dataset is required for the target defects. However, it is challenging to collect sufficient data from degrading building structures. To address the data shortage and imbalance problem, in this study, a data augmentation method was developed using a generative adversarial network (GAN). To confirm the effect of data augmentation in the defect dataset of old structures, two scenarios were compared and experiments were conducted. As a result, in the models that applied the GAN-based data augmentation experimentally, the average performance increased by approximately 0.16 compared to the model trained using a small dataset. Based on the results of the experiments, the GAN-based data augmentation strategy is expected to be a reliable alternative to complement defect datasets with an unbalanced number of objects.

The actual production process is often faced with insufficient fault data. Traditional data augmentation algorithms are prone to problems such as overfitting and pattern collapse. To address the issues, a tractor transmission systems vibration fault data augmentation and diagnosis method based on deep convolutional generative adversarial network is proposed. First, the original vibration signal is converted to a time–frequency diagram, which serves as the input to the generative adversarial network. Subsequently, the two timescale update rule strategy and gradient penalty are applied to stabilize the training process of the generative adversarial network, and the self-attention mechanism is introduced to enhance the feature extraction capability of the generative adversarial network. Furthermore, an enhanced AlexNet based on the convolutional block attention module is developed to improve the diagnostic performance. The approach was verified using datasets from Case Western Reserve University (CWRU), Huazhong University of Science and Technology (HUST), and a laboratory gearbox dataset. Results show 99% diagnostic accuracy for CWRU, over 97.5% for the laboratory dataset, and 98% accuracy under a single condition for the HUST gear dataset, with accuracy exceeding 95.3% under other conditions. This indicates that the method improves diagnostic precision and model generalization under data scarcity.

Evolutionary conditional GANs for supervised data augmentation: The case of assessing berry number per cluster in grapevine

Deep Learning has recently become hugely popular in machine learning for its ability to solve end-to-end learning systems, in which the features and the classifiers are learned simultaneously, providing significant improvements in classification accuracy in the presence of highly-structured and large databases. Its success is due to a combination of recent algorithmic breakthroughs, increasingly powerful computers, and access to significant amounts of data. Researchers have also considered privacy implications of deep learning. Models are typically trained in a centralized manner with all the data being processed by the same training algorithm. If the data is a collection of users' private data, including habits, personal pictures, geographical positions, interests, and more, the centralized server will have access to sensitive information that could potentially be mishandled. To tackle this problem, collaborative deep learning models have recently been proposed where parties locally train their deep learning structures and only share a subset of the parameters in the attempt to keep their respective training sets private. Parameters can also be obfuscated via differential privacy (DP) to make information extraction even more challenging, as proposed by Shokri and Shmatikov at CCS'15. Unfortunately, we show that any privacy-preserving collaborative deep learning is susceptible to a powerful attack that we devise in this paper. In particular, we show that a distributed, federated, or decentralized deep learning approach is fundamentally broken and does not protect the training sets of honest participants. The attack we developed exploits the real-time nature of the learning process that allows the adversary to train a Generative Adversarial Network (GAN) that generates prototypical samples of the targeted training set that was meant to be private (the samples generated by the GAN are intended to come from the same distribution as the training data). Interestingly, we show that record-level differential privacy applied to the shared parameters of the model, as suggested in previous work, is ineffective (i.e., record-level DP is not designed to address our attack).

Crop yield estimation and prediction constitutes a key issue in agricultural management, particularly under the context of demographic pressure and climate change. Currently, the main challenge in estimating crop yields based on remotely sensed data and data-driven methods is how to cope with small datasets and the limited amount of annotated samples. In order to cope with small datasets and the limited amount of annotated samples and improve the accuracy of winter wheat yield estimation in the Guanzhong Plain, PR China, this study proposed a method of combining generative adversarial networks (GANs) and convolutional neural network (CNN) for comprehensive growth monitoring of winter wheat, in which the remotely sensed leaf area index (LAI), vegetation temperature condition index (VTCI) and meteorological data at four growth stages of winter wheat during 2012–2017 were generated as the inputs of multi-layer convolutional neural networks (CNNs), and GAN was employed to artificially increase the number of training samples. Then, a linear regression model between the simulated comprehensive growth monitoring (I) and the measured yields was established to estimate yields of winter wheat in the Guanzhong Plain pixel by pixel. The final results showed when GAN was used to double the size of the training samples, and the simulation values obtained by CNN based on augmented samples using GAN provided a better training (R2 = 0.95, RMSE = 0.05), validation (R2 = 0.54, RMSE = 0.16) and testing (R2 = 0.50, RMSE = 0.14) performance than that just using the original samples. The achieved best pixel-scale yield estimation accuracy of winter wheat (R2 = 0.50, RMSE = 591.46 kg/ha) in the Guanzhong Plain. These results showed that small samples can be enlarged by GAN, thus, more important features for reflecting the growth conditions and yields of winter wheat from the remotely sensed indices and meteorological indices can be extracted, and indicated that CNN accompanied with GAN could contribute a lot to the comprehensive growth monitoring and yield estimation of winter wheat and data augmentation methods are extremely useful for the application of small samples in deep learning.

A large number of effective annotated data is the key support for learning a fault diagnosis model of mechanical equipment. However, the existing practical samples used for training fault classifiers are usually small and interfered by noise .According to this problem the paper presents a rub impact fault recognition method based on the Conditional Generative Adversarial Nets (CGAN) and Acoustic Emission (AE) technology. The data are from the Wind Turbine Train test bed. AE features are extracted from various views such as time, frequency and energy intensity under different operation state. The proposed CGAN model adds useful auxiliary information into each layer of GAN generation model to improve the quality of generated pseudo-samples. It is further to evaluate the probability of samples from the training set or real set. The experimental results show that this model can effectively identify the rub impact fault and have strong robustness. It is an effective way to solve the problem of inadequate sample and improve the recognition performance of rub impact fault.

Fault recognition method for speed-up and speed-down process of rotating machinery based on independent component analysis and Factorial Hidden Markov Model

In recent years, deep learning has attracted much attention and various techniques have been proposed. GAN (Generative adversarial networks) is one such method. GAN uses images as the training set and learns to generate new images that are indistinguishable from the training set. In this study, A GAN-based design method that generates new products from the images of the customer's favorite products is proposed. The product images that customers evaluated as preferable are used as the training set of GAN. If the GAN fulfills its capabilities properly, the images generated from a customer's favorite product are more likely to be preferred by the customer. In the case study, the proposed method was applied to chair design. The generated chair images were first evaluated in terms of image quality, and then evaluated by subjects.

Deep learning has revolutionized the performance of classification and object detection, but meanwhile demands sufficient labeled data for training. Given insufficient data, while many techniques have been developed to help combat overfitting, the challenge remains if one tries to train deep networks, especially in the ill-posed <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">extremely low data regimes</i> : only a small set of labeled data are available, and nothing – including unlabeled data – else. Such regimes arise from practical situations where not only data labeling but also data collection itself is expensive. We propose a deep adversarial data augmentation (DADA) technique to address the problem, in which we elaborately formulate data augmentation as a problem of training a class-conditional and supervised generative adversarial network (GAN). Specifically, a new discriminator loss is proposed to fit the goal of data augmentation, through which both real and augmented samples are enforced to contribute to and be consistent in finding the decision boundaries. Tailored training techniques are developed accordingly. To quantitatively validate its effectiveness, we first perform extensive simulations to show that DADA substantially outperforms both traditional data augmentation and a few GAN-based options. We then extend experiments to three real-world small labeled classification datasets where existing data augmentation and/or transfer learning strategies are either less effective or infeasible. We also demonstrate that DADA to can be extended to the detection task. We improve the pedestrian synthesis work by substitute for our discriminator and training scheme. Validation experiment shows that DADA can improve the detection mean average precision (mAP) compared with some traditional data augmentation techniques in object detection. Source code is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/SchafferZhang/DADA</uri> .