Lightweight Knowledge Distillation-Based Transfer Learning Framework for Rolling Bearing Fault Diagnosis.

The concept of Industry 4.0 uses cyber-physical systems and the Internet of Things to create "smart factories" that enable automated and connected production. However, the complex and diverse nature of cyber-physical systems makes them vulnerable to physical threats, where any fault in their physical components can lead to serious consequences, from financial losses to human safety. To reduce the risk of such failures, it is essential to have techniques in place to monitor the condition of these physical components closely. In recent years, data-driven predictive maintenance methods have become common due to the growth in data and advancements in machine learning. This thesis focuses on fault diagnosis, a key part of predictive maintenance, which aims to detect and identify faults in physical components. Incorporating advanced techniques, including machine learning algorithms and sensor data analysis, enhanced fault diagnosis systems' accuracy and efficiency. However, several real-world barriers challenge the effectiveness of these techniques. This research is structured around three primary objectives. The first objective aims to establish an efficient fault diagnosis system that is highly sensitive to fault occurrences while being resilient to uncertainties such as variability in fault patterns and data imbalance. To address variability in fault patterns, we developed a multi-step approach integrating clustering techniques and an adaptive neuro-fuzzy inference system, which significantly improved diagnosis robustness even under noisy conditions. For imbalanced datasets, a two-step method was introduced, combining permutation entropy for fault detection and envelope analysis for fault isolation. This approach was enhanced by an automatic frequency band selection method leveraging wavelet transform and statistical criteria, resulting in high fault diagnosis accuracy and reliability, even with limited labeled faulty data. The second objective focuses on developing fault diagnosis systems that can maintain high performance despite domain shifts and limited training data in both source and target domains. To address the challenge of domain shifts, a transfer learning technique was introduced, and a systematic literature review was conducted. This study bridges a technical gap by providing a comprehensive taxonomy, state-of-the-art methods, and insights into the challenges of applying transfer learning to predictive maintenance. Additionally, a conceptual framework was proposed to explore scenarios for transferring diagnostic knowledge from different source domains to target domains with insufficient training datasets. In a subsequent study, a new diagnostic framework was developed using digital twins and partial transfer learning to address the lack of real training datasets in both source and target domains as well as label space inconsistencies between them. The framework includes a general approach for developing digital twins of real machines to generate synthetic diagnostic data as source domain data. Furthermore, by incorporating a deep learning network and a double-layer attention mechanism, a partial transfer learning diagnostic model was developed to transfer diagnostic knowledge from the digital twin to the real target domain despite label space inconsistencies. The proposed framework was validated through an industrial case study involving rolling bearings in rotating equipment. The third objective focuses on developing an efficient fault diagnosis system capable of maintaining reliability under unseen and evolving working conditions. To achieve this, a novel self-adaptive fault diagnosis system was proposed, incorporating the MAPE-K loop, a domain generalization network model, and digital twin technologies. This self-adaptive fault diagnosis system leverages the MAPE-K loop to adapt in real time to evolving working conditions. Digital twins were employed to continuously generate data aligned with the real system’s operational context, enabling accurate fault diagnosis in dynamic environments. The domain generalization network model enhances generalization to unseen target working conditions by addressing overfitting and feature loss through multi-domain data augmentation, adversarial learning, and domain discrepancy modules. Validation using multiple rotating machinery datasets demonstrated that the developed fault diagnosis system outperformed state-of-the-art methods across various scenarios. In summary, this dissertation proposes a comprehensive and practical approach to addressing variability in fault patterns, imbalance datasets, data scarcity, domain shifts, and unseen and evolving working conditions challenges in existing fault diagnosis systems.

Named Entity Recognition (NER) is a fundamental task in information extraction from unstructured text. Most previous machine-learning-based NER systems are domain-specific, which implies that they may only perform well on some specific domains (e.g., Newswire ) but tend to adapt poorly to other related but different domains (e.g., Weblog ). Recently, transfer learning techniques have been proposed to NER. However, most transfer learning approaches to NER are developed for binary classification, while NER is a multiclass classification problem in nature. Therefore, one has to first reduce the NER task to multiple binary classification tasks and solve them independently. In this article, we propose a new transfer learning method, named Transfer Joint Embedding (TJE), for cross-domain multiclass classification, which can fully exploit the relationships between classes (labels), and reduce domain difference in data distributions for transfer learning. More specifically, we aim to embed both labels (outputs) and high-dimensional features (inputs) from different domains (e.g., a source domain and a target domain) into a unified low-dimensional latent space, where 1) each label is represented by a prototype and the intrinsic relationships between labels can be measured by Euclidean distance; 2) the distance in data distributions between the source and target domains can be reduced; 3) the source domain labeled data are closer to their corresponding label-prototypes than others. After the latent space is learned, classification on the target domain data can be done with the simple nearest neighbor rule in the latent space. Furthermore, in order to scale up TJE, we propose an efficient algorithm based on stochastic gradient descent (SGD). Finally, we apply the proposed TJE method for NER across different domains on the ACE 2005 dataset, which is a benchmark in Natural Language Processing (NLP). Experimental results demonstrate the effectiveness of TJE and show that TJE can outperform state-of-the-art transfer learning approaches to NER.

Recent progress in machine learning has been mainly due to the availability of large amounts of annotated data used for training complex models with deep architectures. Annotating this training data becomes burdensome and creates a major bottleneck in maintaining machine-learning databases. Moreover, these trained models fail to generalize to new categories or new varieties of the same categories. This is because new categories or new varieties have data distribution different from the training data distribution. To tackle these problems, this thesis proposes to develop a family of transfer-learning techniques that can deal with different training (source) and testing (target) distributions with the assumption that the availability of annotated data is limited in the testing domain. This is done by using the auxiliary data-abundant source domain from which useful knowledge is transferred that can be applied to data-scarce target domain. This transferable knowledge serves as a prior that biases target-domain predictions and prevents the target-domain model from overfitting. Specifically, we explore structural priors that encode relational knowledge between different data entities, which provides more informative bias than traditional priors. The choice of the structural prior depends on the information availability and the similarity between the two domains. Depending on the domain similarity and the information availability, we divide the transfer learning problem into four major categories and propose different structural priors to solve each of these sub-problems. This thesis first focuses on the unsupervised-domain-adaptation problem, where we propose to minimize domain discrepancy by transforming labeled source-domain data to be close to unlabeled target-domain data. For this problem, the categories remain the same across the two domains and hence we assume that the structural relationship between the source-domain samples is carried over to the target domain. Thus, graph or hyper-graph is constructed as the structural prior from both domains and a graph/hyper-graph matching formulation is used to transform samples in the source domain to be closer to samples in the target domain. An efficient optimization scheme is then proposed to tackle the time and memory inefficiencies associated with the matching problem. The few-shot learning problem is studied next, where we propose to transfer knowledge from source-domain categories containing abundantly labeled data to novel categories in the target domain that contains only few labeled data. The knowledge transfer biases the novel category predictions and prevents the model from overfitting. The knowledge is encoded using a neural-network-based prior that transforms a data sample to its corresponding class prototype. This neural network is trained from the source-domain data and applied to the target-domain data, where it transforms the few-shot samples to the novel-class prototypes for better recognition performance. The few-shot learning problem is then extended to the situation, where we do not have access to the source-domain data but only have access to the source-domain class prototypes. In this limited information setting, parametric neural-network-based priors would overfit to the source-class prototypes and hence we seek a non-parametric-based prior using manifolds. A piecewise linear manifold is used as a structural prior to fit the source-domain-class prototypes. This structure is extended to the target domain, where the novel-class prototypes are found by projecting the few-shot samples onto the manifold. Finally, the zero-shot learning problem is addressed, which is an extreme case of the few-shot learning problem where we do not have any labeled data in the target domain. However, we have high-level information for both the source and target domain categories in the form of semantic descriptors. We learn the relation between the sample space and the semantic space, using a regularized neural network so that classification of the novel categories can be carried out in a common representation space. This same neural network is then used in the target domain to relate the two spaces. In case we want to generate data for the novel categories in the target domain, we can use a constrained generative adversarial network instead of a traditional neural network. Thus, we use structural priors like graphs, neural networks and manifolds to relate various data entities like samples, prototypes and semantics for these different transfer learning sub-problems. We explore additional post-processing steps like pseudo-labeling, domain adaptation and calibration and enforce algorithmic and architectural constraints to further improve recognition performance. Experimental results on standard transfer learning image recognition datasets produced competitive results with respect to previous work. Further experimentation and analyses of these methods provided better understanding of machine learning as well.

Bearing is one of the core components of any rotating machine, and its failure is widespread. This reason drives continuous monitoring and detecting bearing faults during machine operation to warn operators and prevent unforeseen damage. This paper proposes an intelligent Passive Thermography (PTG) based fault diagnosis technique for detection of bearing faults using Convolutional Neural Network (CNN) with Transfer Learning (TL) under varying working conditions. The validation of the proposed method has been done on three different datasets; bearing test rig dataset has been taken as a source domain data while Induction Motor (IM), and Machine Fault Simulator (MFS) datasets have been taken as target domain data. The proposed method enables and speeds up the training process of CNN towards accurate adaptation for fault diagnosis approach in the escalated time frame. The experimental results demonstrated that the suggested approach could successfully learn transferable characteristics from the source domain model, which can cope with the issue of limited availability of training data required for the target domain. The classification accuracy on the target domain dataset were varied in the range of 89 -95.4 % in the case of the IM dataset and 96.5-97.5% in the case of the MFS dataset. Moreover, it shows the benefits of the suggested method, which may be utilized as an effective non-invasive diagnostic tool for rotating machines to avoid unexpected system shutdowns.

Traditional machine learning methods rely on the training data and target data having the same feature space and data distribution. The performance may be unacceptable if there is a difference in data distribution between the training and target data, which is called cross-domain learning problem. In recent years, many domain adaptation methods have been proposed to solve this kind of problems and make much progress. However, existing domain adaptation approaches have a common assumption that the number of the data in source domain (labeled data) and target domain (unlabeled data) is matched. In this paper, the scenarios in real manufacturing site are considered, that the target domain data is much less than source domain data at the beginning, but the number of target domain data will increase as time goes by. A novel method is proposed for fault diagnosis of rolling bearing with online imbalanced cross-domain data. Finally, the proposed method which is tested on bearing dataset (CWRU) has achieved prediction accuracy of 95.89% with only 40 target samples. The results have been compared with other traditional methods. The comparisons show that the proposed online domain adaptation fault diagnosis method has achieved significant improvements. In addition, the deep transfer learning model by adaptive- network-based fuzzy inference system (ANFIS) is introduced to interpretation the results.

To address the issues of negative transfer and reduced stability in transfer learning models for rolling bearing fault diagnosis under variable working conditions, an unsupervised multi-adversarial transfer learning fault diagnosis algorithm based on bearing dynamics simulation data is proposed. Firstly, the algorithm constructs both a global domain classifier and a subdomain classifier. In the subdomain classifier, the simulated vibration signal, which contains rich bearing fault label information, is generated by constructing dynamic equations to replace the label prediction of target domain data, thereby achieving alignment of marginal and conditional distributions. Simultaneously, an improved loss function with embedded maximum mean discrepancy is designed to reduce the feature distribution gap between source and target domain data. Finally, a weight allocation mechanism for source domain and simulation domain samples is developed to promote positive transfer and suppress negative transfer. Experiments were conducted using the Paderborn University dataset and the Huazhong University of Science and Technology dataset, achieving accuracy rates of 89.457% and 96.436%, respectively. The results show that, in comparison with existing unsupervised cross-domain fault diagnosis methods, the proposed method demonstrates significant improvements in diagnostic accuracy and stability, demonstrating its superiority in rolling bearing fault diagnosis under variable operational conditions.

A complex process fault diagnosis method based on manifold distribution adaptation

A rolling bearing fault diagnosis method based on the federated feature transfer learning is proposed for the low accuracy of the diagnosis model in the presence of large differences in data distribution under different working conditions, difficulty in obtaining labeled data and non-sharing of data among different users. This method performs wavelet transformation on the time domain vibration data of rolling bearings to obtain a time-frequency diagram. The priori labeled public data and the multi-user island private data are regarded as the source domain and the target domain. The multi-representation feature extraction structure is introduced to improve the original residual network. Based on an improved residual network and multi-representation features in the source domain and the target domain, every local model and a federated global model are constructed. Through verification of bearing data, the proposed method can establish an effective fault diagnosis model with high fault diagnosis accuracy. It can integrate the knowledge of isolated island data without sharing data among multiple users.

Data-driven fault diagnosis method is widely used in the field of engine health management, which uses engine sensor data as input and engine faulty components as output for component-level fault diagnosis of the engine. The application premise of the general data-driven fault diagnosis method is that all data come from the same working conditions, that is, they belong to the same distribution. However, this assumption is not valid in the actual engine fault diagnosis, because the engine state will change with the increase of running time. In the meantime, collecting engine data is usually expensive, time-consuming, and laborious. To solve these problems, extreme learning machine (ELM)-based two transfer learning methods for fault diagnosis of turboshaft engines are proposed in this paper. One is joint solving ELM (JSELM), which regards the information of the target domain and source domain as similar and different parts, respectively, and knowledge is extracted from them at the same time. The other is model transfer-based ELM (MTELM), which uses the idea of pretraining. First, a general ELM classifier is trained with the source domain data and then fine-tuned with the target domain data. Both methods have a good real-time performance as the traditional ELM. When there are a few data in the target domain, they achieve much better classification accuracy than traditional ELM. Finally, experiments are carried out with turboshaft engine simulation data. The results show that both methods are effective, especially MTELM, which has better classification accuracy.

This study aimed to find suitable source domain data in cross-domain transfer learning to extract robust image features. Then, a model was built to preoperatively distinguish lung granulomatous nodules (LGNs) from lung adenocarcinoma (LAC) in solitary pulmonary solid nodules (SPSNs). Data from 841 patients with SPSNs from five centres were collected retrospectively. First, adaptive cross-domain transfer learning was used to construct transfer learning signatures (TLS) under different source domain data and conduct a comparative analysis. The Wasserstein distance was used to assess the similarity between the source domain and target domain data in cross-domain transfer learning. Second, a cross-domain transfer learning radiomics model (TLRM) combining the best performing TLS, clinical factors and subjective CT findings was constructed. Finally, the performance of the model was validated through multicentre validation cohorts. Relative to other source domain data, TLS based on lung whole slide images as source domain data (TLS-LW) had the best performance in all validation cohorts (AUC range: 0.8228-0.8984). Meanwhile, the Wasserstein distance of TLS-LW was 1.7108, which was minimal. Finally, TLS-LW, age, spiculated sign and lobulated shape were used to build the TLRM. In all validation cohorts, The AUC ranges were 0.9074-0.9442. Compared with other models, decision curve analysis and integrated discrimination improvement showed that TLRM had better performance. The TLRM could assist physicians in preoperatively differentiating LGN from LAC in SPSNs. Furthermore, compared with other images, cross-domain transfer learning can extract robust image features when using lung whole slide images as source domain data and has a better effect.

The successful application of traditional machine learning to mechanical fault diagnosis relies on two conditions: the same probability distribution of training data and testing data, and the data containing fault information has labels. However, it is difficult to obtain massive labeled data, and the change of mechanical operation conditions also results in inconsistent distribution of source domain data and target domain data, which makes the labeled data for training model possibly fail in classifying unlabeled data acquired under other conditions. Aiming at solving the above problems, a deep convolution variational autoencoder network is introduced, the pseudo-label information of small samples in target domain is predicted by using label propagation and data fusion methods, combining with the domain adaptability advantages of transfer learning theory, a novel diagnosis method based on transfer learning with deep convolution variational autoencoder (TL-DCVAEN) is presented. In addition, the spectrum data is used as the input of the model to reduce the dependence on artificial feature design and engineering experience. The experimental results indicate that the proposed diagnosis method is suitable for rolling bearing fault diagnosis under variable working conditions, and has better diagnostic performance and generalization.

Transfer learning can realize cross-domain fault diagnosis of rotating machinery, where the model trained on many labeled samples collected in one working condition can be transferred to insufficient samples collected in the target working condition. Currently, the data features cannot be completely extracted by existing methods when the data distribution gap of the samples collected in different working conditions is quite large. In order to fully extract the data features of rotating machinery to achieve cross-domain fault diagnosis, this paper investigated a cross-domain fault diagnosis model of rotating machinery based on graph feature extraction. The proposed method can realize unsupervised fault diagnosis on rotating machinery running under different working conditions by extracting the numerical and structural features of source and target domains. First of all, data features with large data distribution gaps need to be fully extracted, so a convolutional network based on a deformable convolutional network (De-conv) is designed to extract the features with large differences in data distribution under various working conditions. Secondly, features are extracted based on a convolutional neural network for data values in existing domain adaptation (DA) methods while the structure features of machine monitoring data are ignored. Therefore, a composite spectral-based graph convolutional network is designed to extract structural features of data. Thirdly, fully extracted features are input into a universal DA network to achieve cross-domain fault diagnosis of unknown faults in rotating machinery under changing working conditions. Finally, a benchmarking data set and a data set collected from a practical experimental platform are used to verify the effectiveness of the proposed model, and the results show that it is more suitable for cross-domain fault diagnosis of rotating machinery than other comparison models.

Traditional cross-domain transfer learning faces significant challenges in rotating machinery fault diagnosis due to the general scarcity of fault samples. Furthermore, under complex operating conditions, differences in data distribution between the source domain and target domain can also cause negative migration. To overcome these limitations, this paper proposes a Multi-Source Contrastive Learning Domain Adaptation (MSCLDA) method. Firstly, the MSCLDA approach employs a Wasserstein Generative Adversarial Network (WGAN) to synthesize minority-class fault samples. Subsequently, it integrates a Supervised Contrastive Learning (SCL) strategy to optimize feature consistency between real and synthetic samples, thereby effectively mitigating the training bias caused by category imbalance. Furthermore, we devise a prototype-based multi-pseudo label self-correction mechanism. By incorporating this mechanism with a multiple pseudo-label-guided local maximum mean discrepancy (MP-LMMD) strategy, we achieve precise alignment of feature distributions between the source domains and target subdomains. Finally, an adaptive weighting mechanism is introduced to assign higher weights to source domains more relevant to the target domain, thereby reducing the adverse impact of less relevant domains and mitigating negative transfer. Cross-domain experiments on current signals from industrial robots and vibration signals from bearings show that MSCLDA achieves superior fault identification performance under various working conditions, with an average accuracy of up to 99.05%, and effectively suppresses negative transfer.

Domain-transfer learning is a machine learning task to explore a source domain data set to help the learning problem in a target domain. Usually, the source domain has sufficient labeled data, while the target domain does not. In this paper, we propose a novel domain-transfer convolutional model by mapping a target domain data sample to a proxy in the source domain and applying a source domain model to the proxy for the purpose of prediction. In our framework, we firstly represent both source and target domains to feature vectors by two convolutional neural networks and then construct a proxy for each target domain sample in the source domain space. The proxy is supposed to be matched to the corresponding target domain sample convolutional representation vector well. To measure the matching quality, we proposed to maximize their squared-loss mutual information (SMI) between the proxy and target domain samples. We further develop a novel neural SMI estimator based on a parametric density ratio estimation function. Moreover, we also propose to minimize the classification error of both source domain samples and target domain proxies. The classification responses are also smoothened by manifolds of both the source domain and proxy space. By minimizing an objective function of SMI, classification error, and manifold regularization, we learn the convolutional networks of both source and target domains. In this way, the proxy of a target domain sample can be matched to the source domain data and thus benefits from the rich supervision information of the source domain. We design an iterative algorithm to update the parameters alternately and test it over benchmark data sets of abnormal behavior detection in video, Amazon product reviews sentiment analysis, etc.

Timely and accurate crop mapping is crucial for providing essential data support for agricultural production management. Reliable ground truth samples form the foundation for crop mapping using remote sensing imagery, a task that presents significant challenges in regions with limited sample availability. To address this issue, this study evaluates instance-based transfer learning methods, using the Hexi Corridor as a case study to explore crop mapping strategies in areas with scarce samples. High-confidence pixels from the United States Cropland Data Layer (CDL), along with high-density time series data derived from Sentinel-1, Sentinel-2, and Landsat-8 satellite imagery, as well as key vegetation indices, were selected as training samples for the source domain. Various algorithms, including Random Forest (RF), Extreme Gradient Boosting (XGBoost), and TrAdaBoost, were employed to transfer knowledge from the source domain to the target domain for crop type mapping. The results demonstrated that during the transfer learning process using only source domain data—without utilizing any target domain data—the overall classification accuracy reached 73.88%, with optimal accuracies for maize and alfalfa at 88.97% and 85.23%, respectively. As target domain data were gradually incorporated, the total accuracy for all models ranged from 0.77 to 0.92, with F1-scores ranging from 0.76 to 0.92, showing a consistent improvement in model performance. This study highlights the feasibility of employing transfer learning for crop mapping in the Hexi Corridor, demonstrating its potential to reduce labeling costs for target domain samples and providing a valuable reference for crop mapping in regions with limited sample availability.