A strategy for training 3D object recognition models with limited training data using transfer learning

The proliferation of sensors brings an immense volume of spatio-temporal (ST) data in many domains, including monitoring, diagnostics, and prognostics applications. Data curation is a time-consuming process for a large volume of data, making it challenging and expensive to deploy data analytics platforms in new environments. Transfer learning (TL) mechanisms promise to mitigate data sparsity and model complexity by utilizing pre-trained models for a new task. Despite the triumph of TL in fields like computer vision and natural language processing, efforts on complex ST models for anomaly detection (AD) applications are limited. In this study, we present the potential of TL within the context of high-dimensional ST AD with a hybrid autoencoder architecture, incorporating convolutional, graph, and recurrent neural networks. Motivated by the need for improved model accuracy and robustness, particularly in scenarios with limited training data on systems with thousands of sensors, this research investigates the transferability of models trained on different sections of the Hadron Calorimeter of the Compact Muon Solenoid experiment at CERN. The key contributions of the study include exploring TL’s potential and limitations within the context of encoder and decoder networks, revealing insights into model initialization and training configurations that enhance performance while substantially reducing trainable parameters and mitigating data contamination effects.

Production forecasting is vital in the oil and gas sector, empowering engineers with insights for effective reservoir management. This paper introduces the concept of Transfer Learning as a powerful tool in the domain of machine-learning-assisted production forecasting that accounts of 3D spatial distributions of three geological properties, namely porosity, permeability, and saturation, two completion parameters, namely hydraulic fracture height and length, and production constraint. Transfer learning efficiently leverages knowledge from one problem to improve generalization on another, especially when data is scarce and computational resources are limited. To demonstrate the utility of transfer learning, we evaluate two scenarios of transfer learning. The first transfer learning scenario demonstrates the generalization of the forecasting to cases with variable hydraulic fracture spacing using limited training data. The second transfer learning scenario demonstrates the generalization of the forecasting to cases with variable natural fracture spacing and natural fracture permeability using limited training data. Source dataset contained 2000 realizations, while the target dataset contained 20, 40, 60, 80, 100, 250, 500, or 1000 realizations to represent the scenarios of limited training-data availability. The study confirms the benefits of transfer learning when the training dataset size is small (generally less than 100 training realizations); however, under large training dataset size (around 500 or more training realizations), transfer learning is not needed. For the first scenario involving variable hydraulic fracture spacing, the use of transfer learning ensured that the source model can be trained on target dataset with 80 realizations for gas rate forecasting at an accuracy of 25% in terms of MAPE, and with 500 realizations for condensate rate forecasting at an accuracy of 12% in terms of MAPE. Similarly, for the second scenario involving variable natural fracture spacing and natural fracture permeability, the use of transfer learning ensured that the source model can be trained on target dataset with 250 realizations for gas rate forecasting at an accuracy of 24% in terms of MAPE, and with 500 realizations for condensate rate forecasting at an accuracy of 23% in terms of MAPE. This illustrates the potential of transfer learning in improving forecasting models with limited data using a well pre-trained model and enhanced hyperparameter tuning of the transfer learning model. For cases with 500 or more training realizations, transfer learning severely underperforms as compared to training a conventional machine-learning model from scratch. The paper explores two cases of transfer learning.

A convolutional neural network (CNN) is a deep learning (DL) method that has been widely and successfully applied to computer vision tasks including object localization, detection, and image classification. DL for supervised learning tasks is a method that uses the raw data to determine the classification features, in contrast to other machine learning (ML) techniques that require pre-selection of the input features (or attributes). In the geosciences, we hypothesize that deep learning will facilitate the analysis of uninterpreted images that have been neglected due to a limited number of experts, such as fossil images, slabbed cores, or petrographic thin sections. We use transfer learning, which employs previously trained models to shorten the development time for subsequent models, to address a suite of geologic interpretation tasks that may benefit from ML. Using two different base models, MobileNet V2 and Inception V3, we illustrate the successful classification of microfossils, core images, petrographic photomicrographs, and rock and mineral hand sample images. ML does not replace the expert geoscientist. The expert defines the labels (interpretations) needed to train the algorithm and also monitors the results to address incorrect or ambiguous classifications. ML techniques provide a means to apply the expertise of skilled geoscientists to much larger volumes of data.

The application of transfer learning to T1ρ MRI tibiofemoral cartilage segmentation.

Active Learning is a machine learning and data mining technique that selects the most informative samples for labeling and uses them as training data; it is especially useful when there are large amount of unlabeled data and labeling them is expensive. Recently, batch-mode active learning, where a set of samples are selected concurrently for labeling, based on their collective merit, has attracted a lot of attention. The objective of batch-mode active learning is to select a set of informative samples so that a classifier learned on these samples has good generalization performance on the unlabeled data. Most of the existing batch-mode active learning methodologies try to achieve this by selecting samples based on certain criteria. In this article we propose a novel criterion which achieves good generalization performance of a classifier by specifically selecting a set of query samples that minimize the difference in distribution between the labeled and the unlabeled data, after annotation. We explicitly measure this difference based on all candidate subsets of the unlabeled data and select the best subset. The proposed objective is an NP-hard integer programming optimization problem. We provide two optimization techniques to solve this problem. In the first one, the problem is transformed into a convex quadratic programming problem and in the second method the problem is transformed into a linear programming problem. Our empirical studies using publicly available UCI datasets and two biomedical image databases demonstrate the effectiveness of the proposed approach in comparison with the state-of-the-art batch-mode active learning methods. We also present two extensions of the proposed approach, which incorporate uncertainty of the predicted labels of the unlabeled data and transfer learning in the proposed formulation. In addition, we present a joint optimization framework for performing both transfer and active learning simultaneously unlike the existing approaches of learning in two separate stages, that is, typically, transfer learning followed by active learning. We specifically minimize a common objective of reducing distribution difference between the domain adapted source, the queried and labeled samples and the rest of the unlabeled target domain data. Our empirical studies on two biomedical image databases and on a publicly available 20 Newsgroups dataset show that incorporation of uncertainty information and transfer learning further improves the performance of the proposed active learning based classifier. Our empirical studies also show that the proposed transfer-active method based on the joint optimization framework performs significantly better than a framework which implements transfer and active learning in two separate stages.

Monitoring of glaciers is crucial as they are an important source of freshwater, an indicator of global warming, and a contributor to sea level rise. Accurate delineation of glaciers plays a crucial role in glacier monitoring and remote sensing is the most appropriate tool to map glaciers.Existing glacier inventories have shortcomings such as unavailability in recent years and data quality. Traditional glacier mapping methods using remote sensing often rely on spectral band ratio techniques or manual digitizing. However, glacier boundaries achieved from manual digitizing are highly affected by human errors. Moreover, in the band ratio technique challenges arise in mapping debris-covered glaciers as traditional optical methods fail to distinguish debris-covered ice from surrounding rock due to their spectral similarities. Therefore, automatic mapping of glaciers is still challenging.Advanced deep learning methods have demonstrated significant advancements in automatic glacier mapping. However, the potential of state-of-the-art deep learning methods in glacier mapping has not yet been fully explored. When it comes to deep learning, one of the challenges is the amount of training data. With the low amount of training data, the results won't be of the desired accuracy. However, it is still possible to obtain good results using a lower amount of training data and the transfer learning technique.This study focuses on glacier mapping in Poiqu Basin (Central Himalaya), using U-Net and transfer learning. To this purpose, Sentinel-2 images and the Shuttle Radar Topography Mission (SRTM) Digital Elevation Model are deployed.The results indicate that transfer learning leads to considerably better results than training the deep learning network from scratch. Moreover, trying different backbones does not considerably affect the results. This study highlights the efficiency of the transfer learning technique, emphasizing its potential and effectiveness in regions with limited training data.

Inventory management faces increasing challenges, including data limitations and demand uncertainty. To enhance inventory forecasting and optimization in supply chain management, this study proposes a Transfer-learning Bayesian Network (TBN) framework that integrates causal inference and transfer learning. Unlike traditional inventory forecasting models that rely on historical data patterns, the proposed framework introduces a causal inference-based Bayesian network to establish explicit causal relationships between sales volume, sales revenue, and inventory levels. To address data scarcity and improve generalization, a novel transfer learning mechanism is incorporated, leveraging a balanced weight coefficient method to optimize model adaptation from a source domain to a target domain. The results indicate that the proposd approach ensures effective knowledge transfer and maintains prediction accuracy with limited training data. The TBN model consistently outperforms traditional machine learning methods and other Bayesian-based models. On the self-constructed dataset, the TBN framework achieved a mean squared error (MSE) of 4.97 and a mean absolute error (MAE) of 2.78, demonstrating superior predictive accuracy. Additionally, an analysis of the balance weight coefficient further validated its role in enhancing transfer learning efficiency and model robustness, which provides a scalable and adaptable solution for intelligent inventory management in supply chain systems.

Image classification and prediction using transfer learning in colab notebook

Deep learning continues to receive increasing attention from researchers and has been successfully applied to many domains. This paper further extends the work from Zabihi Naeini and Prindle (2018) by adopting and examining two classes of Machine Learning techniques and their applications in geoscience with a pragmatic view. These are Transfer Learning and Automated Machine Learning or Auto-ML (Feurer and Klein, 2015). Although machine learning (ML) is known to be most efficient and accurate when trained on a large volume of data, there are cases in practice where ML methods are also implemented with limited available data. In such cases ML algorithms are less efficient in generalising to new data and it is where Transfer Learning can add value. This is shown in an automatic petrophysical interpretation task where Transfer Learning is compared with training from scratch given a new geological area of interest, i.e., a set of wells in a different area. We show the efficiency of Transfer Learning in obtaining a model that generalizes successfully for the new wells investigated.

The paper presents modern approaches to end-of-line (EoL) quality inspection, i.e. the final inspection stage in the manufacturing process of electric motors, based on artificial intelligence. Traditional EoL systems rely on extensive signal processing and expert-defined features and thresholds, which limits their adaptability and increases dependence on specialist knowledge. To address these limitations, several machine learning approaches are discussed, including ensemble-based classification models, transfer learning, and deep neural networks. The application of ensemble models enables automatic feature selection and implicit threshold determination, resulting in a significant reduction of model complexity while maintaining or improving classification accuracy. Transfer learning is shown to be particularly effective in pre-production scenarios, where only limited training data are available, allowing faster commissioning of quality inspection systems and improved fault detection reliability. Furthermore, deep learning methods based on convolutional and recurrent neural networks, trained directly on vibration and acoustic signals represented as Mel-frequency spectrograms, eliminate the need for manual feature engineering and achieve high classification accuracy even in highly imbalanced industrial datasets. The presented results demonstrate that artificial intelligence-based EoL systems can simplify diagnostic procedures, reduce reliance on expert knowledge, improve robustness to product variations, and enhance the overall adaptability and efficiency of industrial quality inspection processes. Keywords: end-of-line quality inspection, electric motors, machine learning, artificial intelligence, transfer learning, neural networks, vibration, sound

Recent studies on urolithiasis detection using deep learning have demonstrated promising accuracy; however, most rely on large-scale labeled imaging datasets. In clinical practice, only limited and partially labeled computed tomography (CT) scans are typically available, restricting the generalizability of conventional supervised models. This study aimed to propose a data-efficient framework for accurate stone detection from a small CT dataset by integrating self-supervised learning (SSL) and transfer learning (TL). A total of 100 abdominal CT scans were analyzed and labeled as stone present or normal by expert radiologists. To learn generalizable feature representations from limited data, a SimCLR-based SSL framework with a ResNet50 backbone was employed. During the SSL stage, the model learned from augmented image pairs without labels to maximize similarity between positive pairs and minimize similarity between negatives. The pretrained encoder was subsequently fine-tuned using labeled data in the TL stage, with the lower layers frozen and higher blocks optimized using a linear classifier. Model training was performed with 5-fold cross-validation, and performance was evaluated using accuracy, precision, recall, F1-score, and area under the receiver operating characteristic curve (AUC). The proposed SSL+TL model achieved the best performance (AUC, 0.95; F1-score, 0.91), significantly outperforming both the random initialization and TL-only models. These findings indicate that SSL pretraining effectively learns robust and transferable representations even with limited data. The proposed framework demonstrates the feasibility of artificial intelligence-based urolithiasis detection in small-data clinical environments. Combining SSL and TL alleviates data scarcity and provides a foundation for developing generalizable and resource-efficient diagnostic models for urological imaging.

Deep learning usually requires big data, with respect to both volume and variety. However, most remote sensing applications only have limited training data, of which a small subset is labeled. Herein, we review three state-of-the-art approaches in deep learning to combat this challenge. The first topic is transfer learning, in which some aspects of one domain, e.g., features, are transferred to another domain. The next is unsupervised learning, e.g., autoencoders, which operate on unlabeled data. The last is generative adversarial networks, which can generate realistic looking data that can fool the likes of both a deep learning network and human. The aim of this article is to raise awareness of this dilemma, to direct the reader to existing work and to highlight current gaps that need solving.

This paper proposes an English-Chinese machine translation research method based on transfer learning. First, it expounds the theory of neural machine translation and transfer learning and related technologies. Neural machine translation is discussed, the advantages and disadvantages of various models are introduced, and the transformer neural machine translation model framework is selected. For low-resource Chinese-English parallel corpus and Tibetan-Chinese parallel corpus, 30 million Chinese-English parallel corpora, 100,000 Chinese-English low-resource parallel corpora, and 100,000 Tibetan-Chinese parallel corpora were used to pretrain the transformer machine translation architecture. The decoders are all composed of 6 identical hidden layers, the initialization of the model parameters is done by the transformer uniform distribution, and the model training uses Adam as the optimizer. In the model transfer part, the parameters with the better effect of the pretrained model are transferred to the low-resource Chinese-English and Tibetan-Chinese machine translation model training, so as to achieve the purpose of knowledge transfer. The results show that the model transfer learning of low-resource Chinese-English parallel corpus improves the translation system's translation by 3.97 BLEU values compared with the translation system without transfer learning at 0.34 BLEU values. Model transfer learning on low-resource Tibetan-Chinese parallel corpus increases the BLEU value by 2.64 BLEU compared to the translation system without transfer learning. The neural machine translation system that uses BPE technology for preprocessing plus model transfer learning is compared to the translation system that only performs transfer learning and shows an improved 0.26 BLEU value.It is verified that the transfer learning method proposed in this paper has a certain improvement in the effect of low-resource Chinese-English and Tibetan-Chinese neural machine translation models.

Due to the latest advancements in monitoring technologies, interest in the possibility of early-detection of quality issues in components has grown considerably in the manufacturing industry. However, implementation of such techniques has been limited outside of the research environment due to the more demanding scenarios posed by production environments. This paper proposes a method of assessing the health of a machining process and the machine tool itself by applying a range of machine learning (ML) techniques to sensor data. The aim of this work is not to provide complete diagnosis of a condition, but to provide a rapid indication that the machine tool or process has changed beyond acceptable limits; making for a more realistic solution for production environments. Prior research by the authors found good visibility of simulated failure modes in a number of machining operations and machine tool fingerprint routines, through the defined sensor suite. The current research set out to utilise this system, and streamline the test procedure to obtain a large dataset to test ML techniques upon. Various supervised and unsupervised ML techniques were implemented using a range of features extracted from the raw sensor signals, principal component analysis and continuous wavelet transform. The latter were classified using convolutional neural networks (CNN); both custom-made networks, and pre-trained networks through transfer learning. The detection and classification accuracies of the simulated failure modes across all classical ML and CNN techniques tested were promising, with all approaching 100% under certain conditions.

The concept of Industry 4.0 uses cyber-physical systems and the Internet of Things to create "smart factories" that enable automated and connected production. However, the complex and diverse nature of cyber-physical systems makes them vulnerable to physical threats, where any fault in their physical components can lead to serious consequences, from financial losses to human safety. To reduce the risk of such failures, it is essential to have techniques in place to monitor the condition of these physical components closely. In recent years, data-driven predictive maintenance methods have become common due to the growth in data and advancements in machine learning. This thesis focuses on fault diagnosis, a key part of predictive maintenance, which aims to detect and identify faults in physical components. Incorporating advanced techniques, including machine learning algorithms and sensor data analysis, enhanced fault diagnosis systems' accuracy and efficiency. However, several real-world barriers challenge the effectiveness of these techniques. This research is structured around three primary objectives. The first objective aims to establish an efficient fault diagnosis system that is highly sensitive to fault occurrences while being resilient to uncertainties such as variability in fault patterns and data imbalance. To address variability in fault patterns, we developed a multi-step approach integrating clustering techniques and an adaptive neuro-fuzzy inference system, which significantly improved diagnosis robustness even under noisy conditions. For imbalanced datasets, a two-step method was introduced, combining permutation entropy for fault detection and envelope analysis for fault isolation. This approach was enhanced by an automatic frequency band selection method leveraging wavelet transform and statistical criteria, resulting in high fault diagnosis accuracy and reliability, even with limited labeled faulty data. The second objective focuses on developing fault diagnosis systems that can maintain high performance despite domain shifts and limited training data in both source and target domains. To address the challenge of domain shifts, a transfer learning technique was introduced, and a systematic literature review was conducted. This study bridges a technical gap by providing a comprehensive taxonomy, state-of-the-art methods, and insights into the challenges of applying transfer learning to predictive maintenance. Additionally, a conceptual framework was proposed to explore scenarios for transferring diagnostic knowledge from different source domains to target domains with insufficient training datasets. In a subsequent study, a new diagnostic framework was developed using digital twins and partial transfer learning to address the lack of real training datasets in both source and target domains as well as label space inconsistencies between them. The framework includes a general approach for developing digital twins of real machines to generate synthetic diagnostic data as source domain data. Furthermore, by incorporating a deep learning network and a double-layer attention mechanism, a partial transfer learning diagnostic model was developed to transfer diagnostic knowledge from the digital twin to the real target domain despite label space inconsistencies. The proposed framework was validated through an industrial case study involving rolling bearings in rotating equipment. The third objective focuses on developing an efficient fault diagnosis system capable of maintaining reliability under unseen and evolving working conditions. To achieve this, a novel self-adaptive fault diagnosis system was proposed, incorporating the MAPE-K loop, a domain generalization network model, and digital twin technologies. This self-adaptive fault diagnosis system leverages the MAPE-K loop to adapt in real time to evolving working conditions. Digital twins were employed to continuously generate data aligned with the real system’s operational context, enabling accurate fault diagnosis in dynamic environments. The domain generalization network model enhances generalization to unseen target working conditions by addressing overfitting and feature loss through multi-domain data augmentation, adversarial learning, and domain discrepancy modules. Validation using multiple rotating machinery datasets demonstrated that the developed fault diagnosis system outperformed state-of-the-art methods across various scenarios. In summary, this dissertation proposes a comprehensive and practical approach to addressing variability in fault patterns, imbalance datasets, data scarcity, domain shifts, and unseen and evolving working conditions challenges in existing fault diagnosis systems.