Integrating Few-Shot Learning and Pre-trained Models into Similarity-Based PHM using Small Data in Complex Engineering Systems

Prognostics and health management (PHM) represents a paradigm shift from legacy condition based maintenance (CBM) frameworks by expanding the potentials to accurately and robustly detect and diagnose incipient system faults. The ultimate goal of PHM is reliably predicting system failure times to allow for efficient maintenance scheduling either autonomously or by human decision makers (DM). In many industrial settings today the output from PHM systems constitutes a decision support system (DSS) used to aid DM, as entirely autonomous systems have not seen widespread industrial integration. However, there is relatively little support for engineers designing PHM systems in terms of human factors and how to provide the information in a way that actively supports human decision-making and this gap may result in limited use of PHM system by maintainers. The reliability of the information presented is a critical factor in the user acceptance and trust in a system. As a first step in developing such guidance, this paper reviews the implementation of other DSS and presents a design framework whereby PHM reliability levels are mapped against a suggested level of human input to the decision making process regarding required maintenance. The aim is to provide engineers with a guide to the level to which they should consider human factors and the presentation of information in the design of their PHM system. Fundamental to the suggested paradigm is that the uncertainties within a PHM system can be quantified, and as uncertainty increases, the requirement for DM to access additional information not explicitly tied to the PHM output increases. This information can form both explicit and tacit knowledge of a system or indeed industrial contexts surrounding decision implications, such as acceptable maintenance intervention windows in busy production schedules. As the complexity of a system or component being monitored is likely to affect the uncertainty within the PHM system associated with it, we are considering the overall cumulative uncertainty of a model output as the metric through which the required level of human input can be inferred. Coupled to this is the fact that increased model uncertainty is a causal factor in distrust and potential non-use of the model in industrial applications. It is the authors’ belief therefore that designing for increased human-model interaction concurrent with increasing model uncertainty may lead to a better engagement with PHM decision support capabilities, thereby offering the full advantages that PHM has to offer. The framework presented in this paper is an initial step towards facilitating the design of more usable and useful PHM systems.

In the operations and maintenance (O&M) management of industrial robots, data-driven methods often lack interpretability and decision-making support. To address these limitations, this paper proposes a data- and knowledge-co-driven framework for Prognostics and Health Management (PHM). The framework is designed to accommodate the complex structure and multi-source data characteristics of industrial robots, with a focus on three core tasks: condition monitoring, fault diagnosis, and remaining useful life (RUL) prediction. For each task, appropriate deep learning models are selected to fit the PHM architecture. On the knowledge-driven side, a knowledge extraction process based on pre-trained language models is employed. On the knowledge-driven side, a pre-trained language model is employed for knowledge extraction. By integrating equipment structure with fault evolution logic, a knowledge graph is constructed to support the integration and semantic reasoning of operation and maintenance knowledge. This results in a PHM management framework driven by both data and knowledge, providing a feasible design concept and implementation pathway for future research and practical applications.

Self-Supervised Learning for data scarcity in a fatigue damage prognostic problem

Prognostics and Health Management (PHM) is a multidisciplinary field aiming at maintaining physical systems in their optimal functioning conditions. The system under study is assumed to be monitored by sensors from which are obtained measurements reflecting the system’s health state. A health index (HI) is estimated to feed a data-driven PHM solution developed to predict the remaining useful life (RUL). In this paper, the values taken by an HI are assumed imprecise (IHI). An IHI is interpreted as a planar figure called polygon and a case-based reasoning (CBR) approach is adapted to estimate the RUL. This adaptation makes use of computational geometry tools in order to estimate the nearest cases to a given testing instance. The proposed algorithm called RULCLIPPER is assessed and compared on datasets generated by the NASA’s turbofan simulator (C-MAPSS) including the four turbofan testing datasets and the two testing datasets of the PHM’08 data challenge. These datasets represent 1360 testing instances and cover different realistic and difficult cases considering operating conditions and fault modes with unknown characteristics. The problem of feature selection, health index estimation, RUL fusion and ensembles are also tackled. The proposed algorithm is shown to be efficient with few parameter tuning on all datasets.

Aircraft are complex engineering systems composed of many interconnected subsystems with possible uncertainties in their structure. They often function for a long number of flight hours under varying or harsh environments. Hence, prognostic and health management (PHM) of critical subsystems or components within the overall system is crucial for maintaining the safety and reliability of the aircraft. This article reviews the state of the art in aircraft failure prognostic. The main definitions and concepts are presented and discussed. In addition, a selected important failure in the representative aircraft components is outlined, and various categories of prognostic strategies are reviewed. Finally, some recommendations and directions for the most promising research to address the PHM problem in aircraft are outlined.

Prognostics and Health Management (PHM) [1] is a key enabler of Condition Based Maintenance Plus (CBM+) [2]. In essence, it refers to the “Plus” by providing the ability to predict future health status of a system or component, as well as providing the ability to anticipate faults, problems, potential failures, and required maintenance actions. From the perspective of operation and maintenance (O&M) world, the vital knowledge requirements from PHM are indicators of degraded health condition (alarm, warnings, call for inspection, etc.), estimates of time to onset of such indicators, estimate of time to maintenance, and ahead-of-time diagnostics for identification of the root causes (or sources) that will likely cause these maintenance calls. Such knowledge provides lead time to the operators and system maintainers to prepare for inspection and schedule maintenance opportunistically, so as to minimize downtime and optimize maintenance cost. Qualtech Systems, Inc. (QSI) has developed a domain-neutral capability for tracking and trending sensor observations with considerations to operating mode changes, sensor dropouts, and measurement noise. This capability has been introduced to TEAMS® (Testability Engineering and Maintenance Systems) [3] - the health management decision-support software suite of QSI. By leveraging the built-in diagnostic features of TEAMS, this capability provides ‘time-to-alarm’ and ‘time-tomaintenance’ estimates along with the list of potential failure sources (subsystems, components, etc.) responsible for the predicted alarms and maintenance calls. A trend fusion capability has also been introduced for accurately estimating the time to maintenance for components monitored by multiple sensors exhibiting differing observation trends. Introduction of these capabilities facilitates utilization of same dependency model (TEAMS model) for reactive diagnosis and proactive identification of the components that require maintenance within a time horizon set by the operator or the maintainer. This allays the need for developing separate diagnostic and prognostic models, which in general are costly and lengthy work - and thereby offers an efficient and economic enabler for the CBM+ paradigm.

BackgroundNatural language processing (NLP) has become an emerging technology in health care that leverages a large amount of free-text data in electronic health records to improve patient care, support clinical decisions, and facilitate clinical and translational science research. Recently, deep learning has achieved state-of-the-art performance in many clinical NLP tasks. However, training deep learning models often requires large, annotated data sets, which are normally not publicly available and can be time-consuming to build in clinical domains. Working with smaller annotated data sets is typical in clinical NLP; therefore, ensuring that deep learning models perform well is crucial for real-world clinical NLP applications. A widely adopted approach is fine-tuning existing pretrained language models, but these attempts fall short when the training data set contains only a few annotated samples. Few-shot learning (FSL) has recently been investigated to tackle this problem. Siamese neural network (SNN) has been widely used as an FSL approach in computer vision but has not been studied well in NLP. Furthermore, the literature on its applications in clinical domains is scarce.ObjectiveThe aim of our study is to propose and evaluate SNN-based approaches for few-shot clinical NLP tasks.MethodsWe propose 2 SNN-based FSL approaches, including pretrained SNN and SNN with second-order embeddings. We evaluate the proposed approaches on the clinical sentence classification task. We experiment with 3 few-shot settings, including 4-shot, 8-shot, and 16-shot learning. The clinical NLP task is benchmarked using the following 4 pretrained language models: bidirectional encoder representations from transformers (BERT), BERT for biomedical text mining (BioBERT), BioBERT trained on clinical notes (BioClinicalBERT), and generative pretrained transformer 2 (GPT-2). We also present a performance comparison between SNN-based approaches and the prompt-based GPT-2 approach.ResultsIn 4-shot sentence classification tasks, GPT-2 had the highest precision (0.63), but its recall (0.38) and F score (0.42) were lower than those of BioBERT-based pretrained SNN (0.45 and 0.46, respectively). In both 8-shot and 16-shot settings, SNN-based approaches outperformed GPT-2 in all 3 metrics of precision, recall, and F score.ConclusionsThe experimental results verified the effectiveness of the proposed SNN approaches for few-shot clinical NLP tasks.

Despite the increased availability of UAV / drone imagery in Low- to upper  Middle-Income countries and the demonstrated potential of deep learning to support the interpretation of these images for sustainable development purposes, practical operations in these countries are constrained by the need for sufficient labeled data-sets which are often difficult to obtain (especially for tropical forest). This makes it difficult to train suitable networks and assess whether the model is performing well. One such example is the use of drones to monitor the Atlantic Forest in Sao Paulo, Brazil. Here, members of the Sao Paulo Municipal Green and Environment Secretariat (Secretaria do Verde e do Meio Ambiente - SVMA) are starting to use drones to  identify some native and invasive species in their forests. Deep learning will quickly speed up this process, but there is little training data available. This refers to the so called ‘small-data problem’ commonly found in DL for remote sensing applications [1]. A workflow was designed to support this application through a novel zero-shot learning technique and explainable AI methods. A pre-trained tree-crown detection model ‘DeepForest’ [2] is used to identify individual tree crowns in the UAV imagery. The detected tree-crowns are further classified using a Siamese network architecture using zero-shot learning – the model is trained on relevant data-sets but not exposed to species found in the test data-set. A Siamese network architecture is motivated by the need for explainability in DL models – the results will be used for making administrative decision for forest management. A more intricate DL model (such as image segmentation) could be more accurate but at the cost of transparency/explainability. In particular, we apply a variation of the ‘What I Know’ (WIK) explainability method [3] which provides examples from the training set along with the test sample increasing transparency and understanding of the model results.[1] Safonova, Anastasiia, et al. "Ten deep learning techniques to address small data problems with remote sensing." International Journal of Applied Earth Observation and Geoinformation 125 (2023): 103569.[2] Weinstein, Ben G., et al. "DeepForest: A Python package for RGB deep learning tree crown delineation." Methods in Ecology and Evolution 11.12 (2020): 1743-1751.[3] Ishikawa, Shin-nosuke, et al. "Example-based explainable AI and its application for remote sensing image classification." International Journal of Applied Earth Observation and Geoinformation 118 (2023): 103215.  

This study addresses challenges in sentiment analysis for low-resource educational contexts by proposing a framework that integrates Few-Shot Learning (FSL) with Transformer-based ensemble models and boosting techniques. Sentiment analysis of student feedback is crucial for improving teaching quality, yet traditional methods struggle with data scarcity and computational inefficiency. The proposed framework leverages self-attention mechanisms in Transformers and combines models through Gradient Boosting to enhance performance and generalization with minimal labeled data. Evaluated on the UIT-VSFC dataset, comprising Vietnamese student feedback, the framework achieved superior F1-scores in sentiment and topic classification tasks, outperforming individual models. Results demonstrate their potential for extracting actionable insights to enhance educational experiences. Despite its effectiveness, the approach faces limitations such as reliance on pre-trained models and computational complexity. Future work could optimize lightweight models and explore applications in other domains like healthcare and finance.

Few-shot learning for surgical phase recognition: Performance and generalization in cholecystectomy.

Few-shot learning (FSL) is a challenging problem. Transfer learning methods offer a straightforward and effective solution to FSL by leveraging pre-trained models and generalizing them to new tasks. However, pre-trained models often lack the ability to highlight and emphasize salient features, a gap that attention mechanisms can fill. Unfortunately, existing attention mechanisms encounter issues such as high complexity and incomplete attention information. To address these issues, we propose a dimensionally enhanced attention (DEA) module for FSL. This DEA module introduces minimal additional computational overhead while fully attending to both channel and spatial information. Specifically, the feature map is first decomposed into 1D tensors of varying dimensions using strip pooling. Next, a multi-dimensional collaborative learning strategy is introduced, enabling cross-dimensional information interactions through 1D convolutions with adaptive kernel sizes. Finally, the feature representation is enhanced by calculating attention weights for each dimension using a sigmoid function and weighting the original input accordingly. This approach ensures comprehensive attention to different dimensions of information, effectively characterizing data in various directions. Additionally, we have found that knowledge distillation significantly improves FSL performance. To this end, we implement a logit standardization self-distillation method tailored for FSL. This method addresses the issue of exact logit matching, which arises from the shared temperature in the self-distillation process, by employing logit standardization. We present experimental results on several benchmark datasets where the proposed method yields significant performance improvements.

Existing few-shot learning models leverage vision-language pre-trained models to alleviate the data scarcity problem. However, such models usually process visual and text information separately, which causes still inherent disparities between cross-modal features. Therefore, we propose the ProFusion model, which leverages multimodal pre-trained models and prototypical networks to construct multiple prototypes. Specifically, ProFusion generates image and text prototypes symmetrically using the visual encoder and text encoder, while integrating visual and text information through the fusion module to create more expressive multimodal feature fusion prototypes. Additionally, we introduce the alignment module to ensure consistency between image and text prototypes. During inference, ProFusion calculates the similarity of test images to the three types of prototypes separately and applies a weighted sum to generate the final prediction. Experiments demonstrate that ProFusion performs outstanding classification tasks on 15 benchmark datasets.

Few-shot learning(FSL) aims to generalize model to novel categoeries by few labelled samples, which is challenging for machine. Large-scaled pretrained models, especially vision transformers achieve excellent performances benefiting from numerous and diverse data. Researchers have exploited pretrained models in few-shot classification by simply updating the whole parameters and finetuning on few samples. In this paper, we explore two methods: vision prompt tuning and a reparameterization method called ‘scaling&&shift’ to leverage pretrained models in few-shot classification. Vision prompt tuning is for vision transformer only and we first evaluate the method in few-shot setting. ‘Scaling&&shift’ is originally applied in convolution neural networks(CNN). We extend it to vision transformer. The two methods are evaluated on standard benchmarks such as miniImageNet, CUB, CIFAR-FS, clipart and sketch. The results show that ‘scaling&&shift’ reaches the same level compared to updating the whole parameters. Vision prompt tuning is 0%~5% lower than updating the whole parameters over five datasets while it has quite smaller amount of parameters updated.

Unlike in the field of visual scene recognition, where tremendous advances have taken place due to the availability of very large datasets to train deep neural networks, inference from medical images is often hampered by the fact that only small amounts of data may be available. When working with very small dataset problems, of the order of a few hundred items of data, the power of deep learning may still be exploited by using a pre-trained model as a feature extractor and carrying out classic pattern recognition techniques in this feature space, the so-called few-shot learning problem. However, medical images are highly complex and variable, making it difficult for few-shot learning to fully capture and model these features. To address these issues, we focus on the intrinsic characteristics of the data. We find that, in regimes where the dimension of the feature space is comparable to or even larger than the number of images in the data, dimensionality reduction is a necessity and is often achieved by principal component analysis or singular value decomposition (PCA/SVD). In this paper, noting the inappropriateness of using SVD for this setting we explore two alternatives based on discriminant analysis (DA) and non-negative matrix factorization (NMF). Using 14 different datasets spanning 11 distinct disease types we demonstrate that at low dimensions, discriminant subspaces achieve significant improvements over SVD-based subspaces and the original feature space. We also show that at modest dimensions, NMF is a competitive alternative to SVD in this setting. The implementation of the proposed method is accessible via the following link.

An SW-ELM Based Remaining Useful Life Prognostic Approach for Aircraft Engines