Self-Supervised Learning to Increase the Performance of Skin Lesion Classification

Deep learning has emerged as a powerful tool in various domains, revolutionising machine learning research. However, one persistent challenge is the scarcity of labelled training data, which hampers the performance and generalisation of deep learning models. To address this limitation, researchers have developed innovative methods to overcome data scarcity and enhance deep model learning capabilities. Two prevalent techniques that have gained significant attention are transfer learning and self-supervised learning. Transfer learning leverages knowledge learned from pre-training on a large-scale dataset, such as ImageNet, and applies it to a target task with limited labelled data. This approach allows models to benefit from the learned representations and effectively transfer knowledge to new tasks, resulting in improved learning performance and generalisation. On the other hand, self-supervised learning focuses on training models using pretext tasks that do not require manual annotation, allowing them to learn valuable representations from large amounts of unlabelled data. These learned representations can then be fine-tuned for downstream tasks, mitigating the need for extensive labelled data. In recent years, transfer and self-supervised learning have found applications in various fields, including medical image processing, video recognition, and natural language processing. These approaches have demonstrated remarkable achievements, enabling breakthroughs in areas such as disease diagnosis, object recognition, and language understanding. However, while these methods offer numerous advantages, they also have limitations. For example, transfer learning may face domain mismatch issues between the pre-training and target domains, while self-supervised learning requires careful design of pretext tasks to ensure meaningful representations. This review paper explores the recent applications of these pre-training methods in various fields within the past three years. It delves into the advantages and limitations of each approach, assesses the performance of models employing these techniques, and identifies potential directions for future research. By providing a comprehensive review of current pre-training methods, this article offers guidance for selecting the best technique for specific deep learning applications to address the data scarcity issue.

Whether tracking patient progress for clinical decision making, or investigating novel therapies, automated analysis of Computed Tomography (CT) imaging data is essential for the future of digital radiomics. In digital radiomics, as in all medical imaging, well annotated data is scarce, whereas unlabelled images are relatively plentiful. In other fields of image processing and medical imaging, the application of self-supervised learning (SSL) to large quantities of unlabelled data has resulted in great strides forward for fast, scalable, and interpretable image analysis. In this work we present a new approach applying SSL to CT imaging data which allows for: 1) Improved performance on image classification tasks, based on 2) dramatically reduced quantity of annotated CT imaging data, whilst also 3) enabling easy exploration and interpretation of the image regions. We applied a selection of self-supervised approaches (BYOL, DINO, SimCLR, & inpainting) to CT imaging data. Because the high dimensionality of CT data prevents us from using them directly to out-of the box SSL models, we adopt a 3D patching approach to reduce the dimensionality of the neural net input, and process each patch independently. We train our self-supervised models on public datasets (DeepLesion, NSCLC), and we specialize these models for tumor classification tasks that we evaluate on AstraZeneca sponsored clinical trials. The specialization is done in two different ways: 1) we use the pre-trained SSL model as an encoder that transforms the images of the clinical study into embeddings, on which we apply supervised classification models; 2) we use transfer learning to fine-tune supervised classification models that take the patches directly as inputs. We find that self-supervised pre-training significantly improves the accuracy on tumor classification tasks compared against a supervised learning baseline. Additionally, using the SSL embeddings we build an interactive map of CT imaging data enabling quick and intuitive inspection of the relevant regions. Our findings show that SSL constitutes an important tool for medical imaging analysis. SSL results in models that generalize better, and enable improved downstream interpretability and predictions. Furthermore, well trained SSL models can be re-applied to multiple indications because they are pre-trained on broad and diverse CT imaging data. Citation Format: Leon Fedden, Zhenning Zhang, Khan Baykaner, Qin Li, Lucas Bordeaux. DIME-CT: Self-supervised learning for medical image analysis using patch-based embeddings [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2022; 2022 Apr 8-13. Philadelphia (PA): AACR; Cancer Res 2022;82(12_Suppl):Abstract nr 1937.

Recent studies on urolithiasis detection using deep learning have demonstrated promising accuracy; however, most rely on large-scale labeled imaging datasets. In clinical practice, only limited and partially labeled computed tomography (CT) scans are typically available, restricting the generalizability of conventional supervised models. This study aimed to propose a data-efficient framework for accurate stone detection from a small CT dataset by integrating self-supervised learning (SSL) and transfer learning (TL). A total of 100 abdominal CT scans were analyzed and labeled as stone present or normal by expert radiologists. To learn generalizable feature representations from limited data, a SimCLR-based SSL framework with a ResNet50 backbone was employed. During the SSL stage, the model learned from augmented image pairs without labels to maximize similarity between positive pairs and minimize similarity between negatives. The pretrained encoder was subsequently fine-tuned using labeled data in the TL stage, with the lower layers frozen and higher blocks optimized using a linear classifier. Model training was performed with 5-fold cross-validation, and performance was evaluated using accuracy, precision, recall, F1-score, and area under the receiver operating characteristic curve (AUC). The proposed SSL+TL model achieved the best performance (AUC, 0.95; F1-score, 0.91), significantly outperforming both the random initialization and TL-only models. These findings indicate that SSL pretraining effectively learns robust and transferable representations even with limited data. The proposed framework demonstrates the feasibility of artificial intelligence-based urolithiasis detection in small-data clinical environments. Combining SSL and TL alleviates data scarcity and provides a foundation for developing generalizable and resource-efficient diagnostic models for urological imaging.

The classification of marine plankton images is of great significance in ecological studies and environmental monitoring. In practical applications, plankton image classification faces several challenges, including sample imbalance, distinguishing between-class and within-class differences, and recognizing fine-grained features. To address these issues, we propose a few-shot self-supervised transfer learning (FSTL) framework. In FSTL, we design a new loss function that incorporates both supervised and self-supervised learning. The core of FSTL is a hybrid learning objective that integrates self-supervised contrastive learning for robust feature representation. Operating within a transfer learning paradigm, FSTL effectively adapts knowledge from head-classes to boost the few-shot classification performance on tail-classes. We applied FSTL to two datasets, in which plankton images were collected from Daya Bay and provided by the Woods Hole Oceanographic Institution (WHOI) datasets respectively. The experimental results demonstrated that our method showed better adaptability in the classification of plankton images. The findings of this study not only apply to the classification of plankton images but also offer the potential for classifying small-sample categories within long-tailed datasets.

Considering several patients screened due to COVID-19 pandemic, computer-aided detection has strong potential in assisting clinical workflow efficiency and reducing the incidence of infections among radiologists and healthcare providers. Since many confirmed COVID-19 cases present radiological findings of pneumonia, radiologic examinations can be useful for fast detection. Therefore, chest radiography can be used to fast screen COVID-19 during the patient triage, thereby determining the priority of patient's care to help saturated medical facilities in a pandemic situation. In this paper, we propose a new learning scheme called self-supervised transfer learning for detecting COVID-19 from chest X-ray (CXR) images. We compared six self-supervised learning (SSL) methods (Cross, BYOL, SimSiam, SimCLR, PIRL-jigsaw, and PIRL-rotation) with the proposed method. Additionally, we compared six pretrained DCNNs (ResNet18, ResNet50, ResNet101, CheXNet, DenseNet201, and InceptionV3) with the proposed method. We provide quantitative evaluation on the largest open COVID-19 CXR dataset and qualitative results for visual inspection. Our method achieved a harmonic mean (HM) score of 0.985, AUC of 0.999, and four-class accuracy of 0.953. We also used the visualization technique Grad-CAM++ to generate visual explanations of different classes of CXR images with the proposed method to increase the interpretability. Our method shows that the knowledge learned from natural images using transfer learning is beneficial for SSL of the CXR images and boosts the performance of representation learning for COVID-19 detection. Our method promises to reduce the incidence of infections among radiologists and healthcare providers.

Deep learning methods have become an integral part of computer vision and machine learning research by providing significant improvement performed in many tasks such as classification, regression, and detection. These gains have been also observed in the field of remote sensing for Earth observation where most of the state-of-the-art results are now achieved by deep neural networks. However, one downside of these methods is the need for large amounts of annotated data, requiring lots of labor-intensive and expensive human efforts, in particular for specific domains that require expert knowledge such as medical imaging or remote sensing. In order to limit the requirement on data annotations, several self-supervised representation learning methods have been proposed to learn unsupervised image representations that can consequently serve for downstream tasks such as image classification, object detection or semantic segmentation. As a result, self-supervised learning approaches have been considerably adopted in the remote sensing domain within the last few years. In this article, we review the underlying principles developed by various self-supervised methods with a focus on scene classification task. We highlight the main contributions and analyze the experiments, as well as summarize the key conclusions, from each study. We then conduct extensive experiments on two public scene classification datasets to benchmark and evaluate different self-supervised models. Based on comparative results, we investigate the impact of individual augmentations when applied to remote sensing data as well as the use of self-supervised pre-training to boost the classification performance with limited number of labeled samples. We finally underline the current trends and challenges, as well as perspectives of self-supervised scene classification.

Electrocardiogram (ECG) is acquired during Magnetic Resonance Imaging (MRI) to monitor patients and synchronize image acquisition with the heart motion. ECG signals are highly distorted during MRI due to the complex electromagnetic environment. Automated ECG analysis is therefore complicated in this context and there is no reference technique in MRI to classify pathological heartbeats. Imaging arrhythmic patients is hence difficult in MRI. Deep Learning based heartbeat classifier have been suggested but require large databases whereas existing annotated sets of ECG in MRI are very small. We proposed a Siamese network to leverage a large database of unannotated ECGs outside MRI. This was used to develop an efficient representation of ECG signals, further used to develop a heartbeat classifier. We extensively tested several data augmentations and self-supervised learning (SSL) techniques and assessed the generalization of the obtained classifier to ECG signals acquired in MRI. These augmentations included random noises and a model simulating MRI specific artefacts. SSL pretraining improved the generalizability of heartbeat classifiers in MRI (F1=0.75) compared to Deep Learning not relying on SSL (F1=0.46) and another classical machine learning approach (F1=0.40). These promising results seem to indicate that the use of SSL techniques can learn efficient ECG signal representation, and are useful for the development of Deep Learning models even when only scarce annotated medical data are available.

According to Skin Cancer Foundation, skin cancer is by far the most common type of cancer in the United States and worldwide. Early diagnosis of skin cancer is critical because proper treatment at early stages can increase the chance of cure and recovery. However, visual inspection of dermoscopic images by dermatologists is error-prone and time-consuming. To ensure accurate diagnosis and faster treatment of skin cancer, deep learning techniques have been utilized to conduct automated skin lesion segmentation and classification. In this paper, after image processing, a Mask R-CNN model is built for lesion segmentation, where transfer learning is utilized by using the pre-trained weights from Microsoft COCO dataset. The weights of the trained Mask R-CNN model are saved and transferred to the next task - skin lesion classification, to train a Mask R-CNN model for classification. Our experiments are conducted on the benchmark datasets from the International Skin Imaging Collaboration 2018 (ISIC 2018) and evaluated by the same metrics used in ISIC 2018. The lesion boundary segmentation and lesion classification have achieved an accuracy of 96% and a balanced multiclass accuracy of 80%, respectively.

A hybrid approach for multi modal brain tumor segmentation using two phase transfer learning, SSL and a hybrid 3DUNET

Agricultural image processing technology plays a critical role in enabling precise disease detection, accurate yield prediction, and various smart agriculture applications. However, its practical implementation faces key challenges, including environmental interference, data scarcity and imbalance datasets, and the difficulty of deploying models on resource-constrained edge devices. This paper presents a systematic review of recent advances in addressing these challenges, with a focus on three core aspects: environmental robustness, data efficiency, and model deployment. The study identifies that attention mechanisms, Transformers, multi-scale feature fusion, and domain adaptation can enhance model robustness under complex conditions. Self-supervised learning, transfer learning, GAN-based data augmentation, SMOTE improvements, and Focal loss optimization effectively alleviate data limitations. Furthermore, model compression techniques such as pruning, quantization, and knowledge distillation facilitate efficient deployment. Future research should emphasize multi-modal fusion, causal reasoning, edge–cloud collaboration, and dedicated hardware acceleration. Integrating agricultural expertise with AI is essential for promoting large-scale adoption, as well as achieving intelligent, sustainable agricultural systems.

Passive acoustic monitoring combined with deep learning-based bird sound classifiers is an effective tool, particularly in remote areas. While self-supervised learning has recently excelled in natural language processing and image recognition, its application to bird sound recognition remains limited. This study proposes an innovative self-supervised learning approach, which leverages vast amounts of passive acoustic recordings for pre-training, followed by fine-tuning of target species. Compared to the three state-of-the-art models based on transfer learning from ImageNet, the proposed method demonstrated improvements across all species, with even more significant gains for tail-end species. These results confirm that domain-specific pre-training in self-supervised learning enhances downstream recognition tasks and provides greater robustness, benefiting tail-end species in imbalanced ecological datasets. Our experiments further demonstrate that integrating open-source datasets and data augmentation techniques is the most effective strategy for mitigating data imbalances and cross-domain issues. In addition, introducing a ‘catch-all’ category into training datasets has been shown to improve model robustness in open set recognition scenarios. We also identified the minimum viable sample size requirements for our proposed model and explored the impact of overlapping bird vocalizations during dawn choruses on model performance. Targeting 31 bird species in the montane regions of subtropical Taiwan, the model achieved a class-wise mean average precision of 0.782 and an overall precision of 85.6 % at the F0.5 threshold in dawn chorus soundscape recordings. This study confirms the effectiveness and advantages of self-supervised learning in bird sound recognition, supporting long-term monitoring of bird distribution and vocal activity in remote montane areas.

This paper proposes a novel self-supervised learning framework that uses model reinforcement, REference-free LAtent map eXtraction with MOdel REinforcement (RELAX-MORE), for accelerated quantitative MRI (qMRI) reconstruction. The proposed method uses an optimization algorithm to unroll an iterative model-based qMRI reconstruction into a deep learning framework, enabling accelerated MR parameter maps that are highly accurate and robust. Unlike conventional deep learning methods which require large amounts of training data, RELAX-MORE is a subject-specific method that can be trained on single-subject data through self-supervised learning, making it accessible and practically applicable to many qMRI studies. Using quantitative mapping as an example, the proposed method was applied to the brain, knee and phantom data. The proposed method generates high-quality MR parameter maps that correct for image artifacts, removes noise, and recovers image features in regions of imperfect image conditions. Compared with other state-of-the-art conventional and deep learning methods, RELAX-MORE significantly improves efficiency, accuracy, robustness, and generalizability for rapid MR parameter mapping. This work demonstrates the feasibility of a new self-supervised learning method for rapid MR parameter mapping, that is readily adaptable to the clinical translation of qMRI.

Training deep neural networks may be challenging in real world data. Using models as black-boxes, even with transfer learning, can result in poor generalization or inconclusive results when it comes to small datasets or specific applications. This tutorial covers the basic steps as well as more recent options to improve models, in particular, but not restricted to, supervised learning. It can be particularly useful in datasets that are not as well-prepared as those in challenges, and also under scarce annotation and/or small data. We describe basic procedures as data preparation, optimization and transfer learning, but also recent architectural choices such as use of transformer modules, alternative convolutional layers, activation functions, wide/depth, as well as training procedures including curriculum, contrastive and self-supervised learning.

PurposeBreast cancer remains one of the most prevalent cancers globally, necessitating effective early screening and diagnosis. This study investigates the effectiveness and generalizability of our recently proposed data augmentation technique, attention-guided erasing (AGE), across various transfer learning classification tasks for breast abnormality classification in mammography.MethodsAGE utilizes attention head visualizations from DINO self-supervised pretraining to weakly localize regions of interest (ROI) in images. These localizations are then used to stochastically erase non-essential background information from training images during transfer learning. Our research evaluates AGE across two image-level and three patch-level classification tasks. The image-level tasks involve breast density categorization in digital mammography (DM) and malignancy classification in contrast-enhanced mammography (CEM). Patch-level tasks include classifying calcifications and masses in scanned film mammography (SFM), as well as malignancy classification of ROIs in CEM.ResultsAGE significantly boosts classification performance with statistically significant improvements in mean F1-scores across four tasks compared to baselines. Specifically, for image-level classification of breast density in DM and malignancy in CEM, we achieve gains of 2% and 1.5%, respectively. Additionally, for patch-level classification of calcifications in SFM and CEM ROIs, gains of 0.4% and 0.6% are observed, respectively. However, marginal improvement is noted in the mass classification task, indicating the necessity for further optimization in tasks where critical features may be obscured by erasing techniques.ConclusionOur findings underscore the potential of AGE, a dataset- and task-specific augmentation strategy powered by self-supervised learning, to enhance the downstream classification performance of DL models, particularly involving ViTs, in medical imaging.

Direct speech-to-speech translation (S2ST) is an important tool for bridging communication gaps. Direct S2ST translates speech from one language to another without relying on intermediate text, making it particularly useful for languages primarily spoken rather than written. However, the performance of Direct S2ST models on low-resource languages remains limited due to the scarcity or complete absence of parallel speech data required for training. Pretraining and finetuning are widely used techniques to leverage unsupervised speech data to improve model performance. In this work, we employ a cluster-aided, cross-contrastive self-supervised learning (SSL)-based speech representation model as the pre-trained encoder, combined with a multilingual BART (mBART) decoder. The resulting finetuned model outperforms a baseline that uses a contrastive-loss-based SSL model as the encoder. The proposed models improve the BLEU score by 4.14% for Hindi $$\rightarrow $$ English and 8.2% for English $$\rightarrow $$ Hindi compared to their respective baseline models. To train the model for English-to-Hindi, we trained a unit-vocoder on speech quantized using ensemble clustering instead of standard clustering. The resulting unit-vocoder outperformed the one trained on speech quantized using standard k-means for all evaluation metrics.