Abstract 1937: DIME-CT: Self-supervised learning for medical image analysis using patch-based embeddings

Deep learning methods have become an integral part of computer vision and machine learning research by providing significant improvement performed in many tasks such as classification, regression, and detection. These gains have been also observed in the field of remote sensing for Earth observation where most of the state-of-the-art results are now achieved by deep neural networks. However, one downside of these methods is the need for large amounts of annotated data, requiring lots of labor-intensive and expensive human efforts, in particular for specific domains that require expert knowledge such as medical imaging or remote sensing. In order to limit the requirement on data annotations, several self-supervised representation learning methods have been proposed to learn unsupervised image representations that can consequently serve for downstream tasks such as image classification, object detection or semantic segmentation. As a result, self-supervised learning approaches have been considerably adopted in the remote sensing domain within the last few years. In this article, we review the underlying principles developed by various self-supervised methods with a focus on scene classification task. We highlight the main contributions and analyze the experiments, as well as summarize the key conclusions, from each study. We then conduct extensive experiments on two public scene classification datasets to benchmark and evaluate different self-supervised models. Based on comparative results, we investigate the impact of individual augmentations when applied to remote sensing data as well as the use of self-supervised pre-training to boost the classification performance with limited number of labeled samples. We finally underline the current trends and challenges, as well as perspectives of self-supervised scene classification.

Not-so-supervised: A survey of semi-supervised, multi-instance, and transfer learning in medical image analysis.

Deep learning methodologies have significantly advanced the fields of computer vision and machine learning, enhancing performance across various tasks like classification, regression, and detection. In remote sensing for Earth observation, deep neural networks have propelled state-of-the-art results. However, a major drawback is their dependence on large annotated datasets, necessitating extensive human effort, especially in specialized domains like medical imaging or remote sensing. To mitigate this reliance on annotations, several self-supervised representation learning techniques have emerged, aiming to learn unsupervised image representations applicable to downstream tasks such as image classification, object detection, or semantic segmentation. Consequently, self-supervised learning approaches have gained traction in remote sensing. This article surveys the foundational principles of various self-supervised methods, focusing on scene classification tasks. We elucidate key contributions, analyze experimental setups, and synthesize findings from each study. Furthermore, we conduct comprehensive experiments on two public scene classification datasets to evaluate and benchmark different self-supervised models

Background and AimsThe 12-lead electrocardiogram (ECG) remains a cornerstone of cardiac diagnostics, yet existing artificial intelligence (AI) solutions for automated interpretation often lack generalizability, remain closed source, and are primarily trained using supervised learning (SL), which requires extensive labelled datasets and may limit adaptability across diverse clinical settings. Self-supervised learning (SSL) can potentially overcome these limitations by learning robust representations from unlabelled data. To address these challenges, this study developed and compared two open-source foundational ECG models: DeepECG-SL, a supervised multilabel ECG model, and DeepECG-SSL, a self-supervised model.MethodsBoth models were trained on over 1 million ECGs using a standardized preprocessing pipeline and automated free-text extraction from ECG reports to predict 77 cardiac conditions. DeepECG-SSL leveraged unlabelled data through self-supervised contrastive learning and masked lead modelling before fine-tuning for downstream tasks, while DeepECG-SL was trained directly on labelled diagnostic data in an end-to-end fashion. Performance was evaluated across seven private, multilingual healthcare systems and four public ECG repositories, with assessment of fairness by age and sex, and investigation of privacy vulnerabilities as well as memory and compute requirements.ResultsDeepECG-SSL achieved micro-averaged area under the receiver operating characteristic curves (AUROCs) across all 77 cardiac conditions for ECG interpretation of 0.990 [95% confidence interval (CI): 0.990, 0.990] on the internal dataset (MHI-ds), 0.981 (95% CI: 0.981, 0.981) on external public datasets (UKB, CLSA, MIMIC-IV and PTB), and 0.983 (95% CI: 0.983, 0.983) on external private datasets (UW, UCSF, JGH, NYP, MGH, CSH and CHUM), while DeepECG-SL demonstrated AUROCs of 0.992 (95% CI: 0.992, 0.992), 0.980 (95% CI: 0.980, 0.980), and 0.983 (95% CI: 0.983, 0.984), respectively. Fairness analyses revealed minimal disparities (true-positive rate and false-positive rate difference <0.1) across age and sex groups for both models. DeepECG-SSL demonstrated superior performance on limited-data digital biomarker tasks, with the largest improvements in long QT syndrome (LQTS) genotype classification (AUROC 0.931 vs 0.850, P = .026, n = 127 ECGs) and 5 year atrial fibrillation risk prediction (AUROC 0.742 vs 0.734, P < 0.001, n = 132 050 ECGs), while achieving superior performance in left ventricular ejection fraction ≤40% classification (AUROC 0.926 vs 0.917, P < 0.001, n = 25 252 ECGs) and comparable performance in LQTS detection (AUROC 0.767 vs 0.735, P = 0.117, n = 934 ECGs).ConclusionsThis study establishes SSL as a promising paradigm for ECG analysis, particularly in settings with limited annotated data, enhancing accessibility, generalizability, and fairness in AI-driven cardiac diagnostics. By releasing model weights, preprocessing tools, and validation code, this work aims to support robust, data-efficient AI diagnostics across diverse clinical environments and questions.

Due to the limited availability of annotated data and the time-consuming and costly nature of manual labeling, the field of medical imaging faces significant challenges. This issue is particularly pronounced in the case of laryngoscopic images, where the scarcity of labeled data contrasts sharply with the abundance of unlabeled medical images. This study aims to address the problem of limited annotated data in laryngoscopic imaging and improve the accuracy of laryngeal disease classification. To this end, we propose a self-supervised learning (SSL) approach that employs a masked image modeling strategy to train a model capable of extracting meaningful representations from unlabeled data. Specifically, the proposed method adopts the MAE (Masked Autoencoders Are Scalable Vision Learners) framework for pretraining, enabling the model to learn robust features from diverse unlabeled laryngoscopic images. The pretrained model is then applied to a downstream task focused on laryngoscopic image classification, where labeled data are limited. To balance accuracy and computational efficiency, we employ ViT-Wee as the backbone network, containing only 13.4 million parameters. Experimental results demonstrate the effectiveness of the proposed approach, achieving an accuracy of 94.67% in a three-class classification task.

Training deep learning models on medical images heavily depends on experts’ expensive and laborious manual labels. In addition, these images, labels, and even models themselves are not widely publicly accessible and suffer from various kinds of bias and imbalances. In this paper, chest X-ray pre-trained model via self-supervised contrastive learning (CheSS) was proposed to learn models with various representations in chest radiographs (CXRs). Our contribution is a publicly accessible pretrained model trained with a 4.8-M CXR dataset using self-supervised learning with a contrastive learning and its validation with various kinds of downstream tasks including classification on the 6-class diseases in internal dataset, diseases classification in CheXpert, bone suppression, and nodule generation. When compared to a scratch model, on the 6-class classification test dataset, we achieved 28.5% increase in accuracy. On the CheXpert dataset, we achieved 1.3% increase in mean area under the receiver operating characteristic curve on the full dataset and 11.4% increase only using 1% data in stress test manner. On bone suppression with perceptual loss, we achieved improvement in peak signal to noise ratio from 34.99 to 37.77, structural similarity index measure from 0.976 to 0.977, and root-square-mean error from 4.410 to 3.301 when compared to ImageNet pretrained model. Finally, on nodule generation, we achieved improvement in Fréchet inception distance from 24.06 to 17.07. Our study showed the decent transferability of CheSS weights. CheSS weights can help researchers overcome data imbalance, data shortage, and inaccessibility of medical image datasets. CheSS weight is available at https://github.com/mi2rl/CheSS.

Deep learning has revolutionized the detection of diseases and is helping the healthcare sector break barriers in terms of accuracy and robustness to achieve efficient and robust computer-aided diagnostic systems. The application of deep learning techniques empowers automated AI-based utilities requiring minimal human supervision to perform any task related to medical diagnosis of fractures, tumors, and internal hemorrhage; preoperative planning; intra-operative guidance, etc. However, deep learning faces some major threats to the flourishing healthcare domain. This paper traverses the major challenges that the deep learning community of researchers and engineers faces, particularly in medical image diagnosis, like the unavailability of balanced annotated medical image data, adversarial attacks faced by deep neural networks and architectures due to noisy medical image data, a lack of trustability among users and patients, and ethical and privacy issues related to medical data. This study explores the possibilities of AI autonomy in healthcare by overcoming the concerns about trust that society has in autonomous intelligent systems.

Applying self-supervised deep learning improves the processing speed and accuracy of automatic modulation recognition (AMR). It reduces the dependence of previous deep networks on many labeled samples. However, affected by an incomplete signal representation modes set, previous models do not fully utilize the multiview property of signals in self-supervised learning. To deal with this issue, a hybrid-view contrastive model for AMR is proposed in this article based on self-supervised learning framework. First, star video is proposed to complete the set of signal representation modes. Next, a self-supervised learning framework based on hybrid-view contrastive learning, hybrid-view self-supervised framework (HVSF), is established to fully extract the signal features, where signals are augmented across views, including the discrete sequence, image, and video format. Considering the view-exclusive information loss and the model complexity, a weakly contrastive strategy and a Transformer-based view-shared feature extractor are finally constructed. Evaluation on four standard datasets demonstrates that the proposed model, HVSF, outperforms both the self-supervised models and supervised models, affirming its superior performance and stability.

We propose a novel self-supervised method, referred to as Video Cloze Procedure (VCP), to learn rich spatial-temporal representations. VCP first generates “blanks” by withholding video clips and then creates “options” by applying spatio-temporal operations on the withheld clips. Finally, it fills the blanks with “options” and learns representations by predicting the categories of operations applied on the clips. VCP can act as either a proxy task or a target task in self-supervised learning. As a proxy task, it converts rich self-supervised representations into video clip operations (options), which enhances the flexibility and reduces the complexity of representation learning. As a target task, it can assess learned representation models in a uniform and interpretable manner. With VCP, we train spatial-temporal representation models (3D-CNNs) and apply such models on action recognition and video retrieval tasks. Experiments on commonly used benchmarks show that the trained models outperform the state-of-the-art self-supervised models with significant margins.

Medical image analysis using deep neural networks (DNN) has demonstrated state-of-the-art performance in image classification and segmentation tasks, aiding disease diagnosis. The accuracy of the DNN is largely governed by the quality and quantity of the data used to train the model. However, for the medical images, the critical security and privacy concerns regarding sharing of local medical data across medical establishments precludes exploiting the full DNN potential for clinical diagnosis. The federated learning (FL) approach enables the use of local model's parameters to train a global model, while ensuring data privacy and security. In this paper, we review the federated learning applications in medical image analysis with DNNs, highlight the security concerns, cover some efforts to improve FL model performance, and describe the challenges and future research directions.

Contrastive learning is an important technique in the field of machine learning for learning data representations. In the field of self-supervised visual representation learning, the strategy of positive and negative sample selection is key to improving the efficiency and effectiveness of model learning. Traditional self-supervised learning methods often employ random sampling strategies to select positive and negative samples, but this approach may result in uneven sample quality when dealing with complex datasets, thereby affecting the learning outcomes. To alleviate this problem, this study is dedicated to exploring more effective strategies for positive and negative sample selection and processing to optimize the self-supervised learning process. To this end, we propose an improved method of self-supervised learning called contrastive learning with enhanced diversity. On the one hand, this method utilizes the weight parameters of the DINO pre-trained model to initialize the feature extraction network of SimCLR, providing more accurate calculations of feature similarity. On the other hand, by setting a threshold on the feature similarity matrix and penalizing (subtracting 0.5 from) similarity scores that do not exceed this threshold, we reduce the excessive impact of high similarity scores on model training, thereby helping the model better distinguish between positive and negative samples. In downstream image classification tasks, we conducted detailed evaluations of the improved model, specifically including fine-tuning and linear evaluation aspects. Experiments show that the proposed approach improves the performance of the loss functions and improves the accuracy of the proposed SimCLR model.

Manual annotation of medical image datasets is labor-intensive and prone to biases. Moreover, the rate at which image data accumulates significantly outpaces the speed of manual annotation, posing a challenge to the advancement of machine learning, particularly in the realm of supervised learning. Self-supervised learning is an emerging field that capitalizes on unlabeled data for training, thereby circumventing the need for extensive manual labeling. This learning paradigm generates synthetic pseudo-labels through pretext tasks, compelling the network to acquire image representations in a pseudo-supervised manner and subsequently fine-tuning with a limited set of annotated data to achieve enhanced performance. This review begins with an overview of prevalent types and advancements in self-supervised learning, followed by an exhaustive and systematic examination of methodologies within the medical imaging domain from 2018 to September 2024. The review encompasses a range of medical image modalities, including CT, MRI, X-ray, Histology, and Ultrasound. It addresses specific tasks, such as Classification, Localization, Segmentation, Reduction of False Positives, Improvement of Model Performance, and Enhancement of Image Quality. The analysis reveals a descending order in the volume of related studies, with CT and MRI leading the list, followed by X-ray, Histology, and Ultrasound. Except for CT and MRI, there is a greater prevalence of studies focusing on contrastive learning methods over generative learning approaches. The performance of MRI/Ultrasound classification and all image types segmentation still has room for further exploration. Generally, this review can provide conceptual guidance for medical professionals to combine self-supervised learning with their research.

Precision agriculture requires accurate discrimination between winter crops and weeds, but there is a lack of annotated image data for winter cropping systems. This paper investigates a two-stage deep learning approach that integrates self-supervised feature learning with supervised fine-tuning for winter crop and weed image classification. A new winter crop and weed image dataset, WinterCropWeedDB, is proposed and used in this paper, which contains 1,136 high-resolution images of six winter crop species and four weed species collected from agricultural fields in central India. In the first stage of self-supervised learning, an EfficientNet-B3 model is pre-trained using a SimCLR-style self-supervised learning approach with an InfoNCE loss function (temperature τ = 0.5) on the images. The average contrastive loss value reduces from 2.0712 in the first iteration to 1.6835 at the end of pretraining. In the second stage of supervised fine-tuning, the pre-trained EfficientNet-B3 model is fine-tuned with a supervised classifier head on the images and tested on a single internal validation split (30%) of the dataset. The fine-tuned model reaches a maximum validation accuracy of 98.27%, with a macro-averaged F1-score of 0.98. Gradient-weighted class activation mapping (Grad-CAM) and Grad-CAM++ are used on the fine-tuned model to provide a qualitative visualization of the image regions that contribute to class predictions. The experiment outcomes demonstrate the viability of using self-supervised pretraining and supervised fine-tuning for the classification of winter crop and weed images on a region-specific dataset, while also emphasizing the importance of additional testing on independent test sets.

To successfully train a deep neural network, a large amount of human-labeled data is required. Unfortunately, in many areas, collecting and labeling data is a difficult and tedious task. Several ways have been developed to mitigate the problem associated with the shortage of data, the most common of which is transfer learning. However, in many cases, the use of transfer learning as the only remedy is insufficient. In this study, we improve deep neural models training and increase the classification accuracy under a scarcity of data by the use of the self-supervised learning technique. Self-supervised learning allows an unlabeled dataset to be used for pretraining the network, as opposed to transfer learning that requires labeled datasets. The pretrained network can be then fine-tuned using the annotated data. Moreover, we investigated the effect of combining the self-supervised learning approach with transfer learning. It is shown that this strategy outperforms network training from scratch or with transfer learning. The tests were conducted on a very important and sensitive application (skin lesion classification), but the presented approach can be applied to a broader family of applications, especially in the medical domain where the scarcity of data is a real problem.

Recent studies on urolithiasis detection using deep learning have demonstrated promising accuracy; however, most rely on large-scale labeled imaging datasets. In clinical practice, only limited and partially labeled computed tomography (CT) scans are typically available, restricting the generalizability of conventional supervised models. This study aimed to propose a data-efficient framework for accurate stone detection from a small CT dataset by integrating self-supervised learning (SSL) and transfer learning (TL). A total of 100 abdominal CT scans were analyzed and labeled as stone present or normal by expert radiologists. To learn generalizable feature representations from limited data, a SimCLR-based SSL framework with a ResNet50 backbone was employed. During the SSL stage, the model learned from augmented image pairs without labels to maximize similarity between positive pairs and minimize similarity between negatives. The pretrained encoder was subsequently fine-tuned using labeled data in the TL stage, with the lower layers frozen and higher blocks optimized using a linear classifier. Model training was performed with 5-fold cross-validation, and performance was evaluated using accuracy, precision, recall, F1-score, and area under the receiver operating characteristic curve (AUC). The proposed SSL+TL model achieved the best performance (AUC, 0.95; F1-score, 0.91), significantly outperforming both the random initialization and TL-only models. These findings indicate that SSL pretraining effectively learns robust and transferable representations even with limited data. The proposed framework demonstrates the feasibility of artificial intelligence-based urolithiasis detection in small-data clinical environments. Combining SSL and TL alleviates data scarcity and provides a foundation for developing generalizable and resource-efficient diagnostic models for urological imaging.