ContIG: Self-supervised Multimodal Contrastive Learning for Medical Imaging with Genetics

Semi-supervised learning acts as an effective way to leverage massive unlabeled data. In this paper, we propose a novel training strategy, termed as Semi-supervised Contrastive Learning (SsCL), which combines the well-known contrastive loss in self-supervised learning with the cross entropy loss in semi-supervised learning, and jointly optimizes the two objectives in an end-to-end way. The highlight is that different from self-training based semi-supervised learning that conducts prediction and retraining over the same model weights, SsCL interchanges the predictions over the unlabeled data between the two branches, and thus formulates a co-calibration procedure, which we find is beneficial for better prediction and avoid being trapped in local minimum. Towards this goal, the contrastive loss branch models pairwise similarities among samples, using the pseudo labels generated from the cross entropy branch, and in turn calibrates the prediction distribution of the cross entropy branch with the contrastive similarity. We show that SsCL produces more discriminative representation and is beneficial to semi-supervised learning. Notably, on ImageNet with ResNet50 as the backbone, SsCL achieves 60.2% and 72.1% top-1 accuracy with 1% and 10% labeled samples, respectively, which significantly outperforms the baseline, and is better than previous semi-supervised and self-supervised methods.

DrasCLR: A self-supervised framework of learning disease-related and anatomy-specific representation for 3D lung CT images

Deep learning methodologies have significantly advanced the fields of computer vision and machine learning, enhancing performance across various tasks like classification, regression, and detection. In remote sensing for Earth observation, deep neural networks have propelled state-of-the-art results. However, a major drawback is their dependence on large annotated datasets, necessitating extensive human effort, especially in specialized domains like medical imaging or remote sensing. To mitigate this reliance on annotations, several self-supervised representation learning techniques have emerged, aiming to learn unsupervised image representations applicable to downstream tasks such as image classification, object detection, or semantic segmentation. Consequently, self-supervised learning approaches have gained traction in remote sensing. This article surveys the foundational principles of various self-supervised methods, focusing on scene classification tasks. We elucidate key contributions, analyze experimental setups, and synthesize findings from each study. Furthermore, we conduct comprehensive experiments on two public scene classification datasets to evaluate and benchmark different self-supervised models

Deep learning methods have become an integral part of computer vision and machine learning research by providing significant improvement performed in many tasks such as classification, regression, and detection. These gains have been also observed in the field of remote sensing for Earth observation where most of the state-of-the-art results are now achieved by deep neural networks. However, one downside of these methods is the need for large amounts of annotated data, requiring lots of labor-intensive and expensive human efforts, in particular for specific domains that require expert knowledge such as medical imaging or remote sensing. In order to limit the requirement on data annotations, several self-supervised representation learning methods have been proposed to learn unsupervised image representations that can consequently serve for downstream tasks such as image classification, object detection or semantic segmentation. As a result, self-supervised learning approaches have been considerably adopted in the remote sensing domain within the last few years. In this article, we review the underlying principles developed by various self-supervised methods with a focus on scene classification task. We highlight the main contributions and analyze the experiments, as well as summarize the key conclusions, from each study. We then conduct extensive experiments on two public scene classification datasets to benchmark and evaluate different self-supervised models. Based on comparative results, we investigate the impact of individual augmentations when applied to remote sensing data as well as the use of self-supervised pre-training to boost the classification performance with limited number of labeled samples. We finally underline the current trends and challenges, as well as perspectives of self-supervised scene classification.

Siamese evolutionary masking: Enhancing the generalization of self-supervised medical image segmentation model.

The implementation of computer vision generally focuses on resolving a problem of one of two types of image data, whether it was natural image datasets or medical image datasets. Self-supervised learning (SSL), as the most recent approach in computer vision, is typically used to solved tasks related only to natural image datasets such as ImageNet, CIFAR-10, and PASCAL. This study focuses on analyzing the involvement of natural images in the SSL implementation when classifying medical images. We used the ResNet50 architecture comparing the results of human embryo image classification using the conventional transfer learning method and SSL methods. We apply three SSL approaches, namely Rotation, Jigsaw Puzzle, and SimCLRv1. The results of our research, with 1226 embryo images divided into three classes, show that the SSL implementations cannot be said to be efficient enough compared to the conventional model produced by ResNet50 with pretrained weight ImageNet. However, when the models were trained using random weight initialization, the Rotation model with an average accuracy of 78.15% and the Jigsaw Puzzle model with an average accuracy of 80.30% are significantly superior to the conventional model with average accuracy is 71.07%. Moreover, related to our embryos dataset, we discuss the differences between the SSL context-context contrastive learning approach and the context instance contrastive learning approach.

Spatiotemporal self-supervised predictive learning for atmospheric variable prediction via multi-group multi-attention

Deep learning methods, especially convolutional neural networks, have become more and more popular in medical image classifications. However, training a deep neural network from scratch can be a luxury for many medical image datasets as the process requires a large and well-balanced sample to output satisfactory results. Unlike natural image datasets, medical images are expensive to collect owing to labor and equipment costs. Besides, the class labels in medical image datasets are usually severely imbalanced subject to the availability of patients. Further, aggregating medical images from multiple sources can be challenging due to policy restrictions, privacy concerns, communication costs, and data heterogeneity caused by equipment differences and labeling discrepancies. In this paper, we propose to address these issues with the help of transfer learning and artificial samples created by generative models. Instead of requesting medical images from source data, our method only needs a parsimonious supplement of model parameters pre-trained on the source data. The proposed method preserves the data privacy in the source data and significantly reduces the communication cost. Our study shows transfer learning together with artificial samples can improve the pneumonia classification accuracy on a small but heavily imbalanced chest X-ray image dataset by \(11.53\%\) which performs even better than directly augmenting that source data into the training process.KeywordsDeep learningGenerative modelsMedical image classificationPrivacy preservationTransfer learning

Graph neural networks (GNNs) have achieved extraordinary enhancements in various areas including the fields medical imaging and network neuroscience where they displayed a high accuracy in diagnosing challenging neurological disorders such as autism. In the face of medical data scarcity and high-privacy, training such data-hungry models remains challenging. Federated learning brings an efficient solution to this issue by allowing to train models on multiple datasets, collected independently by different hospitals, in fully data-preserving manner. Although both state-of-the-art GNNs and federated learning techniques focus on boosting classification accuracy, they overlook a critical unsolved problem: investigating the reproducibility of the most discriminative biomarkers (i.e., features) selected by the GNN models within a federated learning paradigm. Quantifying the reproducibility of a predictive medical model against perturbations of training and testing data distributions presents one of the biggest hurdles to overcome in developing translational clinical applications. To the best of our knowledge, this presents the first work investigating the reproducibility of federated GNN models with application to classifying medical imaging and brain connectivity datasets. We evaluated our framework using various GNN models trained on medical imaging and connectomic datasets. More importantly, we showed that federated learning boosts both the accuracy and reproducibility of GNN models in such medical learning tasks. Our source code is available at https://github.com/basiralab/reproducibleFedGNN.KeywordsGraph neural networksFederated LearningReproducibilityBrain connectivity graphsPredictive medicine

Self-supervised learning (SSL) is a challenging task in sleep stage classification (SSC) that is capable of mining valuable representations from unlabeled data. However, traditional SSL methods typically focus on single-view learning and do not fully exploit the interactions among information across multiple views. In this study, we focused on a multi-domain view of the same EEG signal and developed a self-supervised multi-view representation learning framework via time series and time-frequency contrasting (MV-TTFC). In the MV-TTFC framework, we built-in a cross-domain view contrastive learning prediction task to establish connections between the temporal view and time-frequency (TF) view, thereby enhancing the information exchange between multiple views. In addition, to improve the quality of the TF view inputs, we introduced an enhanced multisynchrosqueezing transform, which can create high energy concentration TF image views to compensate for the inaccurate representations in traditional TF processing techniques. Finally, integrating temporal, TF, and fusion space contrastive learning effectively captured the latent features in EEG signals. We evaluated MV-TTFC based on two real-world SSC datasets (SleepEDF-78 and SHHS) and compared it with baseline methods in downstream tasks. Our method exhibited state-of-the-art performance, achieving accuracies of 78.64% and 81.45% with SleepEDF-78 and SHHS, respectively, and macro F1-scores of 70.39% with SleepEDF-78 and 70.47% with SHHS.

Existing scene text recognition methods leverage large-scale labeled synthetic data (LSD) to reduce reliance on labor-intensive annotation tasks and improve recognition capability in real-world scenarios. However, the emergence of a synth-to-real domain gap still limits their efficiency and robustness. Consequently, harvesting the meaningful intrinsic qualities of unlabeled real data (URD) is of great importance, given the prevalence of text-laden images. Toward the target, recent efforts have focused on pre-training on URD through sequence-to-sequence self-supervised learning, followed by fine-tuning on LSD via supervised learning. Nevertheless, they encounter three important issues: coarse representation learning units, inflexible data augmentation, and an emerging real-to-synth domain drift. To overcome these challenges, we propose CCDPlus, an accurate character-to-character distillation method for scene text recognition with a joint supervised and self-supervised learning framework. Specifically, tailored for text images, CCDPlus delineates the fine-grained character structures on URD as representation units by transferring knowledge learned from LSD online. Without requiring extra bounding box or pixel-level annotations, this process allows CCDPlus to enable character-to-character distillation flexibly with versatile data augmentation, which effectively extracts general real-world character-level feature representations. Meanwhile, the unified framework combines self-supervised learning on URD with supervised learning on LSD, effectively solving the domain inconsistency and enhancing the recognition performance. Extensive experiments demonstrate that CCDPlus outperforms previous state-of-the-art (SOTA) supervised, semi-supervised, and self-supervised methods by an average of 1.8%, 0.6%, and 1.1% on standard datasets, respectively. Additionally, it achieves a 6.1% improvement on the more challenging Union14M-L dataset.

Adaptive self-supervised learning for sequential recommendation

In Self-Supervised Learning (SSL), various pretext tasks are designed for learning feature representations through contrastive loss. However, previous studies have shown that this loss is less tolerant to semantically similar samples due to the inherent defect of instance discrimination objectives, which may harm the quality of learned feature embeddings used in downstream tasks. To improve the discriminative ability of feature embeddings in SSL, we propose a new loss function called Angular Contrastive Loss (ACL), a linear combination of angular margin and contrastive loss. ACL improves contrastive learning by explicitly adding an angular margin between positive and negative augmented pairs in SSL. Experimental results show that using ACL for both supervised and unsupervised learning significantly improves performance. We validated our new loss function using the FSDnoisy18k dataset, where we achieved 73.6% and 77.1% accuracy in sound event classification using supervised and self-supervised learning, respectively.

Unlocking the Potential of Medical Imaging with ChatGPT's Intelligent Diagnostics

Review on self-supervised image recognition using deep neural networks