ConCL: Concept Contrastive Learning for Dense Prediction Pre-training in Pathology Images

Combining big data and deep learning can analyze large-scale breast cancer pathology images for auxiliary diagnosis. Furthermore, Whole Slide Images (WSIs) of breast cancer pathology offer detailed tissue feature information, which supports the accurate identification of malignant lesions. Current approaches combine Self-Supervised Learning (SSL) and Multiple Instance Learning (MIL) for WSI analysis, aiming to address the issues of billion-level pixels in a single WSI and the lack of precise annotations. However, pseudo-labels produced by SSL frequently lack accuracy, and MIL fails to effectively integrate global information at the WSI level, resulting in performance bottlenecks. This paper proposes the Multi-granularity Self-supervised Second-order MIL (MS2M) to tackle these issues. MS2M first achieves instance-level fine-grained feature learning through multi-granularity SSL and optimizes instance-level representations using bag-level labels within the MIL framework. Then, the transformer captures long-range dependencies between instances. When combined with second-order (covariance) pooling, it also captures high-order relational information. This process generates a robust bag-level representation. MS2M achieves accuracies of 0.9845 and 0.9719 on the CAMELYON16 and private breast cancer WSI datasets, respectively, outperforming existing methods.

Deep learning has become a valuable tool for the automation of certain medical image segmentation tasks, significantly relieving the workload of medical specialists. Some of these tasks require segmentation to be performed on a subset of the input dimensions, the most common case being 3D→2D. However, the performance of existing methods is strongly conditioned by the amount of labeled data available, as there is currently no data efficient method, e.g. transfer learning, that has been validated on these tasks. In this work, we propose a novel convolutional neural network (CNN) and self-supervised learning (SSL) method for label-efficient 3D→2D segmentation. The CNN is composed of a 3D encoder and a 2D decoder connected by novel 3D→2D blocks. The SSL method consists of reconstructing image pairs of modalities with different dimensionality. The approach has been validated in two tasks with clinical relevance: the en-face segmentation of geographic atrophy and reticular pseudodrusen in optical coherence tomography. Results on different datasets demonstrate that the proposed CNN significantly improves the state of the art in scenarios with limited labeled data by up to 8% in Dice score. Moreover, the proposed SSL method allows further improvement of this performance by up to 23%, and we show that the SSL is beneficial regardless of the network architecture. Our code is available at https://github.com/j-morano/multimodal-ssl-fpn.

PathoDuet: Foundation models for pathological slide analysis of H&E and IHC stains

Spatiotemporal self-supervised predictive learning for atmospheric variable prediction via multi-group multi-attention

Adaptive self-supervised learning for sequential recommendation

Self-supervised learning provides a possible solution to extract effective visual representations from unlabeled histopathological images. However, existing methods either fail to make good use of domain-specific knowledge, or rely on side information like spatial proximity and magnification. In this paper, we propose CS-CO, a hybrid self-supervised visual representation learning method tailored for histopathological images, which integrates advantages of both generative and discriminative models. The proposed method consists of two self-supervised learning stages: cross-stain prediction (CS) and contrastive learning (CO), both of which are designed based on domain-specific knowledge and do not require side information. A novel data augmentation approach, stain vector perturbation, is specifically proposed to serve contrastive learning. Experimental results on the public dataset NCT-CRC-HE-100K demonstrate the superiority of the proposed method for histopathological image visual representation. Under the common linear evaluation protocol, our method achieves 0.915 eight-class classification accuracy with only 1,000 labeled data, which is about 1.3% higher than the fully-supervised ResNet18 classifier trained with the whole 89,434 labeled training data. Our code is available at https://github.com/easonyang1996/CS-CO.

Considering several patients screened due to COVID-19 pandemic, computer-aided detection has strong potential in assisting clinical workflow efficiency and reducing the incidence of infections among radiologists and healthcare providers. Since many confirmed COVID-19 cases present radiological findings of pneumonia, radiologic examinations can be useful for fast detection. Therefore, chest radiography can be used to fast screen COVID-19 during the patient triage, thereby determining the priority of patient's care to help saturated medical facilities in a pandemic situation. In this paper, we propose a new learning scheme called self-supervised transfer learning for detecting COVID-19 from chest X-ray (CXR) images. We compared six self-supervised learning (SSL) methods (Cross, BYOL, SimSiam, SimCLR, PIRL-jigsaw, and PIRL-rotation) with the proposed method. Additionally, we compared six pretrained DCNNs (ResNet18, ResNet50, ResNet101, CheXNet, DenseNet201, and InceptionV3) with the proposed method. We provide quantitative evaluation on the largest open COVID-19 CXR dataset and qualitative results for visual inspection. Our method achieved a harmonic mean (HM) score of 0.985, AUC of 0.999, and four-class accuracy of 0.953. We also used the visualization technique Grad-CAM++ to generate visual explanations of different classes of CXR images with the proposed method to increase the interpretability. Our method shows that the knowledge learned from natural images using transfer learning is beneficial for SSL of the CXR images and boosts the performance of representation learning for COVID-19 detection. Our method promises to reduce the incidence of infections among radiologists and healthcare providers.

Deep models trained in supervised mode have achieved remarkable success on a variety of tasks. When labeled samples are limited, self-supervised learning (SSL) is emerging as a new paradigm for making use of large amounts of unlabeled samples. SSL has achieved promising performance on natural language and image learning tasks. Recently, there is a trend to extend such success to graph data using graph neural networks (GNNs). In this survey, we provide a unified review of different ways of training GNNs using SSL. Specifically, we categorize SSL methods into contrastive and predictive models. In either category, we provide a unified framework for methods as well as how these methods differ in each component under the framework. Our unified treatment of SSL methods for GNNs sheds light on the similarities and differences of various methods, setting the stage for developing new methods and algorithms. We also summarize different SSL settings and the corresponding datasets used in each setting. To facilitate methodological development and empirical comparison, we develop a standardized testbed for SSL in GNNs, including implementations of common baseline methods, datasets, and evaluation metrics.

The field of machine learning has recently undergone significant advancements, driven by the development of sophisticated algorithms and the availability of large datasets. Among the various paradigms of machine learning, supervised learning has traditionally dominated, relying heavily on labeled data to train models. However, acquiring and annotating large amounts of labeled data is both time-consuming and costly, leading to a growing interest in self-supervised learning (SSL). This promising approach leverages unlabeled data to learn useful representations, which can then be fine-tuned for specific tasks with minimal labeled data Self-supervised learning creates learning signals from the data itself, thus eliminating the need for manual labeling. By solving auxiliary tasks, such as predicting the rotation angle of an image or filling in missing parts of data, SSL methods can learn robust feature representations that are useful for a wide range of downstream tasks. This paradigm reduces dependency on labeled data and exploits the abundance of unlabeled data available in various domains. The significance of SSL lies in its potential to democratize machine learning, making it more accessible and applicable across different fields. From natural language processing to computer vision and beyond, self-supervised techniques have demonstrated remarkable success in improving model performance while reducing reliance on extensive labeled datasets. As such, SSL represents a critical step towards more efficient and scalable machine learning solutions. This paper explores the development and implementation of a novel self-supervised learning solution based on machine learning principles. The proposed method achieves an accuracy of 97.6%, a mean absolute error (MAE) of 0.403, and a root mean square error (RMSE) of 0.203. A comprehensive overview of the underlying methodologies, the challenges faced, and the potential applications of this approach is provided. By examining the theoretical foundations and practical implementations, this study aims to contribute to the growing body of research in SSL and highlight its transformative impact on the machine learning landscape.Self-Supervised Learning.

The implementation of computer vision generally focuses on resolving a problem of one of two types of image data, whether it was natural image datasets or medical image datasets. Self-supervised learning (SSL), as the most recent approach in computer vision, is typically used to solved tasks related only to natural image datasets such as ImageNet, CIFAR-10, and PASCAL. This study focuses on analyzing the involvement of natural images in the SSL implementation when classifying medical images. We used the ResNet50 architecture comparing the results of human embryo image classification using the conventional transfer learning method and SSL methods. We apply three SSL approaches, namely Rotation, Jigsaw Puzzle, and SimCLRv1. The results of our research, with 1226 embryo images divided into three classes, show that the SSL implementations cannot be said to be efficient enough compared to the conventional model produced by ResNet50 with pretrained weight ImageNet. However, when the models were trained using random weight initialization, the Rotation model with an average accuracy of 78.15% and the Jigsaw Puzzle model with an average accuracy of 80.30% are significantly superior to the conventional model with average accuracy is 71.07%. Moreover, related to our embryos dataset, we discuss the differences between the SSL context-context contrastive learning approach and the context instance contrastive learning approach.

The artificial intelligence (AI) community has recently made tremendous progress in developing self-supervised learning (SSL) algorithms that can learn high-quality data representations from massive amounts of unlabeled data. These methods brought great results even to the fields outside of AI. Due to the joint efforts of researchers in various areas, new SSL methods come out daily. However, such a sheer number of publications make it difficult for beginners to see clearly how the subject progresses. This survey bridges this gap by carefully selecting a small portion of papers that we believe are milestones or essential work. We see these researches as the “dots” of SSL and connect them through how they evolve. Hopefully, by viewing the connections of these dots, readers will have a high-level picture of the development of SSL across multiple disciplines including natural language processing, computer vision, graph learning, audio processing, and protein learning.

1577 Background: Only a subset of non-small cell lung cancer (NSCLC) patients experiences durable benefit from immune checkpoint inhibitors (ICIs), and precise biomarkers remain scarce. Meanwhile, computational pathology, based on digital pathology, has led to significant advancements in NSCLC prognosis prediction. Yet limited generalization and interpretation remain critical challenges in current clinical practice. Self-supervised learning, pretrained on unlabeled data to comprehensively capture underlying biological information in pathological images, may enable expandable, interpretable pathomics models for outcome prediction of ICIs. Methods: H&E-stained slides from pan-cancer patients were digitized as whole slide images (WSIs), then segmented for preprocessing. A self-supervised foundation model (patho-GPT) were performed to extract WSI features, and further fine-tuned for outcome prediction of immunotherapy in NSCLC with progression-free survival (PFS) labels. Its performance was evaluated by accuracy, sensitivity, specificity, and AUC in internal and external validation. Patients were classified as immunotherapy-resistant (R) or immunotherapy-sensitive (S), and ROC/survival curves were generated. The model was also tested in an operable cohort on neoadjuvant immunotherapy. Finally, single-cell RNA (scRNA) sequencing analyses were performed to provide biological insights. Results: A total of 13770 whole slide images (WSIs) from 6589 patients were included to construct the self-supervised patho-GPT model, which utilizes a context encoder, target encoder, and predictor based on the Vision Transformer architecture. There were 771 WSIs from 511 NSCLC patients receiving immunotherapy, labeled using 5.23 months as the cut-off value. All patients were divided into training and validation sets at a 7:3 ratio. For downstream outcome prediction of immunotherapy, the weight of patho-GPT was fine-tuned in the training set, and performance was evaluated in internal and independent external validation sets. The accuracy was 0.828 (AUC 0.774) in the internal set and 0.758 (AUC 0.752) externally. Survival analyses showed the model’s risk group was significantly associated with survival after immunotherapy. In contrast, the ViT-ViT model using initial weights achieved 0.677 accuracy (AUC 0.547) in the external set, which was significantly inferior. ScRNA sequencing and differential analysis between R and S groups were performed, and a high level of H1.2hi Teffs was found in R, linked to dysfunction of cytotoxicity-related genes and immune pathways. Conclusions: The self-supervised patho-GPT can be used for the accurate prediction of immunotherapy outcome, with well generalization and biological interpretation.

Pathological diagnosis assists in saving human lives, but such models are annotation hungry and pathological images are notably expensive to annotate. Contrastive learning could be a promising solution that relies only on the unlabeled training data to generate informative representations. However, the majority of current methods in contrastive learning have the following two issues: (1) positive samples produced through random augmentation are less challenging, and (2) false negative pairs problem caused by negative sampling bias. To alleviate the above issues, we propose a novel contrastive learning method called Cluster-Aware Adversarial Contrastive Learning (CA2CL). Specifically, a mixed data augmentation technique is provided to learn more transferable representations by generating more discriminative sample pairs. Furthermore, to mitigate the effects of inherent false negative pairs, we adopt a cluster-aware loss to identify similarities between instances and incorporate them into the process of contrastive learning. Finally, we generate challenging contrastive data pairs by adversarial learning, and adversarially learn robust representations in the representation space without the labeled training data, which aims to maximize the similarity between the augmented sample and the related adversarial sample. Our proposed CA2CL is evaluated on two public datasets: NCT-CRC-HE and PCam for the fine-tuning and linear evaluation tasks and on two other public datasets: GlaS and CARG for the detection and segmentation tasks, respectively. Extensive experimental results demonstrate the superior performance improvement of our method over several Self-supervised learning (SSL) methods and ImageNet pretraining particularly in scenarios with limited data availability for all four tasks.

This paper proposes a novel self-supervised learning framework that uses model reinforcement, REference-free LAtent map eXtraction with MOdel REinforcement (RELAX-MORE), for accelerated quantitative MRI (qMRI) reconstruction. The proposed method uses an optimization algorithm to unroll an iterative model-based qMRI reconstruction into a deep learning framework, enabling accelerated MR parameter maps that are highly accurate and robust. Unlike conventional deep learning methods which require large amounts of training data, RELAX-MORE is a subject-specific method that can be trained on single-subject data through self-supervised learning, making it accessible and practically applicable to many qMRI studies. Using quantitative mapping as an example, the proposed method was applied to the brain, knee and phantom data. The proposed method generates high-quality MR parameter maps that correct for image artifacts, removes noise, and recovers image features in regions of imperfect image conditions. Compared with other state-of-the-art conventional and deep learning methods, RELAX-MORE significantly improves efficiency, accuracy, robustness, and generalizability for rapid MR parameter mapping. This work demonstrates the feasibility of a new self-supervised learning method for rapid MR parameter mapping, that is readily adaptable to the clinical translation of qMRI.

A limited amount of training data is a well-known challenge in the application of deep learning methods. This issue is particularly relevant in biomedical signal processing, such as the analysis of electrocardiograms, due to the labor-intensive nature of data preparation, which requires the involvement of qualified specialists. Self-supervised learning methods, originally developed in such domains as natural language processing and computer vision, have emerged as a potential approach to addressing this challenge and are increasingly being explored in biomedical signal processing. However, direct adaptation of self-supervised learning techniques from other domains does not fully account for ECG-specific characteristics, such as quasi-periodicity, localized morphological features, and susceptibility to noise. This highlights the relevance of developing ECG-specific self-supervised learning methods. This study presents a novel self-supervised learning approach for pretraining neural networks on unlabeled ECG data. The proposed method is based on predicting the short consecutive signal segment using a preceding one and a learned representation vector. The representation extraction and prediction models are trained jointly on the MIMIC-ECG-IV dataset using backpropagation to minimize the mean squared error between the predicted and original signal segments. As an example of a downstream task, a linear binary classifier was trained on the PTB-XL dataset to diagnose pathological conditions using Lead I. The number of training examples for each diagnosis was limited to thirty-four samples. Firstly, the representation model was pre-trained on the unlabeled MIMIC-ECG-IV dataset, and then linear classifiers were trained on the learned representations for each selected diagnosis in PTB-XL. A comparison was also conducted with a randomly initialized representation model trained jointly with the classifier in a fully supervised manner. The proposed method was evaluated against adaptations of Contrastive Learning, Contrastive Predictive Coding, and Masked Autoencoders method. To ensure a controlled experimental setup, implementations of all considered methods were developed using a unified codebase and shared architectural components. Experimental results demonstrated a significant advantage of all self-supervised learning approaches over joint training of feature extraction and classification models. The proposed SSL method outperformed other tested approaches, particularly for diagnoses with subtle short-term morphological features, such as atrial fibrillation and flutter. These findings suggest the potential for further research in developing ECG-specific self-supervised learning methods as a promising approach to improving neural network performance in scenarios with limited labeled data.