Reusability report: Exploring the transferability of self-supervised learning models from single-cell to spatial transcriptomics

The availability of huge annotated datasets and deep learning have led to astounding Natural Language Processing (NLP) progress. However, there is a plethora of low-resource languages, lack the labelled corpora to train the models in the supervision. This kind of digital divide does not allow fair NLP development across language groups. Self-supervised learning (SSL) is a new paradigm in recent years that leverages large amounts of untagged text to learn powerful representations with little or no manual annotation. This paper is a review of self-supervised deep learning models in low-resource NLP tasks. It begins with the definition of the principles of the SSL and the difference between this approach and supervised and unsupervised approaches. We describe why methods such as masked language modeling, contrastive learning, and autoregressive modeling underlie modern pre-trained transformers such as BERT, GPT, and mBERT. A particular interest is paid to multilingual and cross lingual SSL schemes which allow knowledge transfer between high-resource and low resource languages. Low-resource tasks, including machine translation, sentiment analysis, speech-to-text, and information retrieval are reviewed. As per benchmark studies, even in small labelled samples, the SSL models can achieve big gains in accuracy and generalization. Computation cost, representational bias and morphologically rich and under-documented language evaluation, are however, problematic. The researchers observe that the study suffers certain limitations, such as over-reliance on high-resource pretraining information, inequity between linguistic groups, and the difficulty of deploying large-scale SSL models in resource-constrained circumstances. The future directions include lightweight multilingual models, federation of learning in NLP and symbolic linguistic knowledge and combination with SSL. Self-supervised deep learning bridges an essential gap between the high-resource and low-resource languages, and would be a highly valuable step toward inclusive, global NLP innovation.

Training deep learning models on medical images heavily depends on experts’ expensive and laborious manual labels. In addition, these images, labels, and even models themselves are not widely publicly accessible and suffer from various kinds of bias and imbalances. In this paper, chest X-ray pre-trained model via self-supervised contrastive learning (CheSS) was proposed to learn models with various representations in chest radiographs (CXRs). Our contribution is a publicly accessible pretrained model trained with a 4.8-M CXR dataset using self-supervised learning with a contrastive learning and its validation with various kinds of downstream tasks including classification on the 6-class diseases in internal dataset, diseases classification in CheXpert, bone suppression, and nodule generation. When compared to a scratch model, on the 6-class classification test dataset, we achieved 28.5% increase in accuracy. On the CheXpert dataset, we achieved 1.3% increase in mean area under the receiver operating characteristic curve on the full dataset and 11.4% increase only using 1% data in stress test manner. On bone suppression with perceptual loss, we achieved improvement in peak signal to noise ratio from 34.99 to 37.77, structural similarity index measure from 0.976 to 0.977, and root-square-mean error from 4.410 to 3.301 when compared to ImageNet pretrained model. Finally, on nodule generation, we achieved improvement in Fréchet inception distance from 24.06 to 17.07. Our study showed the decent transferability of CheSS weights. CheSS weights can help researchers overcome data imbalance, data shortage, and inaccessibility of medical image datasets. CheSS weight is available at https://github.com/mi2rl/CheSS.

Unsupervised and self-supervised representation learning has become popular in recent years for learning useful features from unlabelled data. Representation learning has been mostly developed in the neural network literature, and other models for representation learning are surprisingly unexplored. In this work, we introduce and analyze several kernel-based representation learning approaches: Firstly, we define two kernel Self-Supervised Learning (SSL) models using contrastive loss functions and secondly, a Kernel Autoencoder (AE) model based on the idea of embedding and reconstructing data. We argue that the classical representer theorems for supervised kernel machines are not always applicable for (self-supervised) representation learning, and present new representer theorems, which show that the representations learned by our kernel models can be expressed in terms of kernel matrices. We further derive generalisation error bounds for representation learning with kernel SSL and AE, and empirically evaluate the performance of these methods in both small data regimes as well as in comparison with neural network based models.

For over 100 years, the traditional tools of pathology, such as tissue-marking dyes (e.g. the H&E stain) have been used to study the disorganization and dysfunction of cells within tissues. This has represented a principal diagnostic and prognostic tool in cancer. However, in the last 5 years, new technologies have promised to revolutionize histopathology, with Spatial Transcriptomics technologies allowing us to measure gene expression directly in pathology-stained tissue sections. In parallel with these developments, Artificial Intelligence (AI) applied to histopathology tissue images now approaches pathologist level performance in cell type identification. However, these new technologies still have severe limitations, with Spatial Transcriptomics suffering difficulties distinguishing transcriptionally similar cell types, and AI-based pathology tools often performing poorly on real world out-of-batch test datasets. Thus, century-old techniques still represent standard-of-care in most areas of clinical cancer diagnostics and prognostics. Here, we present a new frontier in digital pathology: describing a conceptually novel computational methodology, based on Bayesian probabilistic modelling, that allows Spatial Transcriptomics data to be leveraged together with the output of deep learning-based AI used to computationally annotate H&E-stained sections of the same tumor. By leveraging cell-type annotations from multiple independent pathologists, we show that this integrated methodology achieves better performance than any given pathologist’s manual tissue annotation in the task of identifying regions of immune cell infiltration in breast cancer, and easily outperforms either technology alone. We also show that on a subset of histopathology slides examined, the methodology can identify regions of clinically relevant immune cell infiltration that were missed entirely by an initial pathologist’s manual annotation. While this use case has clear diagnostic and prognostic value in cancer (e.g. predicting response to immunotherapy), our methodology is generalizable to any type of pathology images and also has broad applications in spatial transcriptomics data analytics, where most applications (such as identifying cell-cell interactions) rely on correct cell type annotations having been established a priori. We anticipate that this work will spur many follow-up studies, including new computational innovations building on the approach. The work sets the stage for better-than-pathologist performance in other cell-type annotation tasks, with relevant applications in diagnostics and prognostics across almost all cancers. Citation Format: Asif Zubair, Rich Chapple, Sivaraman Natarajan, William C. Wright, Min Pan, Hyeong-Min Lee, Heather Tillman, John Easton, Paul Geeleher. Jointly leveraging spatial transcriptomics and deep learning models for image annotation achieves better-than-pathologist performance in cell type identification in tumors [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2022; 2022 Apr 8-13. Philadelphia (PA): AACR; Cancer Res 2022;82(12_Suppl):Abstract nr 456.

Histology-Based Virtual RNA Inference Identifies Pathways Associated With Metastasis Risk in Colorectal Cancer.

SummaryMedicine is undergoing an unprecedented digital transformation, as massive amounts of health data are being produced, gathered, and curated, ranging from in-hospital (e.g., intensive care unit [ICU]) to person-generated data (wearables). Annotating all these data for training purposes in order to feed to deep learning models for pattern recognition is impractical. Here, we discuss some exciting recent results of self-supervised learning (SSL) applications to high-resolution health signals. These examples leverage unlabeled data to learn meaningful representations that can generalize to situations where the ground truth is inadequate or simply infeasible to collect due to the high burden or associated costs. The most prominent bottleneck of deep learning today is access to labeled, carefully curated datasets, and self-supervision on health signals opens up new possibilities to eliminate data silos through general-purpose models that can transfer to low-resource environments and tasks.

Deep learning has emerged as a powerful tool in various domains, revolutionising machine learning research. However, one persistent challenge is the scarcity of labelled training data, which hampers the performance and generalisation of deep learning models. To address this limitation, researchers have developed innovative methods to overcome data scarcity and enhance deep model learning capabilities. Two prevalent techniques that have gained significant attention are transfer learning and self-supervised learning. Transfer learning leverages knowledge learned from pre-training on a large-scale dataset, such as ImageNet, and applies it to a target task with limited labelled data. This approach allows models to benefit from the learned representations and effectively transfer knowledge to new tasks, resulting in improved learning performance and generalisation. On the other hand, self-supervised learning focuses on training models using pretext tasks that do not require manual annotation, allowing them to learn valuable representations from large amounts of unlabelled data. These learned representations can then be fine-tuned for downstream tasks, mitigating the need for extensive labelled data. In recent years, transfer and self-supervised learning have found applications in various fields, including medical image processing, video recognition, and natural language processing. These approaches have demonstrated remarkable achievements, enabling breakthroughs in areas such as disease diagnosis, object recognition, and language understanding. However, while these methods offer numerous advantages, they also have limitations. For example, transfer learning may face domain mismatch issues between the pre-training and target domains, while self-supervised learning requires careful design of pretext tasks to ensure meaningful representations. This review paper explores the recent applications of these pre-training methods in various fields within the past three years. It delves into the advantages and limitations of each approach, assesses the performance of models employing these techniques, and identifies potential directions for future research. By providing a comprehensive review of current pre-training methods, this article offers guidance for selecting the best technique for specific deep learning applications to address the data scarcity issue.

The application of machine learning (ML) to genomics has transformed the process of analyzing and interpreting large-scale, complex datasets, leading to important breakthroughs in our knowledge of biological systems. This review provides a comprehensive overview of ML applications in key genomic areas: Whole Genome Sequencing (WGS), Whole Exome Sequencing (WES), single-cell genomics, and spatial transcriptomics. In WGS and WES, ML techniques are employed for variant calling, genome-wide association studies, rare variant analysis, and the prediction of pathogenicity. In single-cell genomics, ML facilitates clustering, trajectory inference, and cell type identification, while in spatial transcriptomics, it aids in deciphering spatial patterns of gene expression and tissue heterogeneity. This review further explores the application of ML in related omics fields, including proteomics, transcriptomics, metagenomics, epigenomics, and microbiome research. These applications encompass protein structure prediction, functional annotation, microbial community profiling, and the analysis of epigenetic modifications. We address the challenges caused by high dimensionality, variability in the data, and the requirement for interpretable machine learning models when dealing with genomic data. Emerging technologies like explainable AI and federated learning are highlighted for their potential to address these challenges. Additionally, the review addresses ethical considerations, data privacy issues, and the necessity for standardized protocols in ML applications. This comprehensive examination underscores the transformative impact of ML in genomics and highlights its potential to drive future innovations in personalized medicine and biological research. Received: 17 August 2024 | Revised: 17 October 2024 | Accepted: 31 October 2024 Conflicts of Interest The authors declare that they have no conflicts of interest to this work. Data Availability Statement The data that support this work are available upon reasonable request to the corresponding author. Author Contribution Statement Saheed Adegbola Adeyanju: Conceptualization, Methodology, Validation, Data curation, Writing - original draft, Writing - review & editing, Supervision. Taiwo Temitope Ogunjobi: Methodology, Investigation, Writing - original draft, Writing - review & editing.

Heterogeneity in colorectal cancer (CRC) manifests in the diversity and interactions among tumor cells, support stroma, and immune components, critically impacting patient prognosis and therapeutic efficacy. In this study, spatial transcriptomics (ST, 10x Genomics Visium Spatial Gene Expression) was performed on 40 retrospectively collected CRC samples, along with detailed clinicopathological parameters and long-term clinical outcomes, with the goal of unraveling the molecular topography of the tumor and surrounding stromal compartments, and associating their spatial interactions to clinical outcomes. Five primary RNA groups were identified through unsupervised clustering on ST data, correlating with tissue classes detected in H&E-stained images via a pre-trained deep learning model. Markers and pathways specific to these tissues were identified. Furthermore, immune cell infiltration in these regions was deconvoluted using reference single-cell RNA sequencing (scRNA-seq) data. Sub-clustering within the tumor groups revealed five major tumor subtypes, each characterized by a distinct set of differentially expressed marker genes. Enrichment analysis across these subtypes highlighted distinct activation and suppression in pathways including KRAS, p53, MAPK, and Integrin. Tumors exhibited intra- and inter-tumoral heterogeneity. Location-specific tumor subtypes were observed, with those in right-sided CRC displaying increased immune cell infiltration, linked to enriched pathways such as Type II Interferon Signaling and Cytokine Signaling. Conversely, left-sided CRC subtypes had reduced immune cell presence, and were characterized by enhanced pathways related to vascular development, Hypoxia, and p53 Signaling. Inferred clonal lineages from copy number variation (CNV) using ST suggested a spatially structured phylogenetic organization corresponding to transcriptional clusters. Additionally, the ST data unveiled distinct spatial distribution patterns across various cancer-associated fibroblast (CAF) phenotypes, such as matrix, inflammatory, vessel-associated, tumor-promoting, and antigen-presenting CAFs. Transcriptional patterns uncovered through scRNA-seq, including epithelial-mesenchymal transition (EMT) signatures and consensus molecular subtypes (CMS), displayed unique spatial layouts within the tumor microenvironment. To our knowledge, this is the most comprehensive study to date on spatial transcriptomic heterogeneity within CRC patients, revealing distinct tumor, stroma and immune compositions and spatial structures. This spatially detailed map provides novel insights into CRC heterogeneity, critical for advancing personalized treatment. Citation Format: Jie Zhou, Todd B. Sheridan, Sergii Domanskyi, Stephanie L. Cowles, Sunghee Park, Ilham Putra, Olga Anczukow, Jeffrey H. Chuang, Jill C. Rubinstein. Unveiling the spatial landscape of tumor and stroma heterogeneity in colorectal cancer with spatial transcriptomics [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2024; Part 1 (Regular Abstracts); 2024 Apr 5-10; San Diego, CA. Philadelphia (PA): AACR; Cancer Res 2024;84(6_Suppl):Abstract nr 1150.

Early identification of endometrial cancer or precancerous lesions from histopathological images is crucial for precise endometrial medical care, which however is increasing hampered by the relative scarcity of pathologists. Computer-aided diagnosis (CAD) provides an automated alternative for confirming endometrial diseases with either feature-engineered machine learning or end-to-end deep learning (DL). In particular, advanced self-supervised learning alleviates the dependence of supervised learning on large-scale human-annotated data and can be used to pre-train DL models for specific classification tasks. Thereby, we develop a novel self-supervised triplet contrastive learning (SSTCL) model for classifying endometrial histopathological images. Specifically, this model consists of one online branch and two target branches. The second target branch includes a simple yet powerful augmentation module named random mosaic masking (RMM), which functions as an effective regularization by mapping the features of masked images close to those of intact ones. Moreover, we add a bottleneck Transformer (BoT) model into each branch as a self-attention module to learn the global information by considering both content information and relative distances between features at different locations. On public endometrial dataset, our model achieved four-class classification accuracies of 77.31 ± 0.84, 80.87 ± 0.48 and 83.22 ± 0.87% using 20, 50 and 100% labeled images, respectively. When transferred to the in-house dataset, our model obtained a three-class diagnostic accuracy of 96.81% with 95% confidence interval of 95.61-98.02%. On both datasets, our model outperformed state-of-the-art supervised and self-supervised methods. Our model may help pathologists to automatically diagnose endometrial diseases with high accuracy and efficiency using limited human-annotated histopathological images.

Importance: Precisely decoding brain dysfunction from high-dimensional functional recordings is crucial for advancing our understanding of brain dysfunction in brain disorders. Self-supervised learning (SSL) models offer a transformative approach for mapping dependencies in functional neuroimaging data. Leveraging the intrinsic organization of brain signals for comprehensive feature extraction, these models enable the analysis of critical neurofunctional features within a clinically relevant framework, overcoming challenges related to data heterogeneity and the scarcity of labeled data. Highlight: This paper provides a comprehensive overview of SSL techniques applied to functional neuroimaging data, such as functional magnetic resonance imaging and electroencephalography, with a specific focus on their applications in various neuropsychiatric disorders. We discuss 3 main categories of SSL methods: contrastive learning, generative learning, and generative-contrastive methods, outlining their basic principles and representative methods. Critically, we highlight the potential of SSL in addressing data scarcity, multimodal integration, and dynamic network modeling for disease detection and prediction. We showcase successful applications of these techniques in understanding and classifying conditions such as Alzheimer’s disease, Parkinson’s disease, and epilepsy, demonstrating their potential in downstream neuropsychological applications. Conclusion: SSL models provide a scalable and effective methodology for individual detection and prediction in brain disorders. Despite current limitations in interpretability and data heterogeneity, the potential of SSL for future clinical applications, particularly in the areas of transdiagnostic psychosis subtyping and decoding task-based brain functional recordings, is substantial.

Adapters are lightweight bottleneck modules introduced to assist pre-trained self-supervised learning (SSL) models to be customized to new tasks. However, searching the appropriate layers to insert adapters on large models has become difficult due to the large number of possible layers and thus a vast search space (2 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">N</sup> possibilities for N layers). In this paper, we propose a technique that achieves automatic insertion of adapters for downstream automatic speech recognition (ASR) and spoken language understanding (SLU) tasks. Our approach is based on two-stage training. First, we train our model for a specific downstream task with additional shallow learnable layers and weight parameters to obtain the weighted summation over the output of each layer in SSL. This training method is established by the SUPERB baseline [1]. This first-stage training determines the most important layers given their respective weights. In the second stage, we proceed to insert adapters to the most important layers, retaining both performance and neural architecture search efficiency. On the CommonVoice dataset[2] we obtain 20.6% absolute improvement in Word Error Rate (WER) on the Welsh language against the conventional method, which inserts the adapter modules into the highest layers without search. In the SLURP SLU task, our method yields 4.0% intent accuracy improvement against the same conventional baseline.

CN-Celeb data set is a popular Chinese data set used for speaker recognition. In order to improve the speaker recognition performance on CN-Celeb, this paper mainly makes efforts in following two aspects: applying scenario information and using the self-supervised learning (SSL) model to the speaker recognition system. Compared to the SSL model, scenario information, named domain embedding, is more effective. We integrate speaker embedding and speaker scenario information abstracted as word embedding and puts forward several back-end processing methods. These methods are more suitable for each registered speaker with multiple enroll utterances. Moreover, data augmentation is also investigated. Experiments show that concatenating and averaging each utterance’s domain embedding dimensions of multiple registered speakers with data augmentation shows better performance. Our proposed method effectively reduces 6.0%, 21.7% and 13.5% on EER, minDCF (P = 0.01) and minDCF (P = 0.001), respectively, compared with the baseline system. We did not observe improvement by using SSL as reported in previous works, and the possible reasons and analysis are also given in this paper.

Recent advancements in spatial transcriptomics technology have enabled the capture of gene expression profiles while maintaining spatial information. Accurately identifying spatial clustering plays a pivotal role in analyzing spatial transcriptomics data and understanding tissue microenvironments. However, current spatial domain identification methods cannot explore the complex relationship of gene expression profiles and spatial topology. To alleviate this issue, we propose STMCCL, a novel self-supervised learning framework that jointly trains a masked autoencoder and cluster-guided contrastive learning. This framework extracts informative latent representations from gene expression profiles and spatial information. Specifically, we first use data augmentation strategies to build augmented views and employ a masked encoder to generate a feature view. Then, encoders are applied to learn view-unique embeddings of each view. Furthermore, we introduce a multiple cluster-perspectives module that considers both geometric and structural relationships between clusters to produce more reliable cluster assignments. Finally, to derive more discriminative positives and negatives, the cluster-guided contrastive module calculates the confidence of each sample based on the initial cluster. Comprehensive experiments on 7 public datasets demonstrate that STMCCL outperforms the state-of-the-art baselines with finer-scale spatial domain identification.

Recent advances in spatial transcriptomics (ST) highlight the need to integrate multiple slices for joint analysis. A key challenge is generating interpretable embeddings that preserve spatial geometry while correcting batch effects. We present MaskGraphene, a graph neural network that integrates ST data via masked self-supervised learning, triplet loss, and cluster-wise local alignment. By establishing indirect “soft-links” and direct “hard-links” across slices, MaskGraphene yields joint embeddings with high geometric fidelity. Benchmarks against eight methods demonstrate superior alignment and interpretability. MaskGraphene enhances downstream applications, including domain identification, trajectory reconstruction, biomarker discovery, and brain-layer mapping, enabling robust ST integration and biological insight.Supplementary InformationThe online version contains supplementary material available at 10.1186/s13059-025-03850-w.