Transformer-Based Spatial Domain Recognition with Cross-Modal Integration

For over 100 years, the traditional tools of pathology, such as tissue-marking dyes (e.g. the H&E stain) have been used to study the disorganization and dysfunction of cells within tissues. This has represented a principal diagnostic and prognostic tool in cancer. However, in the last 5 years, new technologies have promised to revolutionize histopathology, with Spatial Transcriptomics technologies allowing us to measure gene expression directly in pathology-stained tissue sections. In parallel with these developments, Artificial Intelligence (AI) applied to histopathology tissue images now approaches pathologist level performance in cell type identification. However, these new technologies still have severe limitations, with Spatial Transcriptomics suffering difficulties distinguishing transcriptionally similar cell types, and AI-based pathology tools often performing poorly on real world out-of-batch test datasets. Thus, century-old techniques still represent standard-of-care in most areas of clinical cancer diagnostics and prognostics. Here, we present a new frontier in digital pathology: describing a conceptually novel computational methodology, based on Bayesian probabilistic modelling, that allows Spatial Transcriptomics data to be leveraged together with the output of deep learning-based AI used to computationally annotate H&E-stained sections of the same tumor. By leveraging cell-type annotations from multiple independent pathologists, we show that this integrated methodology achieves better performance than any given pathologist’s manual tissue annotation in the task of identifying regions of immune cell infiltration in breast cancer, and easily outperforms either technology alone. We also show that on a subset of histopathology slides examined, the methodology can identify regions of clinically relevant immune cell infiltration that were missed entirely by an initial pathologist’s manual annotation. While this use case has clear diagnostic and prognostic value in cancer (e.g. predicting response to immunotherapy), our methodology is generalizable to any type of pathology images and also has broad applications in spatial transcriptomics data analytics, where most applications (such as identifying cell-cell interactions) rely on correct cell type annotations having been established a priori. We anticipate that this work will spur many follow-up studies, including new computational innovations building on the approach. The work sets the stage for better-than-pathologist performance in other cell-type annotation tasks, with relevant applications in diagnostics and prognostics across almost all cancers. Citation Format: Asif Zubair, Rich Chapple, Sivaraman Natarajan, William C. Wright, Min Pan, Hyeong-Min Lee, Heather Tillman, John Easton, Paul Geeleher. Jointly leveraging spatial transcriptomics and deep learning models for image annotation achieves better-than-pathologist performance in cell type identification in tumors [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2022; 2022 Apr 8-13. Philadelphia (PA): AACR; Cancer Res 2022;82(12_Suppl):Abstract nr 456.

In recent years, spatial transcriptomics (ST) technologies have developed rapidly and have been widely used in constructing spatial tissue atlases and characterizing spatiotemporal heterogeneity of cancers. Currently, ST has been used to profile spatial heterogeneity in multiple cancer types. Besides, ST is a benefit for identifying and comprehensively understanding special spatial areas such as tumor interface and tertiary lymphoid structures (TLSs), which exhibit unique tumor microenvironments (TMEs). Therefore, ST has also shown great potential to improve pathological diagnosis and identify novel prognostic factors in cancer. This review presents recent advances and prospects of applications on cancer research based on ST technologies as well as the challenges.

Spatial transcriptomics is an emerging technology that allows for analysis of cellular and molecular heterogeneity at spatial resolution. The accurate identification of pathological regions in spatial transcriptomics data is essential for understanding tissue heterogeneity and disease progression. We introduce UPSST, a comprehensive framework that integrates tissue morphology, imputes gene expression, and clusters spatial regions using a graph attention neural network (GAT). UPSST was evaluated across multiple spatial transcriptomics datasets, achieving high performance, such as achieving an Adjusted Rand Index (ARI) of 0.737 and a Fowlkes-Mallows Index (FMI) of 0.818 on slice 151671 of the LIBD human dorsolateral prefrontal cortex (DLPFC) dataset. These results highlight the robustness and precision of our approach in identifying pathology domains. Additionally, UPSST facilitates downstream analyses such as differential and enrichment analysis, which are crucial for deriving biological insights. In conclusion, UPSST offers a powerful and reliable tool for spatial transcriptomics analysis, advancing the identification of pathological regions with high accuracy.

Deciphering the principles and mechanisms by which gene activity orchestrates complex cellular arrangements in multicellular organisms has far-reaching implications for research in the life sciences. Recent technological advances in next-generation sequencing- and imaging-based approaches have established the power of spatial transcriptomics to measure expression levels of all or most genes systematically throughout tissue space, and have been adopted to generate biological insights in neuroscience, development andplant biology as well asto investigate a range of disease contexts, including cancer. Similar to datasets made possible by genomic sequencing and population health surveys, the large-scale atlases generated by this technology lend themselves to exploratory data analysis for hypothesis generation. Here we review spatial transcriptomic technologies and describe the repertoire of operations available for paths of analysis of the resulting data. Spatial transcriptomics can also be deployed for hypothesis testing using experimental designs that compare time points or conditions-including genetic or environmental perturbations. Finally, spatial transcriptomic data are naturally amenable to integration with other data modalities, providing an expandable framework for insight into tissue organization.

High-throughput spatial transcriptomics (ST) now profiles hundreds of thousands of cells or locations per section, creating computational bottlenecks for routine analysis. Sketching, or intelligent sub-sampling, addresses scale by selecting small, representative subsets. While effective for scRNA-seq data, existing sketching methods, which optimize coverage in expression space but ignore physical location, can introduce spatial bias when applied to ST data. To explore the impact of sketching on ST analysis, we systematically benchmarked uniform sampling, leverage-score sampling, Geosketch (minimax/Hausdorff), and scSampler (maximin) across multiple real ST datasets (mouse ovary, MERFISH brain, human breast cancer, lung) and simulations, using three input representations: PCA embeddings, spatial coordinates, and spatially smoothed embeddings. We show that expression-only designs capture global transcriptomic heterogeneity but distort tissue architecture by over-sampling high-variability regions and under-sampling homogeneous areas. Coordinate-only sampling restores tissue coverage but misses transcriptional extremes. A simple spatially aware extension, computing leverage scores from a randomized SVD basis smoothed by a spatial weights matrix, strikes a favorable balance, recovering rare cell states while maintaining uniform tissue coverage and avoiding edge effects. Across robust Hausdorff distances, clustering stability (ARI), PCA loading drift, and local cell-type MSE, spatially smoothed leverage scores match or outperform alternatives. These results motivate joint spatial–transcriptomic sketching objectives to enable fast, unbiased analyses of increasingly large ST datasets.

Recent advances in spatial transcriptomics technologies have led to a growing number of diverse datasets, offering unprecedented opportunities to explore tissue organizations and functions within spatial contexts. However, it remains a significant challenge to effectively integrate and interpret these data, often originating from different samples, technologies, and developmental stages. In this paper, we present INSPIRE, a deep learning method for integrative analyses of multiple spatial transcriptomics datasets to address this challenge. With designs of graph neural networks and an adversarial learning mechanism, INSPIRE enables spatially informed and adaptable integration of data from varying sources. By incorporating non-negative matrix factorization, INSPIRE uncovers interpretable spatial factors with corresponding gene programs, revealing tissue architectures, cell type distributions and biological processes. We demonstrate the capabilities of INSPIRE by applying it to human cortex slices from different samples, mouse brain slices with complementary views, mouse hippocampus and embryo slices generated through different technologies, and spatiotemporal organogenesis atlases containing half a million spatial spots. INSPIRE shows superior performance in identifying detailed biological signals, effectively borrowing information across distinct profiling technologies, and elucidating dynamical changes during embryonic development. Furthermore, we utilize INSPIRE to build 3D models of tissues and whole organisms from multiple slices, demonstrating its power and versatility.

Spatial transcriptomics is an emerging technology widely applied to the analyses of tissue architecture and corresponding biological functions. Substantial computational methods have been developed for analyzing spatial transcriptomics data. These methods generate embeddings from gene expression and spatial locations for spot clustering or tissue architecture segmentation. Although the hyperparameters used to produce an embedding can be tuned for a given training set, a fixed embedding has variable performance from case to case due to data distributions. Therefore, selecting an effective embedding for new data in advance would be useful. For this purpose, we developed an embedding evaluation method named message passing-Moran’s I with maximum filtering (MP-MIM), which combines message passing-based embedding transformation with spatial autocorrelation analysis. We applied a graph convolution to aggregate spatial transcriptomics data and employed global Moran’s I to measure spatial autocorrelation and select the most effective embedding to infer tissue architecture. Sixteen spatial transcriptomics samples generated from the human brain were used to validate our method. The results show that MP-MIM can accurately identify high-quality embeddings that produce a high correlation between the predicted tissue architecture and the ground truth. Overall, our study provides a novel method to select embeddings for new test data and enhance the usability of deep learning tools for spatial transcriptome analyses.

Next-generation sequencing (NGS) of bulk cell populations is a useful and ubiquitous tool for the molecular characterization of clinical tumor samples. Bulk NGS reveals transcript abundance within a tumor sample and can further infer cell populations via deconvolution algorithms (PMID:31570899). However, it can’t ascribe the cellular context for a given gene’s expression or elucidate the spatial organization of tumor microenvironments. These additional features are critical to our understanding of tumor biology and are key to the development of immuno-oncology therapeutics. Spatial Transcriptomics (ST) is an emerging technology that characterizes gene expression within the spatial context of tissue. ST data can be generated directly from archival formalin fixed paraffin embedded samples, enabling the study of spatial gene expression in real-world clinical settings. We have studied a dataset comprising 6 samples from non-small cell lung cancer (NSCLC) patients and 1 core needle biopsy from a tumor of unknown origin. We used the 10X Visium CytAssist platform to generate ST data and additionally generated paired bulk RNAseq data. To test the interassay reliability of CytAssist on archival FFPE tissue sections, we compared ST results across 3 sample preparation conditions. We further studied the state of the tumor microenvironment by applying state-of-the-art computational approaches to deconvolve immune cell populations and produce super-resolution ST maps, validated using multiplex immunofluorescence (IF) via CODEX (PMID:30078711). We find key quality control metrics and spatial biomarkers are consistent across all 3 sample preparation conditions. When comparing deconvolution results between bulk and spatially-resolved methods we observe modest correlations for many cell types despite differences in sample preparation, supporting the idea that bulk and spatial samples contain complementary transcriptomic information. However, within samples, we find many of the correlations observed in bulk do not show strong spatial correlation. These comparisons indicate the importance of considering spatial context when studying the tumor immune microenvironment. Finally, we find an agreement between super-resolution ST and multiplex IF across key spatial biomarkers. These results demonstrate clinical archival FFPE samples yield high interassay reliability via the CytAssist platform. Results were consistent through 3 different H&E staining protocols and findings were consistent when superresolution deconvolution was utilized which further strongly correlated with high-resolution multiplex IF. Our findings demonstrate the feasibility and translational utility of ST to discover spatial signatures and the cellular context in retrospective clinical cohorts to empower discovery and translational efforts in precision oncology and therapeutic development. Citation Format: Mario G. Rosasco, Chi-Sing Ho, Tianyou Luo, Michelle M. Stein, Luca Lonini, Martin C. Stumpe, Jagadish Venkataraman, Sonal Khare, Ameen A. Salahudeen. Comparison of interassay similarity and cellular deconvolution in spatial transcriptomics data using Visum CytAssist. [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2023; Part 1 (Regular and Invited Abstracts); 2023 Apr 14-19; Orlando, FL. Philadelphia (PA): AACR; Cancer Res 2023;83(7_Suppl):Abstract nr 4692.

My talk will present three computational frameworks we developed to study cytokine signaling activities and cell-cell communications during the antitumor immune response, using tumor single-cell and spatial transcriptomics. The basic immunology tool to study cytokine signaling mostly measures cytokine release, which is transient and does not represent downstream target activities. Therefore, we first developed the CytoSig platform, providing a database of target genes modulated by cytokines and a predictive model of cytokine signaling cascades from transcriptomic profiles. We collected 20,591 transcriptome profiles for human cytokine, chemokine, and growth factor responses. This atlas of transcriptional patterns induced by cytokines enabled the reliable prediction of signaling activities in distinct cell populations in infectious diseases, chronic inflammation, and cancer using bulk and single-cell transcriptomic data. CytoSig revealed previously unidentified roles of many cytokines, such as BMP6 as an anti-inflammatory factor. Then, based on CytoSig, we developed Tres, a computational model utilizing single-cell transcriptomic data to identify signatures of T cells that are resilient to immunosuppressive signals, such as TGF-β1, TRAIL, and prostaglandin E2. Tres reliably predicts clinical responses to immunotherapy in multiple cancer types using bulk T cell transcriptomic data from pre-treatment patient tumors or infusion/pre-manufacture samples for cellular immunotherapies. Further, Tres identified FIBP as a candidate immunotherapy target to potentiate adoptive cell therapy efficacies. FIBP knockout in T cells enhanced adoptive cell therapy by down-regulating T cells' cholesterol metabolism. Last, I will briefly show our SpaCET framework for deconvolving cell fractions and identifying cell-cell interactions in tumor spatial transcriptomics data. SpaCET resolved several challenges in spatial transcriptomics analysis that previous methods did not address sufficiently. Through coupling cell fractions with ligand-receptor co-expression analysis, SpaCET reveals that intercellular interactions tend to be located at the tumor-immune boundaries. Citation Format: Peng Jiang. Inference of intercellular signaling activities in tumor spatial and single-cell transcriptomics, with applications in identifying cancer immunotherapy targets [abstract]. In: Proceedings of the AACR-NCI-EORTC Virtual International Conference on Molecular Targets and Cancer Therapeutics; 2023 Oct 11-15; Boston, MA. Philadelphia (PA): AACR; Mol Cancer Ther 2023;22(12 Suppl):Abstract nr IA002.

Spatial transcriptomics is a powerful technology for high-resolution mapping of gene expression in tissue samples, enabling a molecular level understanding of tissue architecture. The acquisition entails dissecting and profiling micron-thick tissue slices, with multiple slices often needed for a comprehensive study. However, the lack of a common coordinate framework (CCF) among slices, due to slicing and displacement variations, can hinder data analysis, making data comparison and integration challenging, and potentially compromising analysis accuracy. Here we present a deep learning algorithm STaCker that unifies the coordinates of transcriptomic slices via an image registration process. STaCker derives a composite image representation by integrating tissue image and gene expression that are transformed to be resilient to noise and batch effects. STaCker overcomes the training data scarcity by training exclusively on diverse synthetic data. Its performance on various benchmarking datasets shows a significant increase in spatial concordance in aligned slices, surpassing existing methods. STaCker also successfully harmonizes multiple real spatial transcriptome datasets acquired from various platforms. These results indicate that STaCker is a valuable computational tool for constructing a CCF with spatial transcriptome data.

In recent years, the application of single-cell transcriptomics and spatial transcriptomics analysis techniques has become increasingly widespread. Whether dealing with single-cell transcriptomic or spatial transcriptomic data, dimensionality reduction and clustering are indispensable. Both single-cell and spatial transcriptomic data are often high-dimensional, making the analysis and visualization of such data challenging. Through dimensionality reduction, it becomes possible to visualize the data in a lower-dimensional space, allowing for the observation of relationships and differences between cell subpopulations. Clustering enables the grouping of similar cells into the same cluster, aiding in the identification of distinct cell subpopulations and revealing cellular diversity, providing guidance for downstream analyses. In this review, we systematically summarized the most widely recognized algorithms employed for the dimensionality reduction and clustering analysis of single-cell transcriptomic and spatial transcriptomic data. This endeavor provides valuable insights and ideas that can contribute to the development of novel tools in this rapidly evolving field.

Since spatial transcriptomics can locate and distinguish the gene expression of functional genes in special regions and tissue, it is important for us to investigate the brain development, the development mechanism of brain diseases, and the relationship between brain structure and function in Neuroscience (or Brain science). While previous studies have introduced the crucial spatial transcriptomic techniques and data analysis methods, there are few studies to comprehensively overview the key methods, data resources, and technological applications of spatial transcriptomics in Neuroscience. For these reasons, we first investigate several common spatial transcriptomic data analysis approaches and data resources. Second, we introduce the applications of the spatial transcriptomic data analysis approaches in Neuroscience. Third, we summarize the integrating spatial transcriptomics with other technologies in Neuroscience. Finally, we discuss the challenges and future research directions of spatial transcriptomics in Neuroscience.

Recent advancements in spatial transcriptomics technology have enabled the capture of gene expression profiles while maintaining spatial information. Accurately identifying spatial clustering plays a pivotal role in analyzing spatial transcriptomics data and understanding tissue microenvironments. However, current spatial domain identification methods cannot explore the complex relationship of gene expression profiles and spatial topology. To alleviate this issue, we propose STMCCL, a novel self-supervised learning framework that jointly trains a masked autoencoder and cluster-guided contrastive learning. This framework extracts informative latent representations from gene expression profiles and spatial information. Specifically, we first use data augmentation strategies to build augmented views and employ a masked encoder to generate a feature view. Then, encoders are applied to learn view-unique embeddings of each view. Furthermore, we introduce a multiple cluster-perspectives module that considers both geometric and structural relationships between clusters to produce more reliable cluster assignments. Finally, to derive more discriminative positives and negatives, the cluster-guided contrastive module calculates the confidence of each sample based on the initial cluster. Comprehensive experiments on 7 public datasets demonstrate that STMCCL outperforms the state-of-the-art baselines with finer-scale spatial domain identification.

Recent advances in spatial transcriptomics allow spatially resolved gene expression measurements with cellular or even sub-cellular resolution, directly characterizing the complex spatiotemporal gene expression landscape and cell-to-cell interactions in their native microenvironments. Due to technology limitations, most spatial transcriptomic technologies still yield incomplete expression measurements with excessive missing values. Therefore, gene imputation is critical to filling in missing data, enhancing resolution, and improving overall interpretability. However, existing methods either require additional matched single-cell RNA-seq data, which is rarely available, or ignore spatial proximity or expression similarity information. To address these issues, we introduce Impeller, a path-based heterogeneous graph learning method for spatial transcriptomic data imputation. Impeller has two unique characteristics distinct from existing approaches. First, it builds a heterogeneous graph with two types of edges representing spatial proximity and expression similarity. Therefore, Impeller can simultaneously model smooth gene expression changes across spatial dimensions and capture similar gene expression signatures of faraway cells from the same type. Moreover, Impeller incorporates both short- and long-range cell-to-cell interactions (e.g. via paracrine and endocrine) by stacking multiple GNN layers. We use a learnable path operator in Impeller to avoid the over-smoothing issue of the traditional Laplacian matrices. Extensive experiments on diverse datasets from three popular platforms and two species demonstrate the superiority of Impeller over various state-of-the-art imputation methods. The code and preprocessed data used in this study are available at https://github.com/aicb-ZhangLabs/Impeller and https://zenodo.org/records/11212604.

Spatial transcriptomics is a rapidly emerging field that allows for the study of gene expression within a tissue sample. This technology has the potential to provide valuable insights into the molecular mechanisms underlying cancer progression and could be particularly useful in identifying specific regional cancer progression. By studying the regional spatial trajectory of cancer progression, we can develop more targeted therapies and better predict the likelihood of metastasis. Additionally, analyzing gene expression patterns in specific regions of the tissue can reveal the molecular mechanisms driving cancer growth in those areas, potentially uncovering new therapeutic targets. In this study, we created a spatial trajectory analysis method, named as stLearn - PSeudo-Time-Space (PSTS) algorithm, to investigate specific regional cancer progression. Spatial trajectory analysis is a statistical method that allows for the identification of patterns of transitional gene expression over time and space within a tissue sample. In detail, this method incorporates spatial features, such as tissue sub-structure locations and gene expression changes, to model the trajectory from a single snapshot of transcriptomics data. Also, we evaluated the effectiveness of PSTS by utilizing the cancer progression model in human breast cancer samples. Using this approach, we were able to identify specific regions within the breast tissue that displayed distinct progression branching trajectories patterns of transitional gene expression associated with ductal carcinoma in situ and invasive ductal carcinoma. These regions were found to be characterized by a high level of cellular diversity and were associated with aggressive forms of breast cancer. The results of this study suggest that spatial transcriptomics and spatial trajectory analysis could be a powerful tool for identifying specific regional cancer progression and may help to inform the biomarkers for the development of targeted therapies for breast cancer. Overall, this study highlights the potential of spatial transcriptomics and spatial trajectory analysis to provide valuable insights into the molecular mechanisms underlying cancer progression and could have significant implications for the diagnosis and treatment of cancer. Citation Format: Duy Pham, Quan Nguyen. A novel spatial trajectory inference method for detecting regional breast cancer progression from spatial transcriptomics data [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2023; Part 2 (Clinical Trials and Late-Breaking Research); 2023 Apr 14-19; Orlando, FL. Philadelphia (PA): AACR; Cancer Res 2023;83(8_Suppl):Abstract nr LB076.