MS2M: Multi-granularity Self-supervised Second-order Multiple Instance Learning for Breast Cancer Pathology Image

Multiple instance learning (MIL) has emerged as the best solution for whole slide image (WSI) classification. It consists of dividing each slide into patches, which are treated as a bag of instances labeled with a global label. MIL includes two main approaches: instance-based and embedding-based. In the former, each patch is classified independently, and then, the patch scores are aggregated to predict the bag label. In the latter, bag classification is performed after aggregating patch embeddings. Even if instance-based methods are naturally more interpretable, embedding-based MILs have usually been preferred in the past due to their robustness to poor feature extractors. Recently, the quality of feature embeddings has drastically increased using self-supervised learning (SSL). Nevertheless, many authors continue to endorse the superiority of embedding-based MIL. We conduct 710 experiments across 4 datasets, comparing 10 MIL strategies, 6 self-supervised methods with 4 backbones, 4 foundation models, and various pathology-adapted techniques. Furthermore, we introduce 4 instance-based MIL methods, never used before in the pathology domain. We show that with a good SSL feature extractor, simple instance-based MILs, with very few parameters, obtain similar or better performance than complex, state-of-the-art (SOTA) embedding-based MIL methods, setting new SOTA results on the BRACS and Camelyon16 datasets. As simple instance-based MIL methods are naturally more interpretable and explainable to clinicians, our results suggest that more effort should be put into well-adapted SSL methods for WSI rather than into complex embedding-based MIL methods.

Breast cancer prognostication guides treatment decisions, with transcriptomic assays like Prosigna and Oncotype DX providing valuable risk of recurrence (ROR) scores in ER+/HER2– patients. However, these tests are costly and require sufficient biopsy tissue for accuracy. Pathology images are routinely available and could expedite rapid risk stratification at lower cost. We propose a multi-modal, multi-task deep neural network that learns transcriptomic ROR using digital pathology images and hormone receptor expression status. Because the incorporation of both pathology images and clinical data has proven beneficial in other deep learning models, we hypothesized that incorporation of this complementary information would improve ROR predictions compared to using pathology images alone. We modified the “clustering-constrained-attention multiple-instance learning” (CLAM) deep learning method to perform multi-task regression and infer continuous risk scores (instead of discrete risk categories). We evaluated (1) PAM50 ROR-P, calculated using a multivariate model based on intrinsic subtype centroids and a proliferation score, and (2) a 21-gene assay recurrence score (RS) (research version of Oncotype DX), calculated from component scores relating to proliferation, ER, HER2, and invasion. Using data from 899 patients in the Carolina Breast Cancer Study, a population-based study of diverse patients including 50% Black women and 50% women under age 50, we constructed training (n=714), validation (n=93), and test (n=92) sets. Genomic data were assayed by Nanostring, and digital pathology was based on H&E-stained whole slide images (WSIs). Models were assessed using Pearson correlation between measured and predicted ROR-P or RS. The model with the highest correlations on the test set used H&E WSIs and ER/PR/HER2 expression status as inputs and generated 10 independent outputs: ROR-P, four PAM50 centroid correlations (i.e., Basal, HER2-enriched, Luminal A, and Luminal B), RS, and four RS component scores. Prediction was strong when considering all participants in the test set (ROR-P, 0.78; RS, 0.83, N, 92) but was reduced among more clinically homogeneous ER+/HER2– patients (ROR-P, 0.55; RS, 0.49; N, 49). For RS, correlations were higher in models that included ER/PR/HER2 (Overall, 0.83; ER+/HER2–, 0.50) than in those that relied on pathology alone (Overall, 0.60, ER+/HER2–, 0.34). Future models will consider strategies to enhance performance in ER+/HER2– patients, such as oversampling of ER+/HER2– patients and use of loss functions designed to focus learning on mis-predicted examples. In summary, multi-task, tissue-based regression deep learning models recapitulate transcription-based risk assays with high correlations and, with optimization (especially integration of additional clinicopathologic data), hold promise for personalized treatment decisions from early, routinely-collected biopsy images. Citation Format: Jakub R Kaczmarzyk, Luke A Torre-Healy, Richard A Moffitt, Rajarsi Gupta, Alina M Hamilton, Tahsin M Kurc, Katherine A Hoadley, Melissa A Troester, Joel H Saltz. Early risk stratification of ER+/HER2– breast cancer patients using digital pathology and multi-task, weakly-supervised deep learning [abstract]. In: Proceedings of the AACR Special Conference in Cancer Research: Advances in Breast Cancer Research; 2023 Oct 19-22; San Diego, California. Philadelphia (PA): AACR; Cancer Res 2024;84(3 Suppl_1):Abstract nr B082.

The histopathologic evaluation of regulatory toxicity studies using artificial intelligence (AI) has the potential to increase study efficiency. For example, AI could initially identify and exclude all organs without histopathologic lesions, allowing pathologists to focus solely on evaluating organs with identified lesions. In this study, whole slide images (WSIs) of liver sections from 58 different rat toxicity studies were collected, along with their corresponding histopathologic lesion diagnoses. Each WSI was labeled as either "lesion" or "no lesion" based on the presence or absence of reported histopathologic lesions. Multiple instance learning (MIL) approaches, including a transformer variant, were tested to predict lesions within a weakly supervised framework. Both methods achieved acceptable to excellent area under the receiver operating characteristic curve (AUROC) scores. Heatmap overlays were employed to visually assess the MIL model's effectiveness in detecting lesions, confirming the accuracy of targeted areas on the WSIs. In addition, using transfer learning principles, the MIL model initially developed for liver WSIs was adapted to kidney WSIs, demonstrating the model's versatility. This study showcases the application of weakly supervised learning for lesion detection in rat WSIs from toxicity studies, with the potential to significantly enhance the efficiency of the histopathologic evaluation process.

Whole Slide Image (WSI) classification often relies on weakly supervised Multiple Instance Learning (MIL) methods to handle gigapixel-resolution images. In various MIL methods, attention-based approaches have shown great potential in modern medicine for cancer diagnosis and treatment. These approaches can model the interrelationships among instances to achieve enhanced bag representation using instance scores and thus promote bag-level classification performance. However, the existing attention-based MIL methods face two challenges: (1) The attention-based instance scores cannot accurately represent the contribution of instances to bag-level classification, making it difficult to identify the discriminative regions in WSIs. (2) Whole-slide pathological image analysis frequently suffers from model overfitting and insufficient representation of positive samples for training. To address the problem of poor discriminative regions in WSIs, we design a module to acquire the accurate contribution weights of instances by introducing the Class Activation Map suitable for WSI (WSICAM). For the second challenge, we adopt a Cross-Slide Augmentation (CSA) module to construct new samples with mixed labels on the basis of discriminative instances for model training. Our framework is composed of two WSICAM modules and one CSA module. The experimental results and visualizations demonstrate that our method achieves state-of-the-art in WSI classification on widely used datasets and exhibits robust capabilities in tumor lesion localization.

1648P Deep learning identifies oncogenic genetic alterations in BRAF and NTRK in H&E whole slide images from thyroid carcinomas

Survival outcome prediction is a challenging weakly-supervised and ordinal regression task in computational pathology that involves modeling complex interactions within the tumor microenvironment in gigapixel whole slide images (WSIs). Despite recent progress in formulating WSIs as bags for multiple instance learning (MIL), representation learning of entire WSIs remains an open and challenging problem, especially in overcoming: 1) the computational complexity of feature aggregation in large bags, and 2) the data heterogeneity gap in incorporating biological priors such as genomic measurements. In this work, we present a Multimodal Co-Attention Transformer (MCAT) framework that learns an interpretable, dense co-attention mapping between WSIs and genomic features formulated in an embedding space. Inspired by approaches in Visual Question Answering (VQA) that can attribute how word embed-dings attend to salient objects in an image when answering a question, MCAT learns how histology patches attend to genes when predicting patient survival. In addition to visualizing multimodal interactions, our co-attention trans-formation also reduces the space complexity of WSI bags, which enables the adaptation of Transformer layers as a general encoder backbone in MIL. We apply our proposed method on five different cancer datasets (4,730 WSIs, 67 million patches). Our experimental results demonstrate that the proposed method consistently achieves superior performance compared to the state-of-the-art methods.

When multiple instance learning meets foundation models: Advancing histological whole slide image analysis.

Deep Learning for Detecting BRCA Mutations in High-Grade Ovarian Cancer Based on an Innovative Tumor Segmentation Method From Whole Slide Images

1577 Background: Only a subset of non-small cell lung cancer (NSCLC) patients experiences durable benefit from immune checkpoint inhibitors (ICIs), and precise biomarkers remain scarce. Meanwhile, computational pathology, based on digital pathology, has led to significant advancements in NSCLC prognosis prediction. Yet limited generalization and interpretation remain critical challenges in current clinical practice. Self-supervised learning, pretrained on unlabeled data to comprehensively capture underlying biological information in pathological images, may enable expandable, interpretable pathomics models for outcome prediction of ICIs. Methods: H&E-stained slides from pan-cancer patients were digitized as whole slide images (WSIs), then segmented for preprocessing. A self-supervised foundation model (patho-GPT) were performed to extract WSI features, and further fine-tuned for outcome prediction of immunotherapy in NSCLC with progression-free survival (PFS) labels. Its performance was evaluated by accuracy, sensitivity, specificity, and AUC in internal and external validation. Patients were classified as immunotherapy-resistant (R) or immunotherapy-sensitive (S), and ROC/survival curves were generated. The model was also tested in an operable cohort on neoadjuvant immunotherapy. Finally, single-cell RNA (scRNA) sequencing analyses were performed to provide biological insights. Results: A total of 13770 whole slide images (WSIs) from 6589 patients were included to construct the self-supervised patho-GPT model, which utilizes a context encoder, target encoder, and predictor based on the Vision Transformer architecture. There were 771 WSIs from 511 NSCLC patients receiving immunotherapy, labeled using 5.23 months as the cut-off value. All patients were divided into training and validation sets at a 7:3 ratio. For downstream outcome prediction of immunotherapy, the weight of patho-GPT was fine-tuned in the training set, and performance was evaluated in internal and independent external validation sets. The accuracy was 0.828 (AUC 0.774) in the internal set and 0.758 (AUC 0.752) externally. Survival analyses showed the model’s risk group was significantly associated with survival after immunotherapy. In contrast, the ViT-ViT model using initial weights achieved 0.677 accuracy (AUC 0.547) in the external set, which was significantly inferior. ScRNA sequencing and differential analysis between R and S groups were performed, and a high level of H1.2hi Teffs was found in R, linked to dysfunction of cytotoxicity-related genes and immune pathways. Conclusions: The self-supervised patho-GPT can be used for the accurate prediction of immunotherapy outcome, with well generalization and biological interpretation.

Given the special situation of modeling gigapixel images, multiple instance learning (MIL) has become one of the most important frameworks for Whole Slide Image (WSI) classification. In current practice, most MIL networks often face two unavoidable problems in training: i) insufficient WSI data and ii) the sample memorization inclination inherent in neural networks. These problems may hinder MIL models from adequate and efficient training, suppressing the continuous performance promotion of classification models on WSIs. Inspired by the basic idea of Mixup, this paper proposes a new Pseudo-bag Mixup (PseMix) data augmentation scheme to improve the training of MIL models. This scheme generalizes the Mixup strategy for general images to special WSIs via pseudo-bags so as to be applied in MIL-based WSI classification. Cooperated by pseudo-bags, our PseMix fulfills the critical size alignment and semantic alignment in Mixup strategy. Moreover, it is designed as an efficient and decoupled method, neither involving time-consuming operations nor relying on MIL model predictions. Comparative experiments and ablation studies are specially designed to evaluate the effectiveness and advantages of our PseMix. Experimental results show that PseMix could often assist state-of-the-art MIL networks to refresh their classification performance on WSIs. Besides, it could also boost the generalization performance of MIL models in special test scenarios, and promote their robustness to patch occlusion and label noise. Our source code is available at https://github.com/liupei101/PseMix.

Background: Antibody drug conjugates (ADCs) against HER2 have shown meaningful clinical activity in HER2 low breast cancers, defined as 1+ or 2+ staining on immunohistochemistry (IHC) without gene amplification by in situ hybridization (ISH) techniques. Given that these methods were originally developed for an accurate detection of HER2 3+, their sensitivity and robustness for the detection of low and ultra-low levels of HER2 are questionable. We have recently described a deep learning algorithm that can detect signatures of HER2 expression based on training utilizing scanned H&E whole slide images (WSI) of breast cancers for which IHC and mRNA expression levels of HER2 were available. Here, we report the application of our algorithm to two independent breast cancer cohorts. Methods: A model was developed based on recognition of invasive breast cancer in whole slide images of H&E staining, and then trained via computational neural network with multiple instance learning for binary classification of cases as HER2 “negative” and HER2 “expressed” (low). For training, true negatives were defined as having HER2 IHC-0 and mRNA level < 7.6. HER2-low cases were defined as IHC-1+/2+ and mRNA >9. IHC-0 cases with mRNA >7.6 were excluded from the training cohorts. The resulting model (HER2Complete) was able to distinguish HER2-negatives from HER2-low cases with an AUC of 0.91 (+/- 0.08). Here we use Her2Complete to assess HER2 in two additional cohorts that include 901 ER+/HER2 IHC-0 and 52 HER2 IHC 0+ breast cancers from MSK and TCGA cohorts, respectively. For the TCGA cohort, concomitant transcriptomics data (RNASeq) as a reference for HER2 mRNA expression were retrieved and “HER2 expressed” defined as RNASeq expression of HER2 greater than the 90th percentile of the geometric mean of expression of three reference genes not expressed in breast tissues (TTN, MUC13, OR10A6). Values less than this reference cut-off in the TCGA cohort were considered “HER2 not expressed.” Results: Among the 901 IHC-0 test cases from the MSK cohort, the model identified 82 as ‘negative’, whereas 819 were found to have features of HER2 expression (HER2-Low). Of the 82 negative cases in the MSK cohort, all except 13 cases expressed mRNA levels < 9, and 786/819 of the HER2-low cases expressed mRNA levels >8. Of the 52 IHC 0+ cases in the TCGA cohort, 33 also had “HER2 not expressed” by our reference based RNASeq expression cut-off. Our model identified 15 of these 33 as ‘negative’, while 15 of the 19 TCGA cases with IHC 0+ and HER2 ‘expressed’ by our cut-off were identified as ‘HER2-Low’ by our model. Conclusions: AI tools based on the analysis of WSIs of routinely prepared H&E sections may predict HER2 status in breast cancer. This work requires further investigation using treatment response data to demonstrate that cases with morphologic features of low level HER2 expression will respond to ADCs. Citation Format: Gerard Oakley III, Jorge Reis-Filho, David Klimstra, Marc Goldfinger, Yikan Wang, Antonio Marra. Deep learning-based assessment of HER2-low expression on breast cancer H&E digital whole slide images [abstract]. In: Proceedings of the 2022 San Antonio Breast Cancer Symposium; 2022 Dec 6-10; San Antonio, TX. Philadelphia (PA): AACR; Cancer Res 2023;83(5 Suppl):Abstract nr P5-02-33.

Multiple instance learning (MIL)-based methods have become the mainstream for processing the megapixel-sized whole slide image (WSI) with pyramid structure in the field of digital pathology. The current MIL-based methods usually crop a large number of patches from WSI at the highest magnification, resulting in a lot of redundancy in the input and feature space. Moreover, the spatial relations between patches can not be sufficiently modeled, which may weaken the model's discriminative ability on fine-grained features. To solve the above limitations, we propose a Multi-scale Graph Transformer (MG-Trans) with information bottleneck for whole slide image classification. MG-Trans is composed of three modules: patch anchoring module (PAM), dynamic structure information learning module (SILM), and multi-scale information bottleneck module (MIBM). Specifically, PAM utilizes the class attention map generated from the multi-head self-attention of vision Transformer to identify and sample the informative patches. SILM explicitly introduces the local tissue structure information into the Transformer block to sufficiently model the spatial relations between patches. MIBM effectively fuses the multi-scale patch features by utilizing the principle of information bottleneck to generate a robust and compact bag-level representation. Besides, we also propose a semantic consistency loss to stabilize the training of the whole model. Extensive studies on three subtyping datasets and seven gene mutation detection datasets demonstrate the superiority of MG-Trans.

In computational pathology, extracting and representing spatial features from gigapixel whole slide images (WSIs) are fundamental tasks, but due to their large size, WSIs are typically segmented into smaller tiles. A critical aspect of analyzing WSIs is how information across tiles is aggregated to predict outcomes such as patient prognosis. We introduce a model that combines a message-passing graph neural network (GNN) with a state space model (Mamba) to capture both local and global spatial relationships among the tiles in WSIs. The model’s effectiveness was demonstrated in predicting progression-free survival among patients with early-stage lung adenocarcinomas (LUAD). We compared the model with other state-of-the-art methods for tile-level information aggregation in WSIs, including statistics-based, multiple instance learning (MIL)-based, GNN-based, and GNN-transformer-based aggregation. Our model achieved the highest c-index (0.70) and has the largest number of parameters among comparison models yet maintained a short inference time. Additional experiments showed the impact of different types of node features and different tile sampling strategies on model performance. Code: https://github.com/rina-ding/gat-mamba.

Simple SummaryRecent advancements in AI have revolutionized cancer research, especially in the analysis of histopathological imaging data with minimal human involvement. Early detection of lymph node metastasis in breast cancer is vital for treatment outcomes. This paper introduces a novel approach that combines representation learning and deep learning (DL) to detect small tumors (STs) without neglecting larger ones. The proposed method uses representation learning to identify STs in histopathology images, followed by DL algorithms for breast cancer detection. Extensive evaluation shows remarkable accuracy in detecting STs without compromising larger-lesion detection. This approach enables early detection, timely intervention, and potentially improved treatment outcomes. The integration of representation learning and DL offers a promising solution for ST detection in breast cancer. By reducing human involvement and leveraging AI capabilities, the proposed method achieves impressive accuracy in identifying STs. Further research and validation could enhance diagnostic capabilities and personalized treatment strategies, ultimately benefiting breast cancer patients.The early diagnosis of lymph node metastasis in breast cancer is essential for enhancing treatment outcomes and overall prognosis. Unfortunately, pathologists often fail to identify small or subtle metastatic deposits, leading them to rely on cytokeratin stains for improved detection, although this approach is not without its flaws. To address the need for early detection, multiple-instance learning (MIL) has emerged as the preferred deep learning method for automatic tumor detection on whole slide images (WSIs). However, existing methods often fail to identify some small lesions due to insufficient attention to small regions. Attention-based multiple-instance learning (ABMIL)-based methods can be particularly problematic because they may focus too much on normal regions, leaving insufficient attention for small-tumor lesions. In this paper, we propose a new ABMIL-based model called normal representative keyset ABMIL (NRK-ABMIL), which addresseses this issue by adjusting the attention mechanism to give more attention to lesions. To accomplish this, the NRK-ABMIL creates an optimal keyset of normal patch embeddings called the normal representative keyset (NRK). The NRK roughly represents the underlying distribution of all normal patch embeddings and is used to modify the attention mechanism of the ABMIL. We evaluated NRK-ABMIL on the publicly available Camelyon16 and Camelyon17 datasets and found that it outperformed existing state-of-the-art methods in accurately identifying small tumor lesions that may spread over a few patches. Additionally, the NRK-ABMIL also performed exceptionally well in identifying medium/large tumor lesions.

Image analysis in digital pathology is identified as a challenging field, particularly for AI-driven classification and search tasks. The high-resolution and large-scale nature of whole slide images (WSIs) present significant computational challenges in representing and analyzing these images effectively. The research endeavors to tackle these hurdles by presenting an innovative methodology grounded in self-supervised learning (SSL). Unlike prior SSL approaches that depend on augmenting at the patch level, the novel framework capitalizes on existing primary site information to directly glean effective representations from Whole Slide Images (WSIs). Moreover, the investigation integrates fully supervised contrastive learning to bolster the resilience of these representations for both classification and search endeavors. For experimentation, the study drew upon a dataset encompassing over 6,000 WSIs sourced from The Cancer Genome Atlas (TCGA) repository facilitated by the National Cancer Institute. The proposed architecture underwent training and assessment using this dataset. Evaluation primarily focused on scrutinizing performance across diverse primary sites and cancer subtypes, with particular attention dedicated to lung cancer classification. Impressively, the proposed architecture yielded outstanding outcomes, showcasing robust performance across the majority of primary sites and cancer subtypes. Furthermore, the study garnered the top position in validation for a lung cancer classification task.