MG-Trans: Multi-Scale Graph Transformer With Information Bottleneck for Whole Slide Image Classification.

Combining big data and deep learning can analyze large-scale breast cancer pathology images for auxiliary diagnosis. Furthermore, Whole Slide Images (WSIs) of breast cancer pathology offer detailed tissue feature information, which supports the accurate identification of malignant lesions. Current approaches combine Self-Supervised Learning (SSL) and Multiple Instance Learning (MIL) for WSI analysis, aiming to address the issues of billion-level pixels in a single WSI and the lack of precise annotations. However, pseudo-labels produced by SSL frequently lack accuracy, and MIL fails to effectively integrate global information at the WSI level, resulting in performance bottlenecks. This paper proposes the Multi-granularity Self-supervised Second-order MIL (MS2M) to tackle these issues. MS2M first achieves instance-level fine-grained feature learning through multi-granularity SSL and optimizes instance-level representations using bag-level labels within the MIL framework. Then, the transformer captures long-range dependencies between instances. When combined with second-order (covariance) pooling, it also captures high-order relational information. This process generates a robust bag-level representation. MS2M achieves accuracies of 0.9845 and 0.9719 on the CAMELYON16 and private breast cancer WSI datasets, respectively, outperforming existing methods.

Whole Slide Image (WSI) classification often relies on weakly supervised Multiple Instance Learning (MIL) methods to handle gigapixel-resolution images. In various MIL methods, attention-based approaches have shown great potential in modern medicine for cancer diagnosis and treatment. These approaches can model the interrelationships among instances to achieve enhanced bag representation using instance scores and thus promote bag-level classification performance. However, the existing attention-based MIL methods face two challenges: (1) The attention-based instance scores cannot accurately represent the contribution of instances to bag-level classification, making it difficult to identify the discriminative regions in WSIs. (2) Whole-slide pathological image analysis frequently suffers from model overfitting and insufficient representation of positive samples for training. To address the problem of poor discriminative regions in WSIs, we design a module to acquire the accurate contribution weights of instances by introducing the Class Activation Map suitable for WSI (WSICAM). For the second challenge, we adopt a Cross-Slide Augmentation (CSA) module to construct new samples with mixed labels on the basis of discriminative instances for model training. Our framework is composed of two WSICAM modules and one CSA module. The experimental results and visualizations demonstrate that our method achieves state-of-the-art in WSI classification on widely used datasets and exhibits robust capabilities in tumor lesion localization.

Survival outcome prediction is a challenging weakly-supervised and ordinal regression task in computational pathology that involves modeling complex interactions within the tumor microenvironment in gigapixel whole slide images (WSIs). Despite recent progress in formulating WSIs as bags for multiple instance learning (MIL), representation learning of entire WSIs remains an open and challenging problem, especially in overcoming: 1) the computational complexity of feature aggregation in large bags, and 2) the data heterogeneity gap in incorporating biological priors such as genomic measurements. In this work, we present a Multimodal Co-Attention Transformer (MCAT) framework that learns an interpretable, dense co-attention mapping between WSIs and genomic features formulated in an embedding space. Inspired by approaches in Visual Question Answering (VQA) that can attribute how word embed-dings attend to salient objects in an image when answering a question, MCAT learns how histology patches attend to genes when predicting patient survival. In addition to visualizing multimodal interactions, our co-attention trans-formation also reduces the space complexity of WSI bags, which enables the adaptation of Transformer layers as a general encoder backbone in MIL. We apply our proposed method on five different cancer datasets (4,730 WSIs, 67 million patches). Our experimental results demonstrate that the proposed method consistently achieves superior performance compared to the state-of-the-art methods.

The histopathologic evaluation of regulatory toxicity studies using artificial intelligence (AI) has the potential to increase study efficiency. For example, AI could initially identify and exclude all organs without histopathologic lesions, allowing pathologists to focus solely on evaluating organs with identified lesions. In this study, whole slide images (WSIs) of liver sections from 58 different rat toxicity studies were collected, along with their corresponding histopathologic lesion diagnoses. Each WSI was labeled as either "lesion" or "no lesion" based on the presence or absence of reported histopathologic lesions. Multiple instance learning (MIL) approaches, including a transformer variant, were tested to predict lesions within a weakly supervised framework. Both methods achieved acceptable to excellent area under the receiver operating characteristic curve (AUROC) scores. Heatmap overlays were employed to visually assess the MIL model's effectiveness in detecting lesions, confirming the accuracy of targeted areas on the WSIs. In addition, using transfer learning principles, the MIL model initially developed for liver WSIs was adapted to kidney WSIs, demonstrating the model's versatility. This study showcases the application of weakly supervised learning for lesion detection in rat WSIs from toxicity studies, with the potential to significantly enhance the efficiency of the histopathologic evaluation process.

In computational pathology, extracting and representing spatial features from gigapixel whole slide images (WSIs) are fundamental tasks, but due to their large size, WSIs are typically segmented into smaller tiles. A critical aspect of analyzing WSIs is how information across tiles is aggregated to predict outcomes such as patient prognosis. We introduce a model that combines a message-passing graph neural network (GNN) with a state space model (Mamba) to capture both local and global spatial relationships among the tiles in WSIs. The model’s effectiveness was demonstrated in predicting progression-free survival among patients with early-stage lung adenocarcinomas (LUAD). We compared the model with other state-of-the-art methods for tile-level information aggregation in WSIs, including statistics-based, multiple instance learning (MIL)-based, GNN-based, and GNN-transformer-based aggregation. Our model achieved the highest c-index (0.70) and has the largest number of parameters among comparison models yet maintained a short inference time. Additional experiments showed the impact of different types of node features and different tile sampling strategies on model performance. Code: https://github.com/rina-ding/gat-mamba.

Prediction of cancer recurrence based on compact graphs of whole slide images

1648P Deep learning identifies oncogenic genetic alterations in BRAF and NTRK in H&E whole slide images from thyroid carcinomas

When multiple instance learning meets foundation models: Advancing histological whole slide image analysis.

Given the special situation of modeling gigapixel images, multiple instance learning (MIL) has become one of the most important frameworks for Whole Slide Image (WSI) classification. In current practice, most MIL networks often face two unavoidable problems in training: i) insufficient WSI data and ii) the sample memorization inclination inherent in neural networks. These problems may hinder MIL models from adequate and efficient training, suppressing the continuous performance promotion of classification models on WSIs. Inspired by the basic idea of Mixup, this paper proposes a new Pseudo-bag Mixup (PseMix) data augmentation scheme to improve the training of MIL models. This scheme generalizes the Mixup strategy for general images to special WSIs via pseudo-bags so as to be applied in MIL-based WSI classification. Cooperated by pseudo-bags, our PseMix fulfills the critical size alignment and semantic alignment in Mixup strategy. Moreover, it is designed as an efficient and decoupled method, neither involving time-consuming operations nor relying on MIL model predictions. Comparative experiments and ablation studies are specially designed to evaluate the effectiveness and advantages of our PseMix. Experimental results show that PseMix could often assist state-of-the-art MIL networks to refresh their classification performance on WSIs. Besides, it could also boost the generalization performance of MIL models in special test scenarios, and promote their robustness to patch occlusion and label noise. Our source code is available at https://github.com/liupei101/PseMix.

Multiple instance learning (MIL) has emerged as the best solution for whole slide image (WSI) classification. It consists of dividing each slide into patches, which are treated as a bag of instances labeled with a global label. MIL includes two main approaches: instance-based and embedding-based. In the former, each patch is classified independently, and then, the patch scores are aggregated to predict the bag label. In the latter, bag classification is performed after aggregating patch embeddings. Even if instance-based methods are naturally more interpretable, embedding-based MILs have usually been preferred in the past due to their robustness to poor feature extractors. Recently, the quality of feature embeddings has drastically increased using self-supervised learning (SSL). Nevertheless, many authors continue to endorse the superiority of embedding-based MIL. We conduct 710 experiments across 4 datasets, comparing 10 MIL strategies, 6 self-supervised methods with 4 backbones, 4 foundation models, and various pathology-adapted techniques. Furthermore, we introduce 4 instance-based MIL methods, never used before in the pathology domain. We show that with a good SSL feature extractor, simple instance-based MILs, with very few parameters, obtain similar or better performance than complex, state-of-the-art (SOTA) embedding-based MIL methods, setting new SOTA results on the BRACS and Camelyon16 datasets. As simple instance-based MIL methods are naturally more interpretable and explainable to clinicians, our results suggest that more effort should be put into well-adapted SSL methods for WSI rather than into complex embedding-based MIL methods.

Deep Learning for Detecting BRCA Mutations in High-Grade Ovarian Cancer Based on an Innovative Tumor Segmentation Method From Whole Slide Images

PurposeBevacizumab is a widely studied targeted therapeutic drug used in conjunction with standard chemotherapy for the treatment of recurrent ovarian cancer. While its administration has been shown to increase progression-free survival (PFS) in patients with advanced-stage ovarian cancer, the lack of identifiable biomarkers for predicting patient response has been a major roadblock in its effective adoption towards personalized medicine.MethodsIn this work, we leverage the latest histopathology foundation models trained on large-scale whole slide image (WSI) datasets to extract ovarian tumor tissue features for predicting bevacizumab response from WSIs.ResultsOur extensive experiments across a combination of different histopathology foundation models and multiple instance learning (MIL) strategies demonstrate the capability of these large models in predicting bevacizumab response in ovarian cancer patients with the best models achieving a patient-level balanced accuracy score close to 70%. Furthermore, these models can effectively stratify high- and low-risk patients (p < 0.05) during the first year of bevacizumab treatment.ConclusionThis work highlights the utility of histopathology foundation models to predict response to bevacizumab treatment from WSIs. The high-attention regions of the WSIs highlighted by these models not only aid the model explainability but also serve as promising imaging biomarkers for treatment prognosis.

Pathology report generation from whole slide images with knowledge retrieval and multi-level regional feature selection.

Background: Antibody drug conjugates (ADCs) against HER2 have shown meaningful clinical activity in HER2 low breast cancers, defined as 1+ or 2+ staining on immunohistochemistry (IHC) without gene amplification by in situ hybridization (ISH) techniques. Given that these methods were originally developed for an accurate detection of HER2 3+, their sensitivity and robustness for the detection of low and ultra-low levels of HER2 are questionable. We have recently described a deep learning algorithm that can detect signatures of HER2 expression based on training utilizing scanned H&amp;E whole slide images (WSI) of breast cancers for which IHC and mRNA expression levels of HER2 were available. Here, we report the application of our algorithm to two independent breast cancer cohorts. Methods: A model was developed based on recognition of invasive breast cancer in whole slide images of H&amp;E staining, and then trained via computational neural network with multiple instance learning for binary classification of cases as HER2 “negative” and HER2 “expressed” (low). For training, true negatives were defined as having HER2 IHC-0 and mRNA level &amp;lt; 7.6. HER2-low cases were defined as IHC-1+/2+ and mRNA &amp;gt;9. IHC-0 cases with mRNA &amp;gt;7.6 were excluded from the training cohorts. The resulting model (HER2Complete) was able to distinguish HER2-negatives from HER2-low cases with an AUC of 0.91 (+/- 0.08). Here we use Her2Complete to assess HER2 in two additional cohorts that include 901 ER+/HER2 IHC-0 and 52 HER2 IHC 0+ breast cancers from MSK and TCGA cohorts, respectively. For the TCGA cohort, concomitant transcriptomics data (RNASeq) as a reference for HER2 mRNA expression were retrieved and “HER2 expressed” defined as RNASeq expression of HER2 greater than the 90th percentile of the geometric mean of expression of three reference genes not expressed in breast tissues (TTN, MUC13, OR10A6). Values less than this reference cut-off in the TCGA cohort were considered “HER2 not expressed.” Results: Among the 901 IHC-0 test cases from the MSK cohort, the model identified 82 as ‘negative’, whereas 819 were found to have features of HER2 expression (HER2-Low). Of the 82 negative cases in the MSK cohort, all except 13 cases expressed mRNA levels &amp;lt; 9, and 786/819 of the HER2-low cases expressed mRNA levels &amp;gt;8. Of the 52 IHC 0+ cases in the TCGA cohort, 33 also had “HER2 not expressed” by our reference based RNASeq expression cut-off. Our model identified 15 of these 33 as ‘negative’, while 15 of the 19 TCGA cases with IHC 0+ and HER2 ‘expressed’ by our cut-off were identified as ‘HER2-Low’ by our model. Conclusions: AI tools based on the analysis of WSIs of routinely prepared H&amp;E sections may predict HER2 status in breast cancer. This work requires further investigation using treatment response data to demonstrate that cases with morphologic features of low level HER2 expression will respond to ADCs. Citation Format: Gerard Oakley III, Jorge Reis-Filho, David Klimstra, Marc Goldfinger, Yikan Wang, Antonio Marra. Deep learning-based assessment of HER2-low expression on breast cancer H&amp;E digital whole slide images [abstract]. In: Proceedings of the 2022 San Antonio Breast Cancer Symposium; 2022 Dec 6-10; San Antonio, TX. Philadelphia (PA): AACR; Cancer Res 2023;83(5 Suppl):Abstract nr P5-02-33.

Identification of Nodular Lymphocyte-Predominant Hodgkin Lymphoma Variant Morphology Using Artificial Intelligence