HMIL: Hierarchical Multi-Instance Learning for Fine-Grained Whole Slide Image Classification.

Recent advances in programmed death-1 (PD-1) and programmed death-ligand 1 (PD-L1) immune checkpoint inhibitor (ICI) revolutionized the clinical practice in lung cancer treatment. The PD-L1 immunohistochemistry (IHC) test is a widely used biomarker for ICI responder in lung cancer. However, the accuracy of PD-L1 IHC test for selecting ICI responder is unsatisfactory, especially due to its low specificity. Therefore, methods which could effectively predict the efficacy of ICI are crucial for patient selection. In this article, we apply a deep neural network (DNN) to predict responders for anti-PD1 blockade on the basis of histopathological images of hematoxylin and eosin (H&E) stained tissue. The main difficulty to train DNN for responder prediction is the inability to accurately label at the image patch level. We employed a semi-supervised multi-instance learning (MIL) framework with adaptive positive patch selection in each region-of-interest (ROI). Using a dataset of 250 whole slide images (WSIs) of non-small cell lung cancer (111 responders and 139 nonresponders), we train a DNN-based MIL classifier on a case-level partition of the dataset (150 WSIs) and obtain an area under curve (AUC) of 0.773 for the test dataset (50 WSIs), which outperforms the PD-L1 IHC test (AUC=0.636). These results suggest that a DNN model can be used for assisting clinicians to make decision for treatment plan. We also confirm that the locations of adaptively selected positive patches can give valuable insights into the histological features associated with drug response.

Pathology report generation from whole slide images with knowledge retrieval and multi-level regional feature selection.

Whole Slide Image (WSI) classification often relies on weakly supervised Multiple Instance Learning (MIL) methods to handle gigapixel-resolution images. In various MIL methods, attention-based approaches have shown great potential in modern medicine for cancer diagnosis and treatment. These approaches can model the interrelationships among instances to achieve enhanced bag representation using instance scores and thus promote bag-level classification performance. However, the existing attention-based MIL methods face two challenges: (1) The attention-based instance scores cannot accurately represent the contribution of instances to bag-level classification, making it difficult to identify the discriminative regions in WSIs. (2) Whole-slide pathological image analysis frequently suffers from model overfitting and insufficient representation of positive samples for training. To address the problem of poor discriminative regions in WSIs, we design a module to acquire the accurate contribution weights of instances by introducing the Class Activation Map suitable for WSI (WSICAM). For the second challenge, we adopt a Cross-Slide Augmentation (CSA) module to construct new samples with mixed labels on the basis of discriminative instances for model training. Our framework is composed of two WSICAM modules and one CSA module. The experimental results and visualizations demonstrate that our method achieves state-of-the-art in WSI classification on widely used datasets and exhibits robust capabilities in tumor lesion localization.

From WSI-level to patch-level: Structure prior-guided binuclear cell fine-grained detection.

Reliability of whole slide images as a diagnostic modality for renal allograft biopsies

BackgroundPersonalized medicine and accurate quantification of tumor and biomarker expression have become the cornerstone of cancer diagnostics. This requires Quality Control (QC) of research tissue samples to confirm adequate targeted tumor tissue sampling. Digitalization of stained tissue slides offer a precious way to archive, preserve, and retrieve necessary information when needed. This study is aimed to assess the most significant pre-analytic and analytic factors that might contribute to the efficacy of obtaining accurate whole slide images (WSIs) interpretation. Various studies are needed to identifysuch factors to allow for appropriate AI application and adequate tumor area/percentage quantification. MethodsHematoxylene and Eosine (H&E) satined WSIs collected from tissue specimens provided by the Cooperative Human Tissue Network (CHTN) Midwestern Division (CHTNMWD) were analyzed. Tissue specimens were processed, fixed, stained, and scanned contemporaneously (within 1 month). Two cohorts of malignant, colorectal cancer, 20X WSI (ScanscopeXT, Leica Biosystems, Illinois), were assembled. The study identified a "recent cohort" that included 76 WSIs created on 2018 or later. "Aged cohort" included 73 WSIs from specimens procured in the period of (2012-2014). Twenty recent WSIs of adenocarcinoma cases were used to construct WSIs analysis algorithms (VIS, Visiopharm A/S, Denmark) using machine learning to produce morphometric maps and calculate tissue and tumor areas. ResultsAlgorithmic analysis of 69 WSIs from rescanned aged slides vs. that of contemporaneous WSIs concluded 18 (28%) similar finding in tumor areas (within 10%), 56 (82%) had identicaltissue areas, and 54 (79%) had similar tumor percentages. ConclusionWSIs of aged H&E slides and stained paraffin block re-cuts produce different tumor quantification compared to those of original scanned sslides most likely due to pre-analytical factors. The difference in tumor area detected between original and rescanned WSIs trended upward in the period between 2012 and 2014. Less tumor area was detected as the slides age. Recut and H&E-stained tissues from stored paraffin blocks may detect more tumor due to excess eosinophilia. These results highlights the value of documenting archives of H&E WSIs collected at the procurement time. Such images provide a superior archive over glass slides and Formalin-Fixed Paraffin-Embedded (FFPE) blocks and contribute betterg to WSIs analysis application.

Numerous patch-based methods have recently been proposed for histological image based breast cancer classification. However, their performance could be highly affected by ignoring spatial contextual information in the whole slide image (WSI). To address this issue, we propose a novel hierarchical Graph V-Net by integrating 1) patch-level pre-training and 2) context-based fine-tuning, with a hierarchical graph network. Specifically, a semi-supervised framework based on knowledge distillation is first developed to pre-train a patch encoder for extracting disease-relevant features. Then, a hierarchical Graph V-Net is designed to construct a hierarchical graph representation from neighboring/similar individual patches for coarse-to-fine classification, where each graph node (corresponding to one patch) is attached with extracted disease-relevant features and its target label during training is the average label of all pixels in the corresponding patch. To evaluate the performance of our proposed hierarchical Graph V-Net, we collect a large WSI dataset of 560 WSIs, with 30 labeled WSIs from the BACH dataset (through our further refinement), 30 labeled WSIs and 500 unlabeled WSIs from Yunnan Cancer Hospital. Those 500 unlabeled WSIs are employed for patch-level pre-training to improve feature representation, while 60 labeled WSIs are used to train and test our proposed hierarchical Graph V-Net. Both comparative assessment and ablation studies demonstrate the superiority of our proposed hierarchical Graph V-Net over state-of-the-art methods in classifying breast cancer from WSIs. The source code and our annotations for the BACH dataset have been released at https://github.com/lyhkevin/Graph-V-Net.

BackgroundSince microscopic slides can now be automatically digitized and integrated in the clinical workflow, quality assessment of Whole Slide Images (WSI) has become a crucial issue. We present a no-reference quality assessment method that has been thoroughly tested since 2010 and is under implementation in multiple sites, both public university-hospitals and private entities. It is part of the FlexMIm R&D project which aims to improve the global workflow of digital pathology. For these uses, we have developed two programming libraries, in Java and Python, which can be integrated in various types of WSI acquisition systems, viewers and image analysis tools.MethodsDevelopment and testing have been carried out on a MacBook Pro i7 and on a bi-Xeon 2.7GHz server. Libraries implementing the blur assessment method have been developed in Java, Python, PHP5 and MySQL5. For web applications, JavaScript, Ajax, JSON and Sockets were also used, as well as the Google Maps API. Aperio SVS files were converted into the Google Maps format using VIPS and Openslide libraries.ResultsWe designed the Java library as a Service Provider Interface (SPI), extendable by third parties. Analysis is computed in real-time (3 billion pixels per minute). Tests were made on 5000 single images, 200 NDPI WSI, 100 Aperio SVS WSI converted to the Google Maps format.ConclusionsApplications based on our method and libraries can be used upstream, as calibration and quality control tool for the WSI acquisition systems, or as tools to reacquire tiles while the WSI is being scanned. They can also be used downstream to reacquire the complete slides that are below the quality threshold for surgical pathology analysis. WSI may also be displayed in a smarter way by sending and displaying the regions of highest quality before other regions. Such quality assessment scores could be integrated as WSI's metadata shared in clinical, research or teaching contexts, for a more efficient medical informatics workflow.

Background: Antibody drug conjugates (ADCs) against HER2 have shown meaningful clinical activity in HER2 low breast cancers, defined as 1+ or 2+ staining on immunohistochemistry (IHC) without gene amplification by in situ hybridization (ISH) techniques. Given that these methods were originally developed for an accurate detection of HER2 3+, their sensitivity and robustness for the detection of low and ultra-low levels of HER2 are questionable. We have recently described a deep learning algorithm that can detect signatures of HER2 expression based on training utilizing scanned H&E whole slide images (WSI) of breast cancers for which IHC and mRNA expression levels of HER2 were available. Here, we report the application of our algorithm to two independent breast cancer cohorts. Methods: A model was developed based on recognition of invasive breast cancer in whole slide images of H&E staining, and then trained via computational neural network with multiple instance learning for binary classification of cases as HER2 “negative” and HER2 “expressed” (low). For training, true negatives were defined as having HER2 IHC-0 and mRNA level < 7.6. HER2-low cases were defined as IHC-1+/2+ and mRNA >9. IHC-0 cases with mRNA >7.6 were excluded from the training cohorts. The resulting model (HER2Complete) was able to distinguish HER2-negatives from HER2-low cases with an AUC of 0.91 (+/- 0.08). Here we use Her2Complete to assess HER2 in two additional cohorts that include 901 ER+/HER2 IHC-0 and 52 HER2 IHC 0+ breast cancers from MSK and TCGA cohorts, respectively. For the TCGA cohort, concomitant transcriptomics data (RNASeq) as a reference for HER2 mRNA expression were retrieved and “HER2 expressed” defined as RNASeq expression of HER2 greater than the 90th percentile of the geometric mean of expression of three reference genes not expressed in breast tissues (TTN, MUC13, OR10A6). Values less than this reference cut-off in the TCGA cohort were considered “HER2 not expressed.” Results: Among the 901 IHC-0 test cases from the MSK cohort, the model identified 82 as ‘negative’, whereas 819 were found to have features of HER2 expression (HER2-Low). Of the 82 negative cases in the MSK cohort, all except 13 cases expressed mRNA levels < 9, and 786/819 of the HER2-low cases expressed mRNA levels >8. Of the 52 IHC 0+ cases in the TCGA cohort, 33 also had “HER2 not expressed” by our reference based RNASeq expression cut-off. Our model identified 15 of these 33 as ‘negative’, while 15 of the 19 TCGA cases with IHC 0+ and HER2 ‘expressed’ by our cut-off were identified as ‘HER2-Low’ by our model. Conclusions: AI tools based on the analysis of WSIs of routinely prepared H&E sections may predict HER2 status in breast cancer. This work requires further investigation using treatment response data to demonstrate that cases with morphologic features of low level HER2 expression will respond to ADCs. Citation Format: Gerard Oakley III, Jorge Reis-Filho, David Klimstra, Marc Goldfinger, Yikan Wang, Antonio Marra. Deep learning-based assessment of HER2-low expression on breast cancer H&E digital whole slide images [abstract]. In: Proceedings of the 2022 San Antonio Breast Cancer Symposium; 2022 Dec 6-10; San Antonio, TX. Philadelphia (PA): AACR; Cancer Res 2023;83(5 Suppl):Abstract nr P5-02-33.

Objective: It was the aim of this study to determine the diagnostic accuracy of high-risk prognostic factors and morphological characteristics of retinoblastomas using digital whole slide images (WSI) generated by a scanner. Methods: Forty-seven H&E sections of glass slides with high-risk morphological features of retinoblastoma were analyzed. Slides were scanned as WSI and reviewed. The results were compared with those obtained after reviewing the slides using a regular microscope as the gold standard. McNemar's test (MT), the percentage of agreement (POA), and sensitivity (S) and specificity (Sp) were evaluated between WSI and conventional microscopy. Results: There were no differences with respect to multicentricity, growth type, rosette formation, choroidal invasion, anterior chamber invasion, extraocular extension, scleral extension, optic nerve invasion, necrosis, or Azzopardi effect between WSI analysis and light microscopy (MT, p = 1.0; POA = 100%; S = 100%, and Sp = 100%). Discordance was found in 1 case where calcification could not be found using WSI (MT, p = 1.00; POA = 97.9%; S = 100%, and Sp = 97.8%). Conclusion: To the best of our knowledge, this is the first report using digital pathology (WSI) to evaluate prognostic factors in eyes containing retinoblastomas. Using WSI, the pathologist was able to detect high-risk morphological features in retinoblastoma. To date, WSI is an important tool, in particular for ophthalmic pathologists examining enucleation and exenteration specimens.

Deep learning has significantly enhanced survival prediction using whole slide images (WSIs) by adopting a two-stage learning paradigm: WSI preparation and patient-level prediction. While existing research generally concentrates on developing advanced patient-level prediction modules, the critical importance of WSI preparation has been largely overlooked. In practice, WSI preparation is influenced by numerous factors, including tissue heterogeneity, sampling strategies, and technical considerations. These uncontrollable external factors incur variability in the number of WSIs among patients, introducing significant bias and resulting in inferior performance for patients with few WSIs. To address this challenge, we propose a novel approach named WSI-Diffusion. Unlike existing WSI generation models that produce augmented versions of input WSIs, our method generates entirely new WSIs in representation space to serve as complementary data. WSIDiffusion employs a two-stage hierarchical diffusion process. Two novel modules, WSI-level and patch-level Diffusers are designed to capture complex correlations between WSIs and patches. The generated WSIs are integrated as supplementary data, and a light patient-level prediction module is then trained for survival prediction. Experimental results across five datasets demonstrate the superiority of our proposal.

Develop a predictive model utilizing weakly supervised deep learning techniques to accurately forecast major pathological response (MPR) in patients with resectable non-small cell lung cancer (NSCLC) undergoing neoadjuvant chemoimmunotherapy (NICT), by leveraging whole slide images (WSIs). This retrospective study examined pre-treatment WSIs from 186 patients with non-small cell lung cancer (NSCLC), using a weakly supervised learning framework. We employed advanced deep learning architectures, including DenseNet121, ResNet50, and Inception V3, to analyze WSIs on both micro (patch) and macro (slide) levels. The training process incorporated innovative data augmentation and normalization techniques to bolster the robustness of the models. We evaluated the performance of these models against traditional clinical predictors and integrated them with a novel pathomics signature, which was developed using multi-instance learning algorithms that facilitate feature aggregation from patch-level probability distributions. Univariate and multivariable analyses confirmed histology as a statistically significant prognostic factor for MPR (P-value< 0.05). In patch model evaluations, DenseNet121 led in the validation set with an area under the curve (AUC) of 0.656, surpassing ResNet50 (AUC = 0.626) and Inception V3 (AUC = 0.654), and showed strong generalization in external testing (AUC = 0.611). Further evaluation through visual inspection of patch-level data integration into WSIs revealed XGBoost's superior class differentiation and generalization, achieving the highest AUCs of 0.998 in training and robust scores of 0.818 in validation and 0.805 in testing. Integrating pathomics features with clinical data into a nomogram yielded AUC of 0.819 in validation and 0.820 in testing, enhancing discriminative accuracy. Gradient-weighted Class Activation Mapping (Grad-CAM) and feature aggregation methods notably boosted the model's interpretability and feature modeling. The application of weakly supervised deep learning to WSIs offers a powerful tool for predicting MPR in NSCLC patients treated with NICT.

BackgroundDistributed ledger technology (DLT) enables the creation of tamper-resistant, decentralized, and secure digital ledgers. A non-fungible token (NFT) represents a record on-chain associated with a digital or physical asset, such as a whole-slide image (WSI). The InterPlanetary File System (IPFS) represents an off-chain network, hypermedia, and file sharing peer-to-peer protocol for storing and sharing data in a distributed file system. Today, we need cheaper, more efficient, highly scalable, and transparent solutions for WSI data storage and access of medical records and medical imaging data. MethodsWSIs were created from non-human tissues and H&E-stained sections were scanned on a Philips Ultrafast WSI scanner at 40× magnification objective lens (1 μm/pixel). TIFF images were stored on IPFS, while NFTs were minted on the Ethereum blockchain network in ERC-1155 standard. WSI-NFTs were stored on MetaMask and OpenSea was used to display the WSI-NFT collection. Filebase storage application programing interface (API) were used to create dedicated gateways and content delivery networks (CDN). ResultsA total of 10 WSI-NFTs were minted on the Ethereum blockchain network, found on our collection “Whole Slide Images as Non-fungible Tokens Project” on Open Sea: https://opensea.io/collection/untitled-collection-126765644. WSI TIFF files ranged in size from 1.6 to 2.2 GB and were stored on IPFS and pinned on 3 separate nodes. Under optimal conditions, and using a dedicated CDN, WSI reached retrieved at speeds of over 10 mb/s, however, download speeds and WSI retrieval times varied significantly depending on the file and gateway used. Overall, the public IPFS gateway resulted in variably poorer WSI download retrieval performance compared to gateways provided by Filebase storage API. ConclusionWhole-slide images, as the most complex and substantial data files in healthcare, demand innovative solutions. In this technical report, we identify pitfalls in IPFS, and demonstrate proof-of-concept using a 3-layer architecture for scalable, decentralized storage, and access. Optimized through dedicated gateways and CDNs, which can be effectively applied to all medical data and imaging modalities across the healthcare sector. DLT and off-chain network solutions present numerous opportunities for advancements in clinical care, education, and research. Such approaches uphold the principles of equitable healthcare data ownership, security, and democratization, and are poised to drive significant innovation.

Combining big data and deep learning can analyze large-scale breast cancer pathology images for auxiliary diagnosis. Furthermore, Whole Slide Images (WSIs) of breast cancer pathology offer detailed tissue feature information, which supports the accurate identification of malignant lesions. Current approaches combine Self-Supervised Learning (SSL) and Multiple Instance Learning (MIL) for WSI analysis, aiming to address the issues of billion-level pixels in a single WSI and the lack of precise annotations. However, pseudo-labels produced by SSL frequently lack accuracy, and MIL fails to effectively integrate global information at the WSI level, resulting in performance bottlenecks. This paper proposes the Multi-granularity Self-supervised Second-order MIL (MS2M) to tackle these issues. MS2M first achieves instance-level fine-grained feature learning through multi-granularity SSL and optimizes instance-level representations using bag-level labels within the MIL framework. Then, the transformer captures long-range dependencies between instances. When combined with second-order (covariance) pooling, it also captures high-order relational information. This process generates a robust bag-level representation. MS2M achieves accuracies of 0.9845 and 0.9719 on the CAMELYON16 and private breast cancer WSI datasets, respectively, outperforming existing methods.

In computational pathology, extracting and representing spatial features from gigapixel whole slide images (WSIs) are fundamental tasks, but due to their large size, WSIs are typically segmented into smaller tiles. A critical aspect of analyzing WSIs is how information across tiles is aggregated to predict outcomes such as patient prognosis. We introduce a model that combines a message-passing graph neural network (GNN) with a state space model (Mamba) to capture both local and global spatial relationships among the tiles in WSIs. The model’s effectiveness was demonstrated in predicting progression-free survival among patients with early-stage lung adenocarcinomas (LUAD). We compared the model with other state-of-the-art methods for tile-level information aggregation in WSIs, including statistics-based, multiple instance learning (MIL)-based, GNN-based, and GNN-transformer-based aggregation. Our model achieved the highest c-index (0.70) and has the largest number of parameters among comparison models yet maintained a short inference time. Additional experiments showed the impact of different types of node features and different tile sampling strategies on model performance. Code: https://github.com/rina-ding/gat-mamba.