Deep-Hipo: Multi-scale receptive field deep learning for histopathological image analysis.

Chapter Thirteen - Toward large-scale histopathological image analysis via deep learning

Automated invasive ductal carcinoma detection based using deep transfer learning with whole-slide images

Artificial Intelligence-Powered Digital Pathology to Improve Diagnosis and Personalized Prognostic Assessment in Patient with Myeloid Neoplasms

BackgroundTargeted therapy and immunotherapy put forward higher demands for accurate lung cancer classification, as well as benign versus malignant disease discrimination. Digital whole slide images (WSIs) witnessed the transition from traditional histopathology to computational approaches, arousing a hype of deep learning methods for histopathological analysis. We aimed at exploring the potential of deep learning models in the identification of lung cancer subtypes and cancer mimics from WSIs.MethodsWe initially obtained 741 WSIs from the First Affiliated Hospital of Sun Yat-sen University (SYSUFH) for the deep learning model development, optimization, and verification. Additional 318 WSIs from SYSUFH, 212 from Shenzhen People’s Hospital, and 422 from The Cancer Genome Atlas were further collected for multi-centre verification. EfficientNet-B5- and ResNet-50-based deep learning methods were developed and compared using the metrics of recall, precision, F1-score, and areas under the curve (AUCs). A threshold-based tumour-first aggregation approach was proposed and implemented for the label inferencing of WSIs with complex tissue components. Four pathologists of different levels from SYSUFH reviewed all the testing slides blindly, and the diagnosing results were used for quantitative comparisons with the best performing deep learning model.ResultsWe developed the first deep learning-based six-type classifier for histopathological WSI classification of lung adenocarcinoma, lung squamous cell carcinoma, small cell lung carcinoma, pulmonary tuberculosis, organizing pneumonia, and normal lung. The EfficientNet-B5-based model outperformed ResNet-50 and was selected as the backbone in the classifier. Tested on 1067 slides from four cohorts of different medical centres, AUCs of 0.970, 0.918, 0.963, and 0.978 were achieved, respectively. The classifier achieved high consistence to the ground truth and attending pathologists with high intraclass correlation coefficients over 0.873.ConclusionsMulti-cohort testing demonstrated our six-type classifier achieved consistent and comparable performance to experienced pathologists and gained advantages over other existing computational methods. The visualization of prediction heatmap improved the model interpretability intuitively. The classifier with the threshold-based tumour-first label inferencing method exhibited excellent accuracy and feasibility in classifying lung cancers and confused nonneoplastic tissues, indicating that deep learning can resolve complex multi-class tissue classification that conforms to real-world histopathological scenarios.

This study aims to develop a deep learning (DL) model based on whole-slide images (WSIs) to predict the pathological stage of clear cell renal cell carcinoma (ccRCC). The histopathological images of 513 ccRCC patients were downloaded from The Cancer Genome Atlas (TCGA) database and randomly divided into training set and validation set according to the ratio of 8∶2. The CLAM algorithm was used to establish the DL model, and the stability of the model was evaluated in the external validation set. DL features were extracted from the model to construct a prognostic risk model, which was validated in an external dataset. The results showed that the DL model showed excellent prediction ability with an area under the curve (AUC) of 0.875 and an average accuracy score of 0.809, indicating that the model could reliably distinguish ccRCC patients at different stages from histopathological images. In addition, the prognostic risk model constructed by DL characteristics showed that the overall survival rate of patients in the high-risk group was significantly lower than that in the low-risk group (P = 0.003), and AUC values for predicting 1-, 3- and 5-year overall survival rates were 0.68, 0.69 and 0.69, respectively, indicating that the prediction model had high sensitivity and specificity. The results of the validation set are consistent with the above results. Therefore, DL model can accurately predict the pathological stage and prognosis of ccRCC patients, and provide certain reference value for clinical diagnosis.

Histopathologic whole-slide images (WSI) are generally considered the gold standard for cancer diagnosis and prognosis. Survival prediction based on WSI has recently attracted substantial attention. Nevertheless, it remains a central challenge owing to the inherent difficulties of predicting patient prognosis and effectively extracting informative survival-specific representations from WSI with highly compounded gigapixels. In this study, we present a fully automated cellular-level dual global fusion pipeline for survival prediction. Specifically, the proposed method first describes the composition of different cell populations on WSI. Then, it generates dimension-reduced WSI-embedded maps, allowing for efficient investigation of the tumor microenvironment. In addition, we introduce a novel dual global fusion network to incorporate global and inter-patch features of cell distribution, which enables the sufficient fusion of different types and locations of cells. We further validate the proposed pipeline using The Cancer Genome Atlas lung adenocarcinoma dataset. Our model achieves a C-index of 0.675 (±0.05) in the five-fold cross-validation setting and surpasses comparable methods. Further, we extensively analyze embedded map features and survival probabilities. These experimental results manifest the potential of our proposed pipeline for applications using WSI in lung adenocarcinoma and other malignancies.

Introduction: In digital pathology, large amounts of time and money are spent annotating whole slide images (WSIs). By using an AI algorithm trained on thousands of WSIs, it is possible to remove the process of annotating from the workflow whilst simultaneously finding potential imaging features that correlate with drug response, molecular phenotypes, histologic features (e.g. Tertiary Lymphoid Structures), inflammation at the tumor-stroma interface, artifacts, etc. Method: A pipeline was constructed for ingesting WSI from the set of diagnostic slides available in the cancer genome archive (TCGA). The WSIs were masked for tissue areas and divided into patches extracted at multiple magnification levels. These patches were used to train a U-Net (initialised with a pre-trained ResNet34) to complete a self-supervised in-painting task and evaluated with perceptual loss using VGG16. After training, embeddings were extracted from the bottleneck layer of the U-Net, and these were evaluated for their capacity to cluster according to tissue type and magnification level. Results: The trained embeddings showed strong clustering by tissue type, magnification level, and separation of artefacts. The results were qualitatively evaluated by pathologist with consensus and exceeded the performance of a baseline pre-trained ImageNet model. Conclusion: These results highlight a novel methodology for AI algorithm development that removes the need for numerous pathologists to annotate pathology images by hand and converts their role to reviewing and quality-checking images. AI algorithms based on this approach can have a significant impact in the pathologic evaluation of tissue for oncology clinical trials by extracting a richer set of features than glass microscopy alone would permit. Furthermore, self-supervised methods, such as in-painting allow for the training of embeddings that encapsulate a richer understanding of the data and are consequently more repurposable than supervised/labelled approaches. Citation Format: Jason Hipp, Mona Xu, Lucas Bordeaux, Feng Gu, Carlos Pedrinaci, Khan Baykaner. Unsupervised learning of image embeddings enables new opportunities to extract novel information from digital pathology H&E images [abstract]. In: Proceedings of the AACR Virtual Special Conference on Artificial Intelligence, Diagnosis, and Imaging; 2021 Jan 13-14. Philadelphia (PA): AACR; Clin Cancer Res 2021;27(5_Suppl):Abstract nr PO-076.

The second most common type of malignant tumor worldwide is colorectal cancer. Histopathology image analysis offers crucial data for the clinical diagnosis of colorectal cancer. Currently, deep learning techniques are applied to enhance cancer classification and tumor localization in histopathological image analysis. Moreover, traditional deep learning techniques might loss integrated information in the image while evaluating thousands of patches recovered from whole slide images (WSIs). This research proposes a novel colorectal cancer detection network (CCDNet) that combines coordinate attention transformer with atrous convolution. CCDNet first denoises the input histopathological image using a Wiener based Midpoint weighted non-local means filter (WMW-NLM) for guaranteeing precise diagnoses and maintain image features. Also, a novel atrous convolution with coordinate attention transformer (AConvCAT) is introduced, which successfully combines the advantages of two networks to classify colorectal tissue at various scales by capturing local and global information. Further, coordinate attention model is integrated with a Cross-shaped window (CrSWin) transformer for capturing tiny changes in colorectal tissue from multiple angles. The proposed CCDNet achieved accuracy rates of 98.61% and 98.96%, on the colorectal histological image and NCT-CRC-HE-100 K datasets correspondingly. The comparison analysis demonstrates that the suggested framework performed better than the most advanced methods already in use. In hospitals, clinicians can use the proposed CCDNet to verify the diagnosis.

Simple SummaryThe combination of digital pathology (DP) with artificial intelligence (AI) offers faster, more accurate, and more comprehensive diagnoses, resulting in more precise individualized treatment. As this technology is constantly evolving, it is critical to understand the current state of AI applications in DP. Thus, it is necessary to analyze AI patent applications, assignees, and leaders in the field. In this study, five major patent databases, namely, those of the USPTO, EPO, KIPO, JPO, and CNIPA, were searched using key phrases, such as DP, AI, machine learning, and deep learning, and 523 patents were shortlisted based on the inclusion criteria. Our data demonstrated that the key areas of the patents were whole-slide imaging, segmentation, classification, and detection. In the past five years, an increasing trend in patent filing has been observed, mainly in a few prominent countries, with a focus on the digitization of pathological images and AI technologies that support the critical role of pathologists.The integration of digital pathology (DP) with artificial intelligence (AI) enables faster, more accurate, and thorough diagnoses, leading to more precise personalized treatment. As technology is advancing rapidly, it is critical to understand the current state of AI applications in DP. Therefore, a patent analysis of AI in DP is required to assess the application and publication trends, major assignees, and leaders in the field. We searched five major patent databases, namely, those of the USPTO, EPO, KIPO, JPO, and CNIPA, from 1974 to 2021, using keywords such as DP, AI, machine learning, and deep learning. We discovered 6284 patents, 523 of which were used for trend analyses on time series, international distribution, top assignees; word cloud analysis; and subject category analyses. Patent filing and publication have increased exponentially over the past five years. The United States has published the most patents, followed by China and South Korea (248, 117, and 48, respectively). The top assignees were Paige.AI, Inc. (New York City, NY, USA) and Siemens, Inc. (Munich, Germany) The primary areas were whole-slide imaging, segmentation, classification, and detection. Based on these findings, we expect a surge in DP and AI patent applications focusing on the digitalization of pathological images and AI technologies that support the vital role of pathologists.

Deep learning is a powerful tool for whole slide image (WSI) analysis. Typically, when performing supervised deep learning, a WSI is divided into small patches, trained and the outcomes are aggregated to estimate disease grade. However, patch-based methods introduce label noise during training by assuming that each patch is independent with the same label as the WSI and neglect overall WSI-level information that is significant in disease grading. Here we present a Graph-Transformer (GT) that fuses a graph-based representation of an WSI and a vision transformer for processing pathology images, called GTP, to predict disease grade. We selected 4,818 WSIs from the Clinical Proteomic Tumor Analysis Consortium (CPTAC), the National Lung Screening Trial (NLST), and The Cancer Genome Atlas (TCGA), and used GTP to distinguish adenocarcinoma (LUAD) and squamous cell carcinoma (LSCC) from adjacent non-cancerous tissue (normal). First, using NLST data, we developed a contrastive learning framework to generate a feature extractor. This allowed us to compute feature vectors of individual WSI patches, which were used to represent the nodes of the graph followed by construction of the GTP framework. Our model trained on the CPTAC data achieved consistently high performance on three-label classification (normal versus LUAD versus LSCC: mean accuracy = 91.2 ± 2.5%) based on five-fold cross-validation, and mean accuracy = 82.3 ± 1.0% on external test data (TCGA). We also introduced a graph-based saliency mapping technique, called GraphCAM, that can identify regions that are highly associated with the class label. Our findings demonstrate GTP as an interpretable and effective deep learning framework for WSI-level classification.

Breast cancer is one of the most common cancers occurring in women and is the main cause of mortality among women aged 20 to 59. Detection of breast cancer in the early stage increases survival rate to 80%, and the rate decreases to around 40% if diagnosed in its later stage. There exists significant need to diagnose breast cancer in its early stage itself among women suspected with the disease. The hematoxylin and eosin (H&E)–stained histopathology tissue samples taken by pathologists from suspected breast areas are viewed with whole slide imaging (WSI) and can be stored as digital histopathology images. Manual analysis of histopathology images is time-consuming and leads to inter- and intraobserver variability. Hence, automated classification methods are in research focus. Breast tumors are classified as benign and malignant tumors, which are non-cancerous and cancerous tumors, respectively. Currently, deep learning through convolutional neural networks is an emerging technology and has proven to be successful for image segmentation and classification applications. These deep CNNs require large labeled datasets, which are lacking in the field of medicine, especially for breast cancer images. Also, high computational power, time, and memory are required if CNN is to be trained from scratch. To overcome this, transfer learning from natural images like ImageNet, comprising of 1.2 million natural images belonging to 1,000 classes, is used. Pretrained models like VGG16, ResNet50, Inception, AlexNet, and DenseNet are employed for transfer learning, and fine-tuning is to be performed for refinement. In the proposed method, four different network architectures, namely, AlexNet, VGG16, Inception v3, and DenseNet121, are used for classification of histopathological images using transfer learning. The top dense layers of these architectures are removed, and custom dense layers are added for classification. Also, the weights of the last two layers are fine-tuned with the histopathology input images dataset. The publicly available breast cancer hisopathology images dataset BreakHis is used for validation of the proposed method. The experimental results prove that transfer learning–based CNN gives better classification accuracy compared to networks learned from scratch. Also, it is inferred from the experimental results that the pretrained model of DenseNet121 gives the highest classification accuracy of 0.9520 compared to the other models and state-of-the-art methods.

Deep learning (DL) has transformed medical imaging, particularly in the realm of oral cancer (OC) diagnosis using histopathological images. Timely detection of OC is es-sential for enhancing precision medicine and saving lives. However, incorrect diagnosis may impede effective treatment. In this study, we have proposed a DL model for OC classification, enhanced diagnosis decision-making and interpretability. We achieve this by starting with colour normalisation of histopathology images using the Vaha-dane 3-stain parameter normalisation and watershed segmentation method, followed by tiling and augmentation. Key features are selected using the weighted Fisher score (WFS) to address class imbalance. The U-Net classifier has been improved by using feature-based inputs instead of full images, reducing computational complexity and training time. The integration of Vahadane normalisation for consistent preprocessing across samples, WFS, and explainable artificial intelligence (XAI) addresses critical challenges in histopathological image analysis. The proposed model surpasses exist-ing approaches with a classification accuracy of 99.54%, and it outperforms Dense- Net201 and VGG10 in precision and reliability. The efficiency in handling imbal-anced datasets and explainability features make it suitable for early precise OC detec-tion, which can reduce diagnostic errors and enhance treatment outcomes.

Image analysis in digital pathology is identified as a challenging field, particularly for AI-driven classification and search tasks. The high-resolution and large-scale nature of whole slide images (WSIs) present significant computational challenges in representing and analyzing these images effectively. The research endeavors to tackle these hurdles by presenting an innovative methodology grounded in self-supervised learning (SSL). Unlike prior SSL approaches that depend on augmenting at the patch level, the novel framework capitalizes on existing primary site information to directly glean effective representations from Whole Slide Images (WSIs). Moreover, the investigation integrates fully supervised contrastive learning to bolster the resilience of these representations for both classification and search endeavors. For experimentation, the study drew upon a dataset encompassing over 6,000 WSIs sourced from The Cancer Genome Atlas (TCGA) repository facilitated by the National Cancer Institute. The proposed architecture underwent training and assessment using this dataset. Evaluation primarily focused on scrutinizing performance across diverse primary sites and cancer subtypes, with particular attention dedicated to lung cancer classification. Impressively, the proposed architecture yielded outstanding outcomes, showcasing robust performance across the majority of primary sites and cancer subtypes. Furthermore, the study garnered the top position in validation for a lung cancer classification task.

The increasing adoption of the whole slide image (WSI) technology in histopathology has dramatically transformed pathologists' workflow and allowed the use of computer systems in histopathology analysis. Extensive research in Artificial Intelligence (AI) with a huge progress has been conducted resulting in efficient, effective, and robust algorithms for several applications including cancer diagnosis, prognosis, and treatment. These algorithms offer highly accurate predictions but lack transparency, understandability, and actionability. Thus, explainable artificial intelligence (XAI) techniques are needed not only to understand the mechanism behind the decisions made by AI methods and increase user trust but also to broaden the use of AI algorithms in the clinical setting. From the survey of over 150 papers, we explore different AI algorithms that have been applied and contributed to the histopathology image analysis workflow. We first address the workflow of the histopathological process. We present an overview of various learning‐based, XAI, and actionable techniques relevant to deep learning methods in histopathological imaging. We also address the evaluation of XAI methods and the need to ensure their reliability on the field.This article is categorized under:Application Areas > Health Care

Whole Slide Images (WSIs) or histopathology images are used in digital pathology. WSIs pose great challenges to deep learning models for clinical diagnosis, owing to their size and lack of pixel-level annotations. With the recent advancements in computational pathology, newer multiple-instance learning-based models have been proposed. Multiple-instance learning for WSIs necessitates creating patches and uses the encoding of these patches for diagnosis. These models use generic pre-trained models (ResNet-50 pre-trained on ImageNet) for patch encoding. The recently proposed KimiaNet, a DenseNet121 model pre-trained on TCGA slides, is a domain-specific pre-trained model. This paper shows the effect of domain-specific pre-training on WSI classification. To investigate the effect of domain-specific pre-training, we considered the current state-of-the-art multiple-instance learning models, 1) CLAM, an attention-based model, and 2) TransMIL, a self-attention-based model, and evaluated the models' confidence and predictive performance in detecting primary brain tumors - gliomas. Domain-specific pre-training improves the confidence of the models and also achieves a new state-of-the-art performance of WSI-based glioma subtype classification, showing a high clinical applicability in assisting glioma diagnosis. We will publicly share our code and experimental results at https://github.com/soham-chitnis10/WSI-domain-specific.