Deep learning for digital pathology image analysis: A comprehensive tutorial with selected use cases

Breast cancer prognostication guides treatment decisions, with transcriptomic assays like Prosigna and Oncotype DX providing valuable risk of recurrence (ROR) scores in ER+/HER2– patients. However, these tests are costly and require sufficient biopsy tissue for accuracy. Pathology images are routinely available and could expedite rapid risk stratification at lower cost. We propose a multi-modal, multi-task deep neural network that learns transcriptomic ROR using digital pathology images and hormone receptor expression status. Because the incorporation of both pathology images and clinical data has proven beneficial in other deep learning models, we hypothesized that incorporation of this complementary information would improve ROR predictions compared to using pathology images alone. We modified the “clustering-constrained-attention multiple-instance learning” (CLAM) deep learning method to perform multi-task regression and infer continuous risk scores (instead of discrete risk categories). We evaluated (1) PAM50 ROR-P, calculated using a multivariate model based on intrinsic subtype centroids and a proliferation score, and (2) a 21-gene assay recurrence score (RS) (research version of Oncotype DX), calculated from component scores relating to proliferation, ER, HER2, and invasion. Using data from 899 patients in the Carolina Breast Cancer Study, a population-based study of diverse patients including 50% Black women and 50% women under age 50, we constructed training (n=714), validation (n=93), and test (n=92) sets. Genomic data were assayed by Nanostring, and digital pathology was based on H&E-stained whole slide images (WSIs). Models were assessed using Pearson correlation between measured and predicted ROR-P or RS. The model with the highest correlations on the test set used H&E WSIs and ER/PR/HER2 expression status as inputs and generated 10 independent outputs: ROR-P, four PAM50 centroid correlations (i.e., Basal, HER2-enriched, Luminal A, and Luminal B), RS, and four RS component scores. Prediction was strong when considering all participants in the test set (ROR-P, 0.78; RS, 0.83, N, 92) but was reduced among more clinically homogeneous ER+/HER2– patients (ROR-P, 0.55; RS, 0.49; N, 49). For RS, correlations were higher in models that included ER/PR/HER2 (Overall, 0.83; ER+/HER2–, 0.50) than in those that relied on pathology alone (Overall, 0.60, ER+/HER2–, 0.34). Future models will consider strategies to enhance performance in ER+/HER2– patients, such as oversampling of ER+/HER2– patients and use of loss functions designed to focus learning on mis-predicted examples. In summary, multi-task, tissue-based regression deep learning models recapitulate transcription-based risk assays with high correlations and, with optimization (especially integration of additional clinicopathologic data), hold promise for personalized treatment decisions from early, routinely-collected biopsy images. Citation Format: Jakub R Kaczmarzyk, Luke A Torre-Healy, Richard A Moffitt, Rajarsi Gupta, Alina M Hamilton, Tahsin M Kurc, Katherine A Hoadley, Melissa A Troester, Joel H Saltz. Early risk stratification of ER+/HER2– breast cancer patients using digital pathology and multi-task, weakly-supervised deep learning [abstract]. In: Proceedings of the AACR Special Conference in Cancer Research: Advances in Breast Cancer Research; 2023 Oct 19-22; San Diego, California. Philadelphia (PA): AACR; Cancer Res 2024;84(3 Suppl_1):Abstract nr B082.

1 - Congruence of deep learning in biomedical engineering: future prospects and challenges

Simple SummaryThe combination of digital pathology (DP) with artificial intelligence (AI) offers faster, more accurate, and more comprehensive diagnoses, resulting in more precise individualized treatment. As this technology is constantly evolving, it is critical to understand the current state of AI applications in DP. Thus, it is necessary to analyze AI patent applications, assignees, and leaders in the field. In this study, five major patent databases, namely, those of the USPTO, EPO, KIPO, JPO, and CNIPA, were searched using key phrases, such as DP, AI, machine learning, and deep learning, and 523 patents were shortlisted based on the inclusion criteria. Our data demonstrated that the key areas of the patents were whole-slide imaging, segmentation, classification, and detection. In the past five years, an increasing trend in patent filing has been observed, mainly in a few prominent countries, with a focus on the digitization of pathological images and AI technologies that support the critical role of pathologists.The integration of digital pathology (DP) with artificial intelligence (AI) enables faster, more accurate, and thorough diagnoses, leading to more precise personalized treatment. As technology is advancing rapidly, it is critical to understand the current state of AI applications in DP. Therefore, a patent analysis of AI in DP is required to assess the application and publication trends, major assignees, and leaders in the field. We searched five major patent databases, namely, those of the USPTO, EPO, KIPO, JPO, and CNIPA, from 1974 to 2021, using keywords such as DP, AI, machine learning, and deep learning. We discovered 6284 patents, 523 of which were used for trend analyses on time series, international distribution, top assignees; word cloud analysis; and subject category analyses. Patent filing and publication have increased exponentially over the past five years. The United States has published the most patents, followed by China and South Korea (248, 117, and 48, respectively). The top assignees were Paige.AI, Inc. (New York City, NY, USA) and Siemens, Inc. (Munich, Germany) The primary areas were whole-slide imaging, segmentation, classification, and detection. Based on these findings, we expect a surge in DP and AI patent applications focusing on the digitalization of pathological images and AI technologies that support the vital role of pathologists.

Deep Learning in Medical Ultrasound Analysis: A Review

Artificial Intelligence and the Pathologist: Future Frenemies?

Chapter 11 - Artificial intelligence for pathology

BackgroundPathological complete response (pCR) to neoadjuvant systemic therapy (NAST) is an established prognostic marker in breast cancer (BC). Multimodal deep learning (DL), integrating diverse data sources (radiology, pathology, omics, clinical), holds promise for improving pCR prediction accuracy. This systematic review synthesizes evidence on multimodal DL for pCR prediction and compares its performance against unimodal DL.MethodsFollowing PRISMA, we searched PubMed, Embase, and Web of Science (January 2015–April 2025) for studies applying DL to predict pCR in BC patients receiving NAST, using data from radiology, digital pathology (DP), multi-omics, and/or clinical records, and reporting AUC. Data on study design, DL architectures, and performance (AUC) were extracted. A narrative synthesis was conducted due to heterogeneity.ResultsFifty-one studies, mostly retrospective (90.2%, median cohort 281), were included. Magnetic resonance imaging and DP were common primary modalities. Multimodal approaches were used in 52.9% of studies, often combining imaging with clinical data. Convolutional neural networks were the dominant architecture (88.2%). Longitudinal imaging improved prediction over baseline-only (median AUC 0.91 vs. 0.82). Overall, the median AUC across studies was 0.88, with 35.3% achieving AUC ≥ 0.90. Multimodal models showed a modest but consistent improvement over unimodal approaches (median AUC 0.88 vs. 0.83). Omics and clinical text were rarely primary DL inputs.ConclusionDL models demonstrate promising accuracy for pCR prediction, especially when integrating multiple modalities and longitudinal imaging. However, significant methodological heterogeneity, reliance on retrospective data, and limited external validation hinder clinical translation. Future research should prioritize prospective validation, integration underutilized data (multi-omics, clinical), and explainable AI to advance DL predictors to the clinical setting.

The study focuses on the utilization of artificial intelligence (AI) algorithms in the diagnosis of breast, lung, and prostate cancer. It describes the historical development of the digitalization of pathological processes, the implementation of artificial intelligence, and its current applications in pathology. The study emphasizes machine learning, deep learning, computer vision, and digital pathology, which contribute to the automation and refinement of diagnostics. Special attention is given to specific tools such as the uPath systems from Roche and IBEX Medical Analytics, which enable the analysis of histopathological images, tumor cell classification, and biomarker evaluation. The study also highlights the benefits of AI utilization, including increased diagnostic accuracy and efficiency in laboratory processes, while simultaneously addressing the challenges associated with its implementation, such as ethical and legal considerations, data protection, and liability for errors. The aim of this study is to provide a comprehensive overview of the potential applications of AI in digital pathology and its role in modern oncological diagnostics.

With the rapid advancement in large-throughput scanning technologies, digital pathology has emerged as platform with promise for diagnostic approaches, but also for high-throughput quantitative data extraction and analysis for translational research. Digital pathology and biomarker images are rich sources of information on tissue architecture, cell diversity and morphology, and molecular pathway activation. However, the understanding of disease in three-dimension (3D) has been hampered by their traditional two-dimension (2D) representations on histologic slides. In this paper, we propose a scalable image processing framework to quantitatively investigate 3D phenotypic and cell-specific molecular features from digital pathology and biomarker images in information- lossless 3D tissue space. We also develop a generalized 3D spatial data management framework with multi-level parallelism and provide a sustainable infrastructure for rapid spatial queries through scalable and efficient spatial data processing. The developed framework can facilitate biomedical research by efficiently processing large-scale, 3D pathology and in-situ biomarker imaging data.

Digital pathology analysis is an advancement in medical diagnostics that leverages digital imaging to enhance the examination and interpretation of pathology slides. By converting traditional glass slides into high-resolution digital images, digital pathology enables pathologists to review and analyze samples with greater precision and efficiency. Machine learning-based techniques in digital pathology are attracting significant attention from pathologists and biologists due to their ability to reduce analysis time and provide objective, accurate results. In particular, semantic segmentation approaches have been widely adopted to isolate specific proteins or cell areas within digital pathology images. However, these methods often exhibit biases toward particular datasets which leads to models that can only process specific targets and lack general applicability. To address this limitation, we propose a target-specific segmentation approach using conditional segmentation networks. We introduce a one-hot vector to control the isolation of target proteins in a conditional manner. The proposed method achieved a pixel accuracy of 83.6% and an intersection over union of 0.7954 on a public pathology segmentation dataset.

Digital pathology incorporates the acquisition, management, sharing and interpretation of pathology information in a digital environment. The field of digital pathology is currently regarded as one of the most promising avenues of diagnostic medicine. Many computer-aided detection and diagnostic algorithms has been developed to assist pathologists in their daily clinical routine, with varying degree of success. These include cell detection and counting, tissue classification and cancer grading, among others. Deep learning, or more specifically, deep convolutional neural network, is a machine learning algorithm that has also gained a lot of attention recently due to their ability to achieve state-of-the-art accuracy. In this paper we have constructed and expanded the deep model network to localize tumor regions in histology images of breast carcinoma. We proposed our own deep convolutional neural network with lesser hardware requirement using 64×64×3 input patch. Our proposed method is able to provide reliable tumor region localization, visually and objectively, based on very limited training dataset.

The development of tools for the processing of color images is often complicated by nonstandardness – the notion that different image regions corresponding to the same tissue will occupy different ranges in the color spectrum. In digital pathology (DP), these issues are often caused by variations in slide thickness, staining, scanning parameters, and illumination. Nonstandardness can be addressed via standardization, a pre-processing step that aims to improve color constancy by realigning color distributions of images to match that of a predefined template image. Unlike color normalization methods, which aim to scale (usually linearly or assuming that the transfer function of the system is known) the intensity of individual images, standardization is employed to align distributions in broad tissue classes (e.g. epithelium, stroma) across different DP images irrespective of institution, protocol, or scanner. Intensity standardization has previously been used for addressing the issue of intensity drift in MRI images, where similar tissue regions have different image intensities across scanners and patients. However, this approach is a global standardization (GS) method that aligns histograms of entire images at once. By contrast, histopathological imagery is complicated by the (a) additional information present in color images and (b) heterogeneity of tissue composition. In this paper, we present a novel color Expectation Maximization (EM) based standardization (EMS) scheme to decompose histological images into independent tissue classes (e.g. nuclei, epithelium, stroma, lumen) via the EM algorithm and align the color distributions for each class independently. Experiments are performed on prostate and oropharyngeal histopathology tissues from 19 and 26 patients, respectively. Evaluation methods include (a) a segmentation-based assessment of color consistency in which normalized median intensity (NMI) is calculated from segmented regions across a dataset and (b) a quantitative measure of histogram alignment via mean landmark distance. EMS produces lower NMI standard deviations (i.e. greater consistency) of 0.0054 and 0.0034 for prostate and oropharyngeal cohorts, respectively, than unstandardized (0.034 and 0.026) and GS (0.031 and 0.017) approaches. Similarly, we see decreased mean landmark distance for EMS (2.25 and 4.20) compared to both unstandardized (54.8 and 27.3) and GS (27.1 and 8.8) images. These results suggest that the separation of broad tissue classes in EMS is vital to the standardization of DP imagery and subsequent development of computerized image analysis tools.

Chapter 10 - Artificial intelligence and the interplay between tumor and immunity

Inspired by facial recognition, we define an analogous problem in digital pathology called tissue matching. The objective is to match histology image patches to the patient. Unlike many classification problems in pathology, tissue matching comes with its own ground truth, and does not require subjective annotation. Using tissue-matching via deep neural networks, we developed strategies to overcome common sources of noise in histology data, including H&E and slide scanner variability. We demonstrate that the typical approach of increasing the training set size, in the current study up to 12,000 tissue cores, yields only moderate gains in test accuracy, while including a loss term promoting style-invariance cooperatively improves the impact of scaling training set size. Our best neural network learned features that matched patients from the test set of H&E image patches with 93% accuracy (n=104 patients, baseline accuracy <1%). Leveraging this network’s identification ability, we applied its internal representation as a feature vector (tissue fingerprint) to apply to other problems in pathology. We found that a fingerprint-based classifier could predict estrogen receptor status from breast cancer whole-slides on two separate cohorts with high accuracy (AUCROC=0.89), a sizeable improvement over previous approaches. This work presents a novel, scalable, and transparent approach to train neural networks to identify distinctive features of pathology images. These features have demonstrated utility in classifying biomarkers from morphology and have the potential to assist in a number of difficult tasks in pathology, including tissue subtype classification and patient tumor similarity assessments based on morphometry. Citation Format: Rishi Rawat, Fei Sha, Darryl Shibata, Daniel Ruderman, David Agus. Deep learning tissue "fingerprints" to identify patients and predict clinical subtypes of breast cancer from digital pathology images [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2019; 2019 Mar 29-Apr 3; Atlanta, GA. Philadelphia (PA): AACR; Cancer Res 2019;79(13 Suppl):Abstract nr 1648.

The scarcity of large histopathological datasets can be problematic for Deep Learning in medical imaging and digital pathology. However, transfer Learning has been shown to be promising for the effective training of classifiers on smaller datasets. ImageNet is a popular dataset that is commonly used for transfer learning in various domains. The features extracted from the ImageNet dataset are generalizable and can be applied to alternative tasks and datasets. Deep Learning typically requires a vast amount of data for training, however, in our study we interrogated two datasets with patches extracted from only 30 whole slide images (WSIs) and 60 WSIs respectively. As a consequence, we decided to extract features and feed them into separate classifier models such as a fully connected softmax layer, Support Vector Machines (SVM) and Logistic Regression. This study demonstrated that for the small dataset, the best pretrained feature extractor was DenseNet201, whereas the best model for training was a fully connected softmax layer with a reported accuracy of 88.20% and an average f1-score of 0.881. For the larger dataset size, the best feature extractor was InceptionResNetV2 where the highest accuracy and f1-score of 90.60% and 0.908 was produced when classified using a fully connected softmax layer. All models, apart from ResNet50 demonstrated an improvement in performance when pretraining using ImageNet for bottleneck feature extraction.