Target-Specific Segmentation in Hematoxylin and Eosin (H&E) Stained Pathology Images Using Conditional Segmentation Networks

Breast cancer prognostication guides treatment decisions, with transcriptomic assays like Prosigna and Oncotype DX providing valuable risk of recurrence (ROR) scores in ER+/HER2– patients. However, these tests are costly and require sufficient biopsy tissue for accuracy. Pathology images are routinely available and could expedite rapid risk stratification at lower cost. We propose a multi-modal, multi-task deep neural network that learns transcriptomic ROR using digital pathology images and hormone receptor expression status. Because the incorporation of both pathology images and clinical data has proven beneficial in other deep learning models, we hypothesized that incorporation of this complementary information would improve ROR predictions compared to using pathology images alone. We modified the “clustering-constrained-attention multiple-instance learning” (CLAM) deep learning method to perform multi-task regression and infer continuous risk scores (instead of discrete risk categories). We evaluated (1) PAM50 ROR-P, calculated using a multivariate model based on intrinsic subtype centroids and a proliferation score, and (2) a 21-gene assay recurrence score (RS) (research version of Oncotype DX), calculated from component scores relating to proliferation, ER, HER2, and invasion. Using data from 899 patients in the Carolina Breast Cancer Study, a population-based study of diverse patients including 50% Black women and 50% women under age 50, we constructed training (n=714), validation (n=93), and test (n=92) sets. Genomic data were assayed by Nanostring, and digital pathology was based on H&E-stained whole slide images (WSIs). Models were assessed using Pearson correlation between measured and predicted ROR-P or RS. The model with the highest correlations on the test set used H&E WSIs and ER/PR/HER2 expression status as inputs and generated 10 independent outputs: ROR-P, four PAM50 centroid correlations (i.e., Basal, HER2-enriched, Luminal A, and Luminal B), RS, and four RS component scores. Prediction was strong when considering all participants in the test set (ROR-P, 0.78; RS, 0.83, N, 92) but was reduced among more clinically homogeneous ER+/HER2– patients (ROR-P, 0.55; RS, 0.49; N, 49). For RS, correlations were higher in models that included ER/PR/HER2 (Overall, 0.83; ER+/HER2–, 0.50) than in those that relied on pathology alone (Overall, 0.60, ER+/HER2–, 0.34). Future models will consider strategies to enhance performance in ER+/HER2– patients, such as oversampling of ER+/HER2– patients and use of loss functions designed to focus learning on mis-predicted examples. In summary, multi-task, tissue-based regression deep learning models recapitulate transcription-based risk assays with high correlations and, with optimization (especially integration of additional clinicopathologic data), hold promise for personalized treatment decisions from early, routinely-collected biopsy images. Citation Format: Jakub R Kaczmarzyk, Luke A Torre-Healy, Richard A Moffitt, Rajarsi Gupta, Alina M Hamilton, Tahsin M Kurc, Katherine A Hoadley, Melissa A Troester, Joel H Saltz. Early risk stratification of ER+/HER2– breast cancer patients using digital pathology and multi-task, weakly-supervised deep learning [abstract]. In: Proceedings of the AACR Special Conference in Cancer Research: Advances in Breast Cancer Research; 2023 Oct 19-22; San Diego, California. Philadelphia (PA): AACR; Cancer Res 2024;84(3 Suppl_1):Abstract nr B082.

With the rapid advancement in large-throughput scanning technologies, digital pathology has emerged as platform with promise for diagnostic approaches, but also for high-throughput quantitative data extraction and analysis for translational research. Digital pathology and biomarker images are rich sources of information on tissue architecture, cell diversity and morphology, and molecular pathway activation. However, the understanding of disease in three-dimension (3D) has been hampered by their traditional two-dimension (2D) representations on histologic slides. In this paper, we propose a scalable image processing framework to quantitatively investigate 3D phenotypic and cell-specific molecular features from digital pathology and biomarker images in information- lossless 3D tissue space. We also develop a generalized 3D spatial data management framework with multi-level parallelism and provide a sustainable infrastructure for rapid spatial queries through scalable and efficient spatial data processing. The developed framework can facilitate biomedical research by efficiently processing large-scale, 3D pathology and in-situ biomarker imaging data.

Deep learning for digital pathology image analysis: A comprehensive tutorial with selected use cases

In this chapter, the goal is to discuss various types of standards which are relevant to digital pathology images, especially whole slide imaging (WSI), discussing the role they play as well as their current status. We will consider standards primarily in relation to their use in the clinical practice of pathology. In the last ten years much work has been done to extend the DICOM medical imaging standard to include additional metadata relevant to pathology images and specimens as well as to accommodate the special requirements of whole slide microscopic images. This work, along with efforts underway in other organizations, such as HL7 and Integrating the Healthcare Enterprise are laying the foundations for standards based interoperability in pathology that should allow for integration of digital imaging into the clinical workflow of pathologists.

Inspired by facial recognition, we define an analogous problem in digital pathology called tissue matching. The objective is to match histology image patches to the patient. Unlike many classification problems in pathology, tissue matching comes with its own ground truth, and does not require subjective annotation. Using tissue-matching via deep neural networks, we developed strategies to overcome common sources of noise in histology data, including H&E and slide scanner variability. We demonstrate that the typical approach of increasing the training set size, in the current study up to 12,000 tissue cores, yields only moderate gains in test accuracy, while including a loss term promoting style-invariance cooperatively improves the impact of scaling training set size. Our best neural network learned features that matched patients from the test set of H&E image patches with 93% accuracy (n=104 patients, baseline accuracy <1%). Leveraging this network’s identification ability, we applied its internal representation as a feature vector (tissue fingerprint) to apply to other problems in pathology. We found that a fingerprint-based classifier could predict estrogen receptor status from breast cancer whole-slides on two separate cohorts with high accuracy (AUCROC=0.89), a sizeable improvement over previous approaches. This work presents a novel, scalable, and transparent approach to train neural networks to identify distinctive features of pathology images. These features have demonstrated utility in classifying biomarkers from morphology and have the potential to assist in a number of difficult tasks in pathology, including tissue subtype classification and patient tumor similarity assessments based on morphometry. Citation Format: Rishi Rawat, Fei Sha, Darryl Shibata, Daniel Ruderman, David Agus. Deep learning tissue "fingerprints" to identify patients and predict clinical subtypes of breast cancer from digital pathology images [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2019; 2019 Mar 29-Apr 3; Atlanta, GA. Philadelphia (PA): AACR; Cancer Res 2019;79(13 Suppl):Abstract nr 1648.

Digital pathology images from different hospitals often exhibit variations in color styles due to differences in staining processes, tissue handling, and scanning devices. Integrating data from multiple centers is essential for developing artificial intelligence-driven digital pathology (AIDP) models with improved generalization. However, privacy concerns complicate data sharing, hindering this integration. Here, we propose a self-supervised model, stain lookup table (StainLUT), that leverages the inherent structural similarity between pathology tissue samples of the same disease type across different medical centers and enables stain normalization without the need for cross-center data transfer. Applied to single-center AIDP models, we achieve cross-center tumor localization at the whole-slide level and tumor classification at the patch level, performing comparably to AIDP models trained on centralized or same-center data. StainLUT offers a privacy-preserving solution for stain normalization in unseen medical centers, and holds the potential to facilitate the future deployment of AIDP foundational models under privacy regulations.

BackgroundStaining tissue samples to visualise cellular detail and tissue structure is at the core of pathology diagnosis, but variations in staining can result in significantly different appearances of the tissue sample. While the human visual system is adept at compensating for stain variation, with the growth of digital imaging in pathology, the impact of this variation can be more profound. Despite the ubiquity of haematoxylin and eosin staining in clinical practice worldwide, objective quantification is not yet available. We propose a method for quantitative haematoxylin and eosin stain assessment to facilitate quality assurance of histopathology staining, enabling truly quantitative quality control and improved standardisation.MethodsThe stain quantification method comprises conventional microscope slides with a stain-responsive biopolymer film affixed to one side, called stain assessment slides. The stain assessment slides were characterised with haematoxylin and eosin, and implemented in one clinical laboratory to quantify variation levels.ResultsStain assessment slide stain uptake increased linearly with duration of haematoxylin and eosin staining (r = 0.99), and demonstrated linearly comparable staining to samples of human liver tissue (r values 0.98–0.99). Laboratory implementation of this technique quantified intra- and inter-instrument variation of staining instruments at one point in time and across a five-day period.ConclusionThe proposed method has been shown to reliably quantify stain uptake, providing an effective laboratory quality control method for stain variation. This is especially important for whole slide imaging and the future development of artificial intelligence in digital pathology.

Shortcomings and areas for improvement in digital pathology image segmentation challenges.

Introduction: Colorectal cancer (CRC) is a heterogeneous malignancy, treatment for which has principally been cytotoxic chemotherapy. Herein we describe the assessment of immune-biomarkers in a large patient cohort, demonstrating the therapeutic potential of immunotherapy in CRC. Methods: By immunohistochemistry, employing REMARK criteria, using well-validated antibodies, 4 biomarkers (CD3, CD4, CD8, FOXP3) in over 1000 cases, represented across 33 TMAs, were assessed for their protein expression. Digital pathological assessment of immune biomarker densities were quantified via the application of QuPath, an open source digital pathology platform, developed at Queens University, Belfast (https://qupath.github.io.). Biomarker densities were dichotomised by R.O.C analysis. Kaplan-Meier curves (Log-rank p values) were used to assess the impact of immune cell densities on overall survival (OS) and progression-free survival (PFS). Hazard ratios were assessed using the Cox proportional hazards regression model for univariate analysis. Spearman's rank was used to assess correlations of the biomarkers. Results: Expression of two biomarkers were found to vary by stage and three significantly affected OS. Across the patient cohort CD3, CD4 and CD8 were individually shown to be positively prognostic for OS alone (log-rank p = 0.0007, p = 8.65e-06, p = 0.006, respectively). CD4 demonstrated a more significant impact on OS in later stage disease (Log-rank stage II p = 0.0017, stage III p = 0.0002), while conversely CD8 only showed significance in earlier stage disease (Log-rank stage II p = 0.009, stage III p = 0.546). Although significantly associated with OS and within all stages, CD3 showed no differential expression in a particular stage. Correlative analysis demonstrated a statistically significant association between biomarkers. Upon univariate analysis, all biomarkers remained significantly prognostic for OS. All analyses were performed using R. Conclusions: These data demonstrate the prognostic significance of adaptive immune markers in a large CRC cohort on OS. Moreover, stage specific expression maybe clinically relevant and assessment within stages II and III may provide important stratifications for specific immunotherapy treatment regimes. We additionally add weight to the beneficial utility of digital pathology and its potential for integration into routine tissue biomarker assessment. Additional analysis of immune checkpoint and adaptive immune biomarkers in CRC is warranted. Citation Format: Matthew P. Humphries, Stephanie Craig, Victoria Bingham, Maurice Loughrey, Helen Coleman, Liam Murray, Maughan Tim, Stephen McQuaid, Jacqueline James, Manuel Salto-Tellez. Assessment of immune biomarkers by digital pathological analysis across a large colorectal cancer patient cohort predicts patient outcome and may provide a clinically relevant therapeutic index for immunotherapeutic treatment stratification [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2018; 2018 Apr 14-18; Chicago, IL. Philadelphia (PA): AACR; Cancer Res 2018;78(13 Suppl):Abstract nr 4049.

Digital pathology (DP) offers advantages over glass slide microscopy (GS), but data demonstrating a statistically valid equivalent (i.e. non-inferior) performance of DP against GS are required to permit its use in diagnosis. The aim of this study is to provide evidence of non-inferiority. Seventeen pathologists re-reported 3017 cases by DP. Of these, 1009 were re-reported by the same pathologist, and 2008 by a different pathologist. Re-examination of 10138 scanned slides (2.22terabytes) produced 72 variances between GS and DP reports, including 21 clinically significant variances. Ground truth lay with GS in 12 cases and with DP in nine cases. These results are within the 95% confidence interval for existing intraobserver and interobserver variability, proving that DP is non-inferior to GS. In three cases, the digital platform was deemed to be responsible for the variance, including a gastric biopsy, where Helicobacter pylori only became visible on slides scanned at the ×60 setting, and a bronchial biopsy and penile biopsy, where dysplasia was reported on DP but was not present on GS. This is one of the largest studies proving that DP is equivalent to GS for the diagnosis of histopathology specimens. Error rates are similar in both platforms, although some problems e.g. detection of bacteria, are predictable.

Advances in computer technology continue to bring new innovations to departments of anatomic pathology. This article briefly reviews the present status of digital optical imaging, and explores the directions that this technology may lead over the next several years. Technical requirements for digital microscopic and gross imaging, and the available options for image archival and retrieval are summarized. The advantages of digital images over conventional photography in the conference room, and the usefulness of digital imaging in the frozen section suite and gross room, as an adjunct to surgical signout and as a resource for training and education, are discussed. An approach to the future construction of digital histologic sections and the computer as microscope is described. The digital technologic applications that are now available as components of the surgical pathologist's workstation are enumerated. These include laboratory information systems, computerized voice recognition, and on-line or CD-based literature searching, texts and atlases and, in some departments, on-line image databases. The authors suggest that, in addition to these resources that are already available, tomorrow's surgical pathology workstation will include network-linked digital histologic databases, on-line software for image analysis and 3-D image enhancement, expert systems, and ultimately, advanced pattern recognition capabilities. In conclusion, the authors submit that digital optical imaging is likely to have a significant and positive impact on the future development of anatomic pathology.

Digital pathology and microscope image analysis is widely used in comprehensive studies of cell morphology. Identification and analysis of leukocytes in blood smear images, acquired from bright field microscope, are vital for diagnosing many diseases such as hepatitis, leukaemia and acquired immune deficiency syndrome (AIDS). The major challenge for robust and accurate identification and segmentation of leukocyte in blood smear images lays in the large variations of cell appearance such as size, colour and shape of cells, the adhesion between leukocytes (white blood cells, WBCs) and erythrocytes (red blood cells, RBCs), and the emergence of substantial dyeing impurities in blood smear images. In this paper, an end-to-end leukocyte localization and segmentation method is proposed, named LeukocyteMask, in which pixel-level prior information is utilized for supervisor training of a deep convolutional neural network, which is then employed to locate the region of interests (ROI) of leukocyte, and finally segmentation mask of leukocyte is obtained based on the extracted ROI by forward propagation of the network. Experimental results validate the effectiveness of the propose method and both the quantitative and qualitative comparisons with existing methods indicate that LeukocyteMask achieves a state-of-the-art performance for the segmentation of leukocyte in terms of robustness and accuracy .

Immunoscore (IS) is an important evaluation method for the tumor immune microenvironment (TIME); however, formal IS analysis requires designated reagents and a specific digital pathology software and image data analysis. This study aimed to investigate whether simplified IS (s-IS) can substitute formal IS upon modifying the location of the assessment of the numbers of immune cells and verify that the addition of T cell subset markers to s-IS can enhance the prognostic impact in patients with colorectal cancer (CRC). A total of 82 CRC cases were included in this study. Immunohistochemical analysis was performed using CD3/CD8/CD45RO/FOXP3 on tissue specimens; the expression levels were calculated in the center and perimeter of the tumors using digital pathology. The clinical prognostic significance of the expression of these markers was investigated by concordance index comparison according to their location of assessment and combinations. In the univariate analysis, the CD3, CD8, and FOXP3 levels were significant prognostic factors. Moreover, for each T cell subset marker, the assessment of each T cell subset marker at the tumor perimeter had a stronger prognostic power than that in the tumor center. The modified s-IS (s-IS plus FOXP3 evaluation) was an independent prognostic factor for recurrence-free survival and overall survival through multivariate analysis and demonstrated the best prognostic power compared to other T subset marker combinations. In CRC, TIME evaluation could be simplified by assessing CD3- and CD8-positive T cells in the perimeter of the tumor, and additional FOXP3 evaluation would empower the ability of s-IS evaluation in prognostic assessment.

The rise of Digital Pathology during the past few years is leading to the digitisation of the pathology field; the widespread use of Whole Slide Images (WSI) and the digitisation of the diagnostic process have allowed the introduction of AI-based methods to aid some parts of the process. In this framework, the CADIA project was raised in response to the Galician healthcare system digitisation needs. CADIA aims to develop an AI-based medical image analysis solution for the diagnosis of several pathologies and its demonstration on breast cancer diagnosis from WSIs. In this paper, we describe the development of CADIA, from the capture of requirements to the deployment and integration of the solution into the healthcare system infrastructure. We describe the opportunities, challenges and lessons learned during the project development.KeywordsDigital PathologyWhole Slide ImagesBreast cancer diagnosis

The grading of non-muscle invasive bladder cancer (NMIBC) continues to face challenges due to subjective interpretations, which affect the assessment of its severity. To address this challenge, we are developing an innovative artificial intelligence (AI) system aimed at objectively grading NMIBC. This system uses a novel convolutional neural network (CNN) architecture called the multi-scale pyramidal pretrained CNN to analyze both local and global pathology markers extracted from digital pathology images. The proposed CNN structure takes as input three levels of patches, ranging from small patches (e.g., 128×128\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$128 \ imes 128$$\\end{document}) to the largest size patches (512×512\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$512 \ imes 512$$\\end{document}). These levels are then fused by random forest (RF) to estimate the severity grade of NMIBC. The optimal patch sizes and other model hyperparameters are determined using a grid search algorithm. For each patch size, the proposed system has been trained on 32K patches (comprising 16K low-grade and 16K high-grade samples) and subsequently tested on 8K patches (consisting of 4K low-grade and 4K high-grade samples), all annotated by two pathologists. Incorporating light and efficient processing, defining new benchmarks in the application of AI to histopathology, the ShuffleNet-based AI system achieved notable metrics on the testing data, including 94.25% ± 0.70% accuracy, 94.47% ± 0.93% sensitivity, 94.03% ± 0.95% specificity, and a 94.29% ± 0.70% F1-score. These results highlight its superior performance over traditional models like ResNet-18. The proposed system’s robustness in accurately grading pathology demonstrates its potential as an advanced AI tool for diagnosing human diseases in the domain of digital pathology.