CNN stability training improves robustness to scanner and IHC-based image variability for epithelium segmentation in cervical histology

Non‐Small Cell Lung Cancer (NSCLC) has the highest cancer‐related mortality rate worldwide. While biopsy‐based diagnosis is critical for prognosis and treatment, the intricate anatomical features in Whole Slide Images (WSIs) make manual classification challenging for pathologists. Current deep learning models have been developed to aid in the automatic classification of NSCLC, but many rely on extensive manual annotations and lack efficient multi‐scale feature extraction, limiting their ability to capture diverse patterns in WSIs. There is a need to explore multipath, multi‐scale Convolutional Neural Networks (CNN) that can effectively capture these diverse patterns in WSIs. This study proposes a novel deep learning model, a Multi‐scale, Dual‐Path CNN (MDP‐CNN), designed to automatically classify NSCLC subtypes by capturing heterogeneous patterns and features in WSIs. The model was trained on two independent datasets, LC25000 and The Cancer Genome Atlas (TCGA), demonstrating notable improvements in performance metrics, achieving accuracy scores of 0.981 and 0.958, Area Under Curve (AUC) scores of 0.978 and 0.995, and kappa scores of 0.957 and 0.903 for the LC25000 and TCGA datasets, respectively. Extensive analyses, including ablation studies, interpretation plots, and cross‐dataset analysis, were conducted to demonstrate the efficacy of the proposed model. Multi‐scale processing improved the model's precision in classifying lung cancer subtypes by capturing variations in histopathological features across different resolutions. The proposed model outperformed state‐of‐the‐art models by approximately 8% in accuracy and 3% in AUC, demonstrating the effectiveness of MDP CNNs in improving WSI‐based diagnostics and supporting automated NSCLC classification and clinical decisions.

Deep Convolutional Neural Networks (CNN) are at the backbone of the state–of–the art methods to automatically analyze Whole Slide Images (WSIs) of digital tissue slides. One challenge to train fully-supervised CNN models with WSIs is providing the required amount of costly, manually annotated data. This paper presents a semi-weakly supervised model for classifying prostate cancer tissue. The approach follows a teacher-student learning paradigm that allows combining a small amount of annotated data (tissue microarrays with regions of interest traced by pathologists) with a large amount of weakly-annotated data (whole slide images with labels extracted from the diagnostic reports). The task of the teacher model is to annotate the weakly-annotated images. The student is trained with the pseudo-labeled images annotated by the teacher and fine-tuned with the small amount of strongly annotated data. The evaluation of the methods is in the task of classification of four Gleason patterns and the Gleason score in prostate cancer images. Results show that the teacher-student approach improves significatively the performance of the fully-supervised CNN, both at the Gleason pattern level in tissue microarrays (respectively \(\kappa = 0.594 \pm 0.022\) and \(\kappa = 0.559 \pm 0.034\)) and at the Gleason score level in WSIs (respectively \(\kappa = 0.403 \pm 0.046\) and \(\kappa = 0.273 \pm 0.12\)). Our approach opens the possibility of transforming large weakly–annotated (and unlabeled) datasets into valuable sources of supervision for training robust CNN models in computational pathology.

Ki-67 index is an important diagnostic factor in gastrointestinal neuroendocrine tumor (GI-NET). The current gold standard for grading GI-NETs involves the visual screening of histopathologically stained tissues, for hot-spots containing high amounts of proliferating tumor cells (stained with Ki-67 antibody). Subsequently, the Ki-67 index, i.e. the percentage of proliferating tumor cells within the hot-spot is manually obtained. To automate this subjective and time consuming process, we have developed an integrated pipeline, termed SKIE (synaptophysin-Ki-67 index estimator), combining double-immunohistochemical (IHC) staining for synaptophysin (stains tumor) and Ki-67, with whole slide image (WSI) analysis. The Ki-67 index for 50 human GI-NET WSIs were estimated by SKIE and compared with three pathologists' assessment, and the gold standard (exhaustive counting by a fourth pathologist) based on the double-stained image. All four pathologists unanimously graded 38 WSIs, among which, SKIE achieved 94.74% accuracy. One discrepant case was attributed to staining inconsistencies and the other to SKIE selecting a better hot-spot. The remaining 12 WSIs had discrepant grades among pathologists, and hence, the gold standard was chosen for comparison, wherein, 10 WSI grades matched with that of the gold standard, and SKIE assigned a lower and higher grade to two cases. Overall, SKIE agreed with the gold standard with a substantial linear weighted Cohen's kappa κ = 0.622 with CI [0.286, 0.958]. We further expanded our method to deep-SKIE, wherein, a deep convolutional neural network (DCNN) was trained and validated using 13,736 hotspot-sized tiles from 40 WSIs, each categorized into one of four classes (background, non-tumor, tumor grade 1, tumor grade 2) by SKIE and tested on 9 WSIs. Deep-SKIE achieved an accuracy of 91.63% with near-perfect agreement (κ = 0.88 with CI [0.87, 0.89]) with the gold standard.

BackgroundMicroinfarcts, characteristic lesions of vascular dementia (VaD), are heterogenous and vary in appearance, which pose a considerable challenge for VaD grading as there is great interrater variability in microinfarct assessment. We propose a novel application of machine learning (ML) in the automated screening of microinfarcts, addressing a gap in the post‐mortem analysis of VaD in whole slide images (WSIs) from human brain.MethodOur study adapts a patch‐based pipeline with convolutional neural networks (CNNs) to automate microinfarct screening in WSIs. We compare performance and computational cost of different fields‐of‐view (FOV) for the patch‐based method. Additionally, we propose a postprocessing step and leverage multiple FOVs to mitigate false positives. This study is validated against 66 annotated WSIs (N = 40 infarcted, N = 26 un‐infarcted) from Frontal, Parietal, and Occipital white matter regions across 22 cases. Annotations are from a single trained expert, who delineated microinfarct regions and graded white matter rarefaction.ResultWe report screening performance, i.e, the ability to distinguish infarct‐positive from infarct‐negative WSIs, and detection performance, i.e, the ability to localize the microinfarct regions within a WSI. Despite the inherent challenges of the inexact boundaries of microinfarcts, our models demonstrate notable efficacy in screening WSIs for infarcts, with ResNet‐18 achieving 100% accuracy at WSI level. However, the sensitivity in detecting infarct regions is below 50%, based on Intersection over Union thresholds above 20% overlap (a performance metric used to evaluate the accuracy of object detection models). Leveraging multiple FOV improves both detection and screening performances, reducing false positives.ConclusionThis work presents a proof‐of‐concept pipeline driven by machine learning to automate microinfarct screening & detection in WSIs. This workflow provides new avenues in the automated analysis of neuropathological images of microinfarcts, bringing potential advancement in ML‐based dementia and neuropathology research.

Automated segmentation of cell membranes to evaluate HER2 status in whole slide images using a modified deep learning network

The diagnosis of plasma cell neoplasms requires accurate, and ideally precise, percentages. This plasma cell percentage is often determined by visual estimation of CD138-stained bone marrow biopsies and clot sections. While not necessarily inaccurate, estimates are by definition imprecise. For this study, we hypothesized that deep learning can be used to improve precision. We trained a semantic segmentation-based convolutional neural network (CNN) using annotations of CD138+ and CD138- cells provided by one pathologist on small image patches of bone marrow and validated the CNN on an independent test set of image patches using annotations from two pathologists and a non-deep learning commercial software. On validation, we found that the intraclass correlation coefficients for plasma cell percentages between the CNN and pathologist #1, a non-deep learning commercial software and pathologist #1, and pathologists #1 and #2 were 0.975, 0.892, and 0.994, respectively. The overall results show that CNN labels were almost as accurate as pathologist labels at a cell-by-cell level. Once satisfied with performance, we scaled-up the CNN to evaluate whole slide images (WSIs), and deployed the system as a workflow friendly web application to measure plasma cell percentages using snapshots taken from microscope cameras.

Terminal duct lobular unit (TDLU) involution is the regression of milk-producing structures in the breast. Women with less TDLU involution are more likely to develop breast cancer. A major bottleneck in studying TDLU involution in large cohort studies is the need for labor-intensive manual assessment of TDLUs. We developed a computational pathology solution to automatically capture TDLU involution measures. Whole slide images (WSIs) of benign breast biopsies were obtained from the Nurses’ Health Study. A set of 92 WSIs was annotated for acini, TDLUs and adipose tissue to train deep convolutional neural network (CNN) models for detection of acini, and segmentation of TDLUs and adipose tissue. These networks were integrated into a single computational method to capture TDLU involution measures including number of TDLUs per tissue area, median TDLU span and median number of acini per TDLU. We validated our method on 40 additional WSIs by comparing with manually acquired measures. Our CNN models detected acini with an F1 score of 0.73±0.07, and segmented TDLUs and adipose tissue with Dice scores of 0.84±0.13 and 0.87±0.04, respectively. The inter-observer ICC scores for manual assessments on 40 WSIs of number of TDLUs per tissue area, median TDLU span, and median acini count per TDLU were 0.71, 0.81 and 0.73, respectively. Intra-observer reliability was evaluated on 10/40 WSIs with ICC scores of >0.8. Inter-observer ICC scores between automated results and the mean of the two observers were: 0.80 for number of TDLUs per tissue area, 0.57 for median TDLU span, and 0.80 for median acini count per TDLU. TDLU involution measures evaluated by manual and automated assessment were inversely associated with age and menopausal status. We developed a computational pathology method to measure TDLU involution. This technology eliminates the labor-intensiveness and subjectivity of manual TDLU assessment, and can be applied to future breast cancer risk studies.

Breast cancer prognostication guides treatment decisions, with transcriptomic assays like Prosigna and Oncotype DX providing valuable risk of recurrence (ROR) scores in ER+/HER2– patients. However, these tests are costly and require sufficient biopsy tissue for accuracy. Pathology images are routinely available and could expedite rapid risk stratification at lower cost. We propose a multi-modal, multi-task deep neural network that learns transcriptomic ROR using digital pathology images and hormone receptor expression status. Because the incorporation of both pathology images and clinical data has proven beneficial in other deep learning models, we hypothesized that incorporation of this complementary information would improve ROR predictions compared to using pathology images alone. We modified the “clustering-constrained-attention multiple-instance learning” (CLAM) deep learning method to perform multi-task regression and infer continuous risk scores (instead of discrete risk categories). We evaluated (1) PAM50 ROR-P, calculated using a multivariate model based on intrinsic subtype centroids and a proliferation score, and (2) a 21-gene assay recurrence score (RS) (research version of Oncotype DX), calculated from component scores relating to proliferation, ER, HER2, and invasion. Using data from 899 patients in the Carolina Breast Cancer Study, a population-based study of diverse patients including 50% Black women and 50% women under age 50, we constructed training (n=714), validation (n=93), and test (n=92) sets. Genomic data were assayed by Nanostring, and digital pathology was based on H&E-stained whole slide images (WSIs). Models were assessed using Pearson correlation between measured and predicted ROR-P or RS. The model with the highest correlations on the test set used H&E WSIs and ER/PR/HER2 expression status as inputs and generated 10 independent outputs: ROR-P, four PAM50 centroid correlations (i.e., Basal, HER2-enriched, Luminal A, and Luminal B), RS, and four RS component scores. Prediction was strong when considering all participants in the test set (ROR-P, 0.78; RS, 0.83, N, 92) but was reduced among more clinically homogeneous ER+/HER2– patients (ROR-P, 0.55; RS, 0.49; N, 49). For RS, correlations were higher in models that included ER/PR/HER2 (Overall, 0.83; ER+/HER2–, 0.50) than in those that relied on pathology alone (Overall, 0.60, ER+/HER2–, 0.34). Future models will consider strategies to enhance performance in ER+/HER2– patients, such as oversampling of ER+/HER2– patients and use of loss functions designed to focus learning on mis-predicted examples. In summary, multi-task, tissue-based regression deep learning models recapitulate transcription-based risk assays with high correlations and, with optimization (especially integration of additional clinicopathologic data), hold promise for personalized treatment decisions from early, routinely-collected biopsy images. Citation Format: Jakub R Kaczmarzyk, Luke A Torre-Healy, Richard A Moffitt, Rajarsi Gupta, Alina M Hamilton, Tahsin M Kurc, Katherine A Hoadley, Melissa A Troester, Joel H Saltz. Early risk stratification of ER+/HER2– breast cancer patients using digital pathology and multi-task, weakly-supervised deep learning [abstract]. In: Proceedings of the AACR Special Conference in Cancer Research: Advances in Breast Cancer Research; 2023 Oct 19-22; San Diego, California. Philadelphia (PA): AACR; Cancer Res 2024;84(3 Suppl_1):Abstract nr B082.

Today, pathology imaging is one of the most common and inexpensive diagnostic/prognostic tools used in oncology, while more sophisticated methods such as next generation sequencing (NGS) remain relatively expensive and not routinely used in a clinical setting. Deep convolutional neural networks (CNNs) have emerged as an important image analysis technology enhancing the workflow of pathologists and improving the prediction of patient prognosis and response to treatment​​. Recently, a few attempts have been made to predict molecular features from tissue imaging using CNNs. While these preliminary results are encouraging, there have been no systematic attempts to link Whole Slide Images (WSIs) to transcriptomic profiles. In this study, we developed a cutting-edge deep learning model named HE2RNA, specifically customized for the direct prediction of gene expression from H&E-stained WSIs without need for annotation from pathologists. Our model was trained and tested on 8,725 patients from 28 different cancer types available at The Cancer Genome Atlas (TCGA). HE2RNA accurately predicted the expression of six gene signatures related to well known cancer hallmarks (angiogenesis, hypoxia, DNA repair, cell cycle and immunity) and performed particularly well for signalling pathways involved in immune cell activation. This indicates that suitably designed deep learning models can recognize subtle structures in tissue imaging and relate them to molecular portraits. Moreover, HE2RNA is designed to generate a spatial representation (virtual map) of any well-predicted gene expression overlaying the H&E slide. Such a virtual map was validated on a double-stained H&E/CD3 slide obtained from an independent hepatocellular carcinoma sample. This spatialization could be useful in augmenting the pathologists' workflow by providing a virtual multiplexed staining for each H&E slide while overcoming the technical issues associated with immunohistochemistry. Various important prognostic factors, such as microsatellite instability (MSI), are derived from molecular features. Microsatellite instability refers to the hypermutability of short repetitive genomic sequences caused by impaired DNA mismatch repair. These mutations frequently observed in gastric and colorectal cancer are associated with better response to immunotherapy. We show that the transcriptomic representation learned by our model can be used to improve the performance of MSI status prediction for small datasets of WSI. This type of setting is common since large databases of matched RNA-Seq profiles and WSI are widely available, while databases of matched MSI status and WSI are more scarce. In the future, such technologies could therefore facilitate universal screening of molecular biomarkers and improved identification of patients that could benefit from new therapeutic strategies. Citation Format: Elodie Pronier, Benoît Schmauch, Alberto Romagnoni, Charlie Saillard, Pascale Maillé, Julien Calderaro, Meriem Sefta, Sylvain Toldo, Mikhail Zaslavskiy, Thomas Clozel, Matahi Moarii, Pierre Courtiol, Gilles Wainrib. HE2RNA: A deep learning model for transcriptomic learning from digital pathology [abstract]. In: Proceedings of the Annual Meeting of the American Association for Cancer Research 2020; 2020 Apr 27-28 and Jun 22-24. Philadelphia (PA): AACR; Cancer Res 2020;80(16 Suppl):Abstract nr 2105.

MMO-Net (Multi-Magnification Organ Network): A use case for Organ Identification using Multiple Magnifications in Preclinical Pathology Studies

Gastric cancer (GC) is one of the leading causes of cancer-related deaths globally, with a 5-year survival rate of less than 40%. While immune checkpoint inhibitors have provided promising therapeutic options for advanced GC, only a small proportion of patients benefit. In this study, we developed a deep learning model using whole-slide images to predict prognoses and sensitivity to immune checkpoint inhibitors in GC patients by predicting a novel marker. Formalin-fixed, paraffin-embedded whole-slide images from 292 patients in the Cancer Genome Atlas-Stomach Adenocarcinoma cohort were analyzed. Tumor regions were identified using a ResNet50-based tumor detection model and validated in HiESD dataset. Tiles classified as malignant were extracted for subsequent analysis. Risk score prediction models were developed using convolutional neural networks, clustering-constrained attention multiple-instance learning (CLAM), and dual-stream multiple-instance learning (DSMIL). Attention heatmap visualization was used to interpret tumor microenvironment (TME) features and a multi-model classification framework utilizing a support vector machine (SVM) was developed to assess the impact of clinical variables in model performances. The results were evaluated using the area under the receiver operating characteristic curve (AUROC), accuracy, sensitivity, specificity, and F1 score. The tumor detection model achieved an AUROC of 0.99 on the training set, 0.92 on the test set, and 0.87 on external test set. Among risk score prediction models, DSMIL demonstrated the highest performance, with an AUROC of 0.73 and accuracy of 0.73 on the training set and AUROC of 0.70 and accuracy of 0.68 on the internal test set. High-risk patients exhibited worse survival outcomes and lower immunotherapy response rates compared with low-risk patients. Feature attribution analysis using attention heatmaps confirmed that the model prioritized TME components, specifically regions with dense lymphocytic infiltration. The muti-modal analysis showed that the image features alone were superior to the model combining image features with clinicopathological data. Deep learning models leveraging whole-slide images show potential in predicting prognoses and immunotherapy responses in GC. By integrating tumor-specific and tumor microenvironmental features, this approach offers a scalable, objective tool for personalized treatment planning, improving precision oncology strategies.

Pathology report generation from whole slide images with knowledge retrieval and multi-level regional feature selection.

Background/AimsTo develop a deep learning system (DLS) that can automatically detect malignant melanoma (MM) in the eyelid from histopathological sections with colossal information density.MethodsSetting: Double institutional study.Study population: We retrospectively...

BackgroundMicrovascular invasion (MVI) is essential for the management of hepatocellular carcinoma (HCC). However, MVI is hard to evaluate in patients without sufficient peri-tumoral tissue samples, which account for over a half of HCC patients.MethodsWe established an MVI deep-learning (MVI-DL) model with a weakly supervised multiple-instance learning framework, to evaluate MVI status using only tumor tissues from the histological whole slide images (WSIs). A total of 350 HCC patients (2917 WSIs) from the First Affiliated Hospital of Sun Yat-sen University (FAHSYSU cohort) were divided into a training and test set. One hundred and twenty patients (504 WSIs) from Dongguan People’s Hospital and Shunde Hospital of Southern Medical University (DG-SD cohort) formed an external test set. Unsupervised clustering and class activation mapping were applied to visualize the key histological features.ResultsIn the FAHSYSU and DG-SD test set, the MVI-DL model achieved an AUC of 0.904 (95% CI 0.888–0.920) and 0.871 (95% CI 0.837–0.905), respectively. Visualization results showed that macrotrabecular architecture with rich blood sinus, rich tumor stroma and high intratumor heterogeneity were identified as the key features associated with MVI ( +), whereas severe immune infiltration and highly differentiated tumor cells were associated with MVI (−). In the simulation of patients with only one WSI or biopsies only, the AUC of the MVI-DL model reached 0.875 (95% CI 0.855–0.895) and 0.879 (95% CI 0.853–0.906), respectively.ConclusionThe effective, interpretable MVI-DL model has potential as an important tool with practical clinical applicability in evaluating MVI status from the tumor areas on the histological slides.Graphical abstract

Histopathological identification of tumor tissue is one of the routine pathological diagnoses for pathologists. Recently, computational pathology has been successfully interpreted by a variety of deep learning-based applications. Nevertheless, the high-efficient and spatial-correlated processing of individual patches have always attracted attention in whole-slide image (WSI) analysis. In this paper, we propose a high-throughput system to detect tumor regions in colorectal cancer histology slides precisely. We train a deep convolutional neural network (CNN) model and design a Monte Carlo (MC) adaptive sampling method to estimate the most representative patches in a WSI. Two conditional random field (CRF) models are designed, namely the correction CRF and the prediction CRF are integrated for spatial dependencies of patches. We use three datasets of colorectal cancer from The Cancer Genome Atlas (TCGA) to evaluate the performance of the system. The overall diagnostic time can be reduced from 56.7 percent to 71.7 percent on the slides of a varying tumor distribution, with an increase in classification accuracy.