Vision Transformers and CNN-Based Knowledge-Distillation for Histopathological Image Classification

Non‐Small Cell Lung Cancer (NSCLC) has the highest cancer‐related mortality rate worldwide. While biopsy‐based diagnosis is critical for prognosis and treatment, the intricate anatomical features in Whole Slide Images (WSIs) make manual classification challenging for pathologists. Current deep learning models have been developed to aid in the automatic classification of NSCLC, but many rely on extensive manual annotations and lack efficient multi‐scale feature extraction, limiting their ability to capture diverse patterns in WSIs. There is a need to explore multipath, multi‐scale Convolutional Neural Networks (CNN) that can effectively capture these diverse patterns in WSIs. This study proposes a novel deep learning model, a Multi‐scale, Dual‐Path CNN (MDP‐CNN), designed to automatically classify NSCLC subtypes by capturing heterogeneous patterns and features in WSIs. The model was trained on two independent datasets, LC25000 and The Cancer Genome Atlas (TCGA), demonstrating notable improvements in performance metrics, achieving accuracy scores of 0.981 and 0.958, Area Under Curve (AUC) scores of 0.978 and 0.995, and kappa scores of 0.957 and 0.903 for the LC25000 and TCGA datasets, respectively. Extensive analyses, including ablation studies, interpretation plots, and cross‐dataset analysis, were conducted to demonstrate the efficacy of the proposed model. Multi‐scale processing improved the model's precision in classifying lung cancer subtypes by capturing variations in histopathological features across different resolutions. The proposed model outperformed state‐of‐the‐art models by approximately 8% in accuracy and 3% in AUC, demonstrating the effectiveness of MDP CNNs in improving WSI‐based diagnostics and supporting automated NSCLC classification and clinical decisions.

Today, pathology imaging is one of the most common and inexpensive diagnostic/prognostic tools used in oncology, while more sophisticated methods such as next generation sequencing (NGS) remain relatively expensive and not routinely used in a clinical setting. Deep convolutional neural networks (CNNs) have emerged as an important image analysis technology enhancing the workflow of pathologists and improving the prediction of patient prognosis and response to treatment​​. Recently, a few attempts have been made to predict molecular features from tissue imaging using CNNs. While these preliminary results are encouraging, there have been no systematic attempts to link Whole Slide Images (WSIs) to transcriptomic profiles. In this study, we developed a cutting-edge deep learning model named HE2RNA, specifically customized for the direct prediction of gene expression from H&E-stained WSIs without need for annotation from pathologists. Our model was trained and tested on 8,725 patients from 28 different cancer types available at The Cancer Genome Atlas (TCGA). HE2RNA accurately predicted the expression of six gene signatures related to well known cancer hallmarks (angiogenesis, hypoxia, DNA repair, cell cycle and immunity) and performed particularly well for signalling pathways involved in immune cell activation. This indicates that suitably designed deep learning models can recognize subtle structures in tissue imaging and relate them to molecular portraits. Moreover, HE2RNA is designed to generate a spatial representation (virtual map) of any well-predicted gene expression overlaying the H&E slide. Such a virtual map was validated on a double-stained H&E/CD3 slide obtained from an independent hepatocellular carcinoma sample. This spatialization could be useful in augmenting the pathologists' workflow by providing a virtual multiplexed staining for each H&E slide while overcoming the technical issues associated with immunohistochemistry. Various important prognostic factors, such as microsatellite instability (MSI), are derived from molecular features. Microsatellite instability refers to the hypermutability of short repetitive genomic sequences caused by impaired DNA mismatch repair. These mutations frequently observed in gastric and colorectal cancer are associated with better response to immunotherapy. We show that the transcriptomic representation learned by our model can be used to improve the performance of MSI status prediction for small datasets of WSI. This type of setting is common since large databases of matched RNA-Seq profiles and WSI are widely available, while databases of matched MSI status and WSI are more scarce. In the future, such technologies could therefore facilitate universal screening of molecular biomarkers and improved identification of patients that could benefit from new therapeutic strategies. Citation Format: Elodie Pronier, Benoît Schmauch, Alberto Romagnoni, Charlie Saillard, Pascale Maillé, Julien Calderaro, Meriem Sefta, Sylvain Toldo, Mikhail Zaslavskiy, Thomas Clozel, Matahi Moarii, Pierre Courtiol, Gilles Wainrib. HE2RNA: A deep learning model for transcriptomic learning from digital pathology [abstract]. In: Proceedings of the Annual Meeting of the American Association for Cancer Research 2020; 2020 Apr 27-28 and Jun 22-24. Philadelphia (PA): AACR; Cancer Res 2020;80(16 Suppl):Abstract nr 2105.

BackgroundIn digital pathology, image properties such as color, brightness, contrast and blurriness may vary based on the scanner and sample preparation. Convolutional Neural Networks (CNNs) are sensitive to these variations and may underperform on images from a different domain than the one used for training. Robustness to these image property variations is required to enable the use of deep learning in clinical practice and large scale clinical research.AimsCNN Stability Training (CST) is proposed and evaluated as a method to increase CNN robustness to scanner and Immunohistochemistry (IHC)-based image variability.MethodsCST was applied to segment epithelium in immunohistological cervical Whole Slide Images (WSIs). CST randomly distorts input tiles and factors the difference between the CNN prediction for the original and distorted inputs within the loss function. CNNs were trained using 114 p16-stained WSIs from the same scanner, and evaluated on 6 WSI test sets, each with 23 to 24 WSIs of the same tissue but different scanner/IHC combinations. Relative robustness (rAUC) was measured as the difference between the AUC on the training domain test set (i.e., baseline test set) and the remaining test sets.ResultsAcross all test sets, The AUC of CST models outperformed “No CST” models (AUC: 0.940–0.989 vs. 0.905–0.986, p < 1e − 8), and obtained an improved robustness (rAUC: [−0.038, −0.003] vs. [−0.081, −0.002]). At a WSI level, CST models showed an increase in performance in 124 of the 142 WSIs. CST models also outperformed models trained with random on-the-fly data augmentation (DA) in all test sets ([0.002, 0.021], p < 1e-6).ConclusionCST offers a path to improve CNN performance without the need for more data and allows customizing distortions to specific use cases. A python implementation of CST is publicly available at https://github.com/TIGACenter/CST_v1.

Deep Convolutional Neural Networks (CNN) are at the backbone of the state–of–the art methods to automatically analyze Whole Slide Images (WSIs) of digital tissue slides. One challenge to train fully-supervised CNN models with WSIs is providing the required amount of costly, manually annotated data. This paper presents a semi-weakly supervised model for classifying prostate cancer tissue. The approach follows a teacher-student learning paradigm that allows combining a small amount of annotated data (tissue microarrays with regions of interest traced by pathologists) with a large amount of weakly-annotated data (whole slide images with labels extracted from the diagnostic reports). The task of the teacher model is to annotate the weakly-annotated images. The student is trained with the pseudo-labeled images annotated by the teacher and fine-tuned with the small amount of strongly annotated data. The evaluation of the methods is in the task of classification of four Gleason patterns and the Gleason score in prostate cancer images. Results show that the teacher-student approach improves significatively the performance of the fully-supervised CNN, both at the Gleason pattern level in tissue microarrays (respectively \(\kappa = 0.594 \pm 0.022\) and \(\kappa = 0.559 \pm 0.034\)) and at the Gleason score level in WSIs (respectively \(\kappa = 0.403 \pm 0.046\) and \(\kappa = 0.273 \pm 0.12\)). Our approach opens the possibility of transforming large weakly–annotated (and unlabeled) datasets into valuable sources of supervision for training robust CNN models in computational pathology.

Deep-Hipo: Multi-scale receptive field deep learning for histopathological image analysis.

Ki-67 index is an important diagnostic factor in gastrointestinal neuroendocrine tumor (GI-NET). The current gold standard for grading GI-NETs involves the visual screening of histopathologically stained tissues, for hot-spots containing high amounts of proliferating tumor cells (stained with Ki-67 antibody). Subsequently, the Ki-67 index, i.e. the percentage of proliferating tumor cells within the hot-spot is manually obtained. To automate this subjective and time consuming process, we have developed an integrated pipeline, termed SKIE (synaptophysin-Ki-67 index estimator), combining double-immunohistochemical (IHC) staining for synaptophysin (stains tumor) and Ki-67, with whole slide image (WSI) analysis. The Ki-67 index for 50 human GI-NET WSIs were estimated by SKIE and compared with three pathologists' assessment, and the gold standard (exhaustive counting by a fourth pathologist) based on the double-stained image. All four pathologists unanimously graded 38 WSIs, among which, SKIE achieved 94.74% accuracy. One discrepant case was attributed to staining inconsistencies and the other to SKIE selecting a better hot-spot. The remaining 12 WSIs had discrepant grades among pathologists, and hence, the gold standard was chosen for comparison, wherein, 10 WSI grades matched with that of the gold standard, and SKIE assigned a lower and higher grade to two cases. Overall, SKIE agreed with the gold standard with a substantial linear weighted Cohen's kappa κ = 0.622 with CI [0.286, 0.958]. We further expanded our method to deep-SKIE, wherein, a deep convolutional neural network (DCNN) was trained and validated using 13,736 hotspot-sized tiles from 40 WSIs, each categorized into one of four classes (background, non-tumor, tumor grade 1, tumor grade 2) by SKIE and tested on 9 WSIs. Deep-SKIE achieved an accuracy of 91.63% with near-perfect agreement (κ = 0.88 with CI [0.87, 0.89]) with the gold standard.

Automated segmentation of cell membranes to evaluate HER2 status in whole slide images using a modified deep learning network

BackgroundMicroinfarcts, characteristic lesions of vascular dementia (VaD), are heterogenous and vary in appearance, which pose a considerable challenge for VaD grading as there is great interrater variability in microinfarct assessment. We propose a novel application of machine learning (ML) in the automated screening of microinfarcts, addressing a gap in the post‐mortem analysis of VaD in whole slide images (WSIs) from human brain.MethodOur study adapts a patch‐based pipeline with convolutional neural networks (CNNs) to automate microinfarct screening in WSIs. We compare performance and computational cost of different fields‐of‐view (FOV) for the patch‐based method. Additionally, we propose a postprocessing step and leverage multiple FOVs to mitigate false positives. This study is validated against 66 annotated WSIs (N = 40 infarcted, N = 26 un‐infarcted) from Frontal, Parietal, and Occipital white matter regions across 22 cases. Annotations are from a single trained expert, who delineated microinfarct regions and graded white matter rarefaction.ResultWe report screening performance, i.e, the ability to distinguish infarct‐positive from infarct‐negative WSIs, and detection performance, i.e, the ability to localize the microinfarct regions within a WSI. Despite the inherent challenges of the inexact boundaries of microinfarcts, our models demonstrate notable efficacy in screening WSIs for infarcts, with ResNet‐18 achieving 100% accuracy at WSI level. However, the sensitivity in detecting infarct regions is below 50%, based on Intersection over Union thresholds above 20% overlap (a performance metric used to evaluate the accuracy of object detection models). Leveraging multiple FOV improves both detection and screening performances, reducing false positives.ConclusionThis work presents a proof‐of‐concept pipeline driven by machine learning to automate microinfarct screening &amp; detection in WSIs. This workflow provides new avenues in the automated analysis of neuropathological images of microinfarcts, bringing potential advancement in ML‐based dementia and neuropathology research.

Terminal duct lobular unit (TDLU) involution is the regression of milk-producing structures in the breast. Women with less TDLU involution are more likely to develop breast cancer. A major bottleneck in studying TDLU involution in large cohort studies is the need for labor-intensive manual assessment of TDLUs. We developed a computational pathology solution to automatically capture TDLU involution measures. Whole slide images (WSIs) of benign breast biopsies were obtained from the Nurses’ Health Study. A set of 92 WSIs was annotated for acini, TDLUs and adipose tissue to train deep convolutional neural network (CNN) models for detection of acini, and segmentation of TDLUs and adipose tissue. These networks were integrated into a single computational method to capture TDLU involution measures including number of TDLUs per tissue area, median TDLU span and median number of acini per TDLU. We validated our method on 40 additional WSIs by comparing with manually acquired measures. Our CNN models detected acini with an F1 score of 0.73±0.07, and segmented TDLUs and adipose tissue with Dice scores of 0.84±0.13 and 0.87±0.04, respectively. The inter-observer ICC scores for manual assessments on 40 WSIs of number of TDLUs per tissue area, median TDLU span, and median acini count per TDLU were 0.71, 0.81 and 0.73, respectively. Intra-observer reliability was evaluated on 10/40 WSIs with ICC scores of >0.8. Inter-observer ICC scores between automated results and the mean of the two observers were: 0.80 for number of TDLUs per tissue area, 0.57 for median TDLU span, and 0.80 for median acini count per TDLU. TDLU involution measures evaluated by manual and automated assessment were inversely associated with age and menopausal status. We developed a computational pathology method to measure TDLU involution. This technology eliminates the labor-intensiveness and subjectivity of manual TDLU assessment, and can be applied to future breast cancer risk studies.

The diagnosis of plasma cell neoplasms requires accurate, and ideally precise, percentages. This plasma cell percentage is often determined by visual estimation of CD138-stained bone marrow biopsies and clot sections. While not necessarily inaccurate, estimates are by definition imprecise. For this study, we hypothesized that deep learning can be used to improve precision. We trained a semantic segmentation-based convolutional neural network (CNN) using annotations of CD138+ and CD138- cells provided by one pathologist on small image patches of bone marrow and validated the CNN on an independent test set of image patches using annotations from two pathologists and a non-deep learning commercial software. On validation, we found that the intraclass correlation coefficients for plasma cell percentages between the CNN and pathologist #1, a non-deep learning commercial software and pathologist #1, and pathologists #1 and #2 were 0.975, 0.892, and 0.994, respectively. The overall results show that CNN labels were almost as accurate as pathologist labels at a cell-by-cell level. Once satisfied with performance, we scaled-up the CNN to evaluate whole slide images (WSIs), and deployed the system as a workflow friendly web application to measure plasma cell percentages using snapshots taken from microscope cameras.

The last decade Digital Pathology is coming as a relevant and promising area for cancer research and clinical practice thanks to two main trends, 1) the availability of whole slide scanners for complete pathology slide digitalization, and 2) the development of several computational method for histopathology image analysis. However, there are very few works addressed to analyze the whole-slide digitized images (WSI) because their large resolution (e.g. 80,000 × 80,000 pixels at 40× magnification) resulting in huge computational cost for automatic analysis. This paper presents an application design of a meta-heuristic optimization method based on a genetic algorithm (GA) for exploration and exploitation of regions of interest for diagnosis in a WSI in combination with a Convolutional Neural Network (CNN) trained in previous works [10], [11]. The preliminary results show that presented solution scales in computing time given the initial number of samples (initial population). The developed application in Java including the GA method for WSI analysis could be used for diagnosis support by pathologists thanks of its usability and visual interpretability through a probability map of the invasive tumor regions in the WSI.

MMO-Net (Multi-Magnification Organ Network): A use case for Organ Identification using Multiple Magnifications in Preclinical Pathology Studies

Gastric cancer (GC) is one of the leading causes of cancer-related deaths globally, with a 5-year survival rate of less than 40%. While immune checkpoint inhibitors have provided promising therapeutic options for advanced GC, only a small proportion of patients benefit. In this study, we developed a deep learning model using whole-slide images to predict prognoses and sensitivity to immune checkpoint inhibitors in GC patients by predicting a novel marker. Formalin-fixed, paraffin-embedded whole-slide images from 292 patients in the Cancer Genome Atlas-Stomach Adenocarcinoma cohort were analyzed. Tumor regions were identified using a ResNet50-based tumor detection model and validated in HiESD dataset. Tiles classified as malignant were extracted for subsequent analysis. Risk score prediction models were developed using convolutional neural networks, clustering-constrained attention multiple-instance learning (CLAM), and dual-stream multiple-instance learning (DSMIL). Attention heatmap visualization was used to interpret tumor microenvironment (TME) features and a multi-model classification framework utilizing a support vector machine (SVM) was developed to assess the impact of clinical variables in model performances. The results were evaluated using the area under the receiver operating characteristic curve (AUROC), accuracy, sensitivity, specificity, and F1 score. The tumor detection model achieved an AUROC of 0.99 on the training set, 0.92 on the test set, and 0.87 on external test set. Among risk score prediction models, DSMIL demonstrated the highest performance, with an AUROC of 0.73 and accuracy of 0.73 on the training set and AUROC of 0.70 and accuracy of 0.68 on the internal test set. High-risk patients exhibited worse survival outcomes and lower immunotherapy response rates compared with low-risk patients. Feature attribution analysis using attention heatmaps confirmed that the model prioritized TME components, specifically regions with dense lymphocytic infiltration. The muti-modal analysis showed that the image features alone were superior to the model combining image features with clinicopathological data. Deep learning models leveraging whole-slide images show potential in predicting prognoses and immunotherapy responses in GC. By integrating tumor-specific and tumor microenvironmental features, this approach offers a scalable, objective tool for personalized treatment planning, improving precision oncology strategies.

BackgroundTargeted therapy and immunotherapy put forward higher demands for accurate lung cancer classification, as well as benign versus malignant disease discrimination. Digital whole slide images (WSIs) witnessed the transition from traditional histopathology to computational approaches, arousing a hype of deep learning methods for histopathological analysis. We aimed at exploring the potential of deep learning models in the identification of lung cancer subtypes and cancer mimics from WSIs.MethodsWe initially obtained 741 WSIs from the First Affiliated Hospital of Sun Yat-sen University (SYSUFH) for the deep learning model development, optimization, and verification. Additional 318 WSIs from SYSUFH, 212 from Shenzhen People’s Hospital, and 422 from The Cancer Genome Atlas were further collected for multi-centre verification. EfficientNet-B5- and ResNet-50-based deep learning methods were developed and compared using the metrics of recall, precision, F1-score, and areas under the curve (AUCs). A threshold-based tumour-first aggregation approach was proposed and implemented for the label inferencing of WSIs with complex tissue components. Four pathologists of different levels from SYSUFH reviewed all the testing slides blindly, and the diagnosing results were used for quantitative comparisons with the best performing deep learning model.ResultsWe developed the first deep learning-based six-type classifier for histopathological WSI classification of lung adenocarcinoma, lung squamous cell carcinoma, small cell lung carcinoma, pulmonary tuberculosis, organizing pneumonia, and normal lung. The EfficientNet-B5-based model outperformed ResNet-50 and was selected as the backbone in the classifier. Tested on 1067 slides from four cohorts of different medical centres, AUCs of 0.970, 0.918, 0.963, and 0.978 were achieved, respectively. The classifier achieved high consistence to the ground truth and attending pathologists with high intraclass correlation coefficients over 0.873.ConclusionsMulti-cohort testing demonstrated our six-type classifier achieved consistent and comparable performance to experienced pathologists and gained advantages over other existing computational methods. The visualization of prediction heatmap improved the model interpretability intuitively. The classifier with the threshold-based tumour-first label inferencing method exhibited excellent accuracy and feasibility in classifying lung cancers and confused nonneoplastic tissues, indicating that deep learning can resolve complex multi-class tissue classification that conforms to real-world histopathological scenarios.

Colorectal cancer (CRC) stands as one of the deadliest cancer types worldwide. Identifying specific gene mutations crucial for targeted therapy is pivotal for CRC treatment. However, the current genetic testing method involves liquid biopsy, which is often cumbersome and cost-intensive. Histopathological whole-slide images (WSIs) are routinely generated for diagnostic purposes and can offer valuable insights into cellular heterogeneity. In this study, we aim to develop an integrated deep learning pipeline for predicting TP53 mutations in CRC, one of the most frequently mutated genes in CRC. We commenced by collecting 424 diagnostic WSIs from the TCGA-COAD cohorts. Each WSI was split as small patches at a resolution of 0.5 micrometers per pixel for following analyses. Quality control was implemented using Otsu thresholding and Gaussian blur filtering, both at the slide-level masking and patch-level filtering stages. Additionally, we applied the Macenko algorithm for color normalization to eliminate systematic errors. Following image preprocessing, we constructed a tissue-type classification model based on VGG19, which was trained and tested using the NCT-CRC-HE-100K and CRC-VAL-HE-7K datasets, both containing pixels with tissue types labeled by human experts. The trained classifier was then used to identify adenocarcinoma epithelium patches from TCGA-COAD. Subsequently, we applied multiple pre-trained Convolutional Neural Network (CNN) models, including ResNet-101, DenseNet-201, and Inception-ResNet-v2, to predict TP53 mutations for the identified adenocarcinoma epithelium patches from TCGA-COAD. A novel integration approach was employed to aggregate patch-level predictions into slide-level outcomes. We evaluated the trained models using various metrics, including accuracy, precision, recall, F1 score, AUC (Area Under the Curve), and Cohen's Kappa coefficient. The tissue-type classification model exhibited robust performance, achieving an overall accuracy of 94.7%, a Cohen's Kappa coefficient of 94%, a precision of 95.3%, a recall of 99.76%, and an F1 score of 97.48% for identifying adenocarcinoma epithelium patches. Regarding the TP53 mutation prediction models, ResNet-101, DenseNet-201, and Inception-ResNet-v2 exhibit the predictive ability to both patch level (AUC=65%) and slide level (AUC=70%). In summary, the proposed deep learning pipeline offers an efficient way to acquire gene mutation information from WSIs. Moreover, it demonstrates a capability to discover potential molecular aberration from the clinical images. Citation Format: Geng-Yun Tien, Liang-Chuan Lai, Tzu-Pin Lu, Mong-Hsun Tsai, Hsiang-Han Chen, Eric Y. Chuang. Prediction of gene mutation from colorectal adenocarcinoma whole slide images via integrated deep learning pipeline [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2024; Part 1 (Regular Abstracts); 2024 Apr 5-10; San Diego, CA. Philadelphia (PA): AACR; Cancer Res 2024;84(6_Suppl):Abstract nr 4938.