Deep learning in detecting tumor spread through air spaces

Vegetation cover classification using mixed or low-resolution scalar images is challenging. Fortunately, recently deep learning object detection methods have emerged as a replacement to the conventional machine learning methods for the detection and classification of land use and land cover. This paper presents a deep learning object detection approach for land use and land cover detection using low/mixed resolution satellite images acquired from Google Earth satellite images. Google Earth images are accessible freely using the Google Earth Pro desktop application. Our dataset consists of two (02) classes (vegetation and non-vegetation) with a total of 450 labeled images captured from different parts of Pakistan. We present a comparison of the recent anchor-free object detection model YOLOX with the anchor-based object detection model YOLOR for solving real-time problems. The end-to-end differentiability, efficient GPU utilization, and absence of hand-crafted parameters make anchor-free models a compelling choice in object detection, and yet not been explored on Land cover classification using satellite images. Our experimental study shows that YOLOX delivers an overall accuracy of 83.50% on Vegetation and 86% on Non-Vegetation classes, which outperformed YOLOR by 30% on Vegetation classes and 34% on non-Vegetation classes for our dataset. We also show how an object detection system can be used for Vegetation and Non-Vegetation classification tasks, which can then be used for change monitoring and assisting in developing geographical maps using low/mixed resolution freely available satellite images.

BackgroundTargeted therapy and immunotherapy put forward higher demands for accurate lung cancer classification, as well as benign versus malignant disease discrimination. Digital whole slide images (WSIs) witnessed the transition from traditional histopathology to computational approaches, arousing a hype of deep learning methods for histopathological analysis. We aimed at exploring the potential of deep learning models in the identification of lung cancer subtypes and cancer mimics from WSIs.MethodsWe initially obtained 741 WSIs from the First Affiliated Hospital of Sun Yat-sen University (SYSUFH) for the deep learning model development, optimization, and verification. Additional 318 WSIs from SYSUFH, 212 from Shenzhen People’s Hospital, and 422 from The Cancer Genome Atlas were further collected for multi-centre verification. EfficientNet-B5- and ResNet-50-based deep learning methods were developed and compared using the metrics of recall, precision, F1-score, and areas under the curve (AUCs). A threshold-based tumour-first aggregation approach was proposed and implemented for the label inferencing of WSIs with complex tissue components. Four pathologists of different levels from SYSUFH reviewed all the testing slides blindly, and the diagnosing results were used for quantitative comparisons with the best performing deep learning model.ResultsWe developed the first deep learning-based six-type classifier for histopathological WSI classification of lung adenocarcinoma, lung squamous cell carcinoma, small cell lung carcinoma, pulmonary tuberculosis, organizing pneumonia, and normal lung. The EfficientNet-B5-based model outperformed ResNet-50 and was selected as the backbone in the classifier. Tested on 1067 slides from four cohorts of different medical centres, AUCs of 0.970, 0.918, 0.963, and 0.978 were achieved, respectively. The classifier achieved high consistence to the ground truth and attending pathologists with high intraclass correlation coefficients over 0.873.ConclusionsMulti-cohort testing demonstrated our six-type classifier achieved consistent and comparable performance to experienced pathologists and gained advantages over other existing computational methods. The visualization of prediction heatmap improved the model interpretability intuitively. The classifier with the threshold-based tumour-first label inferencing method exhibited excellent accuracy and feasibility in classifying lung cancers and confused nonneoplastic tissues, indicating that deep learning can resolve complex multi-class tissue classification that conforms to real-world histopathological scenarios.

Purpose: This study employs cutting-edge deep learning techniques to comprehensively analyze hematoxylin and eosin-stained (H&E) whole slide images (WSIs) to predict which tumors carry driver gene alterations, mutational signatures, and additional molecular features among lung cancers from never-smokers (LCINS)— the seventh leading cause of cancer death worldwide. Method: A total of 464 H&E-stained WSIs of lung adenocarcinomas from the Sherlock-Lung study were utilized, with 325 WSIs for model training and 139 for testing. Alongside the WSIs, genomic data, including mutations (overall nonsynonymous mutations and driver mutations only), fusions, or copy numbers in driver genes (TP53, KRAS, EGFR, CDKN2A, MDM2, ALK, RBM10), mutational signatures (APOBEC), molecular features (Kataegies, Whole Genome Doubling (WGD) status, Tumor Mutational Burden (TMB)), and specific hotspot driver mutation in EGFR (p.L858R and p.E746_A750del) and KRAS (p.G12C, p.G12V, and p.G12D), were included for all tumors. Each WSI was assigned different labels based on the presence or absence of these genomic alterations or features. We used a customized multilabel binary deep learning model based on ResNet50, a residual convolutional neural network, for analyzing the data. The network was trained from scratch, with an initial epoch set to 100. Results: Our methodology demonstrated high predictive performance, measured by the area under the receiver operating characteristic (AUROC). In the presence or absence category, we achieved high AUROC for detecting any alterations in these driver genes: TP53 (0.81), KRAS (0.92), EGFR (0.93), CDKN2A deletion (0.88), MDM2 amplification (0.94), and ALK fusion (0.86). We also evaluated molecular features, obtaining AUROC scores of 0.79 for WGD status and 0.86 for Kataegies. Moderate AUROC scores were observed for tumors with RBM10 mutations (0.67), high TMB (0.64), and APOBEC signatures (0.56). When focusing on driver mutations only, we achieved high AUROC scores for EGFR (0.88), KRAS (0.84), TP53 (0.93), and RBM10 (0.78). In addition, our model successfully predicted tumors with specific mutations in EGFR (p.L858R = 0.79; p.E746_A750del = 0.77) and KRAS (p.G12C = 0.77). Performance was suboptimal for KRAS p.G12V (0.41) and KRAS p.G12D (0.47). Conclusions: Our deep learning network achieves high prediction scores in identifying tumors with critical driver gene alterations and actionable mutations, holding promise for potential clinical use. In the future, the model could be optimized as a screening assay to guide molecular testing and therapeutic management of patients with LCINS. Citation Format: Monjoy Saha, Tongwu Zhang, Praphulla Bhawsar, Wei Zhao, Jianxin Shi, Soo Ryum Yang, Jonas Almeida, Maria Teresa Landi. Deep learning-based molecular characterization of lung cancers from never smokers using hematoxylin and eosin-stained whole slide images [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2024; Part 2 (Late-Breaking, Clinical Trial, and Invited Abstracts); 2024 Apr 5-10; San Diego, CA. Philadelphia (PA): AACR; Cancer Res 2024;84(7_Suppl):Abstract nr LB243.

Breast cancer prognostication guides treatment decisions, with transcriptomic assays like Prosigna and Oncotype DX providing valuable risk of recurrence (ROR) scores in ER+/HER2– patients. However, these tests are costly and require sufficient biopsy tissue for accuracy. Pathology images are routinely available and could expedite rapid risk stratification at lower cost. We propose a multi-modal, multi-task deep neural network that learns transcriptomic ROR using digital pathology images and hormone receptor expression status. Because the incorporation of both pathology images and clinical data has proven beneficial in other deep learning models, we hypothesized that incorporation of this complementary information would improve ROR predictions compared to using pathology images alone. We modified the “clustering-constrained-attention multiple-instance learning” (CLAM) deep learning method to perform multi-task regression and infer continuous risk scores (instead of discrete risk categories). We evaluated (1) PAM50 ROR-P, calculated using a multivariate model based on intrinsic subtype centroids and a proliferation score, and (2) a 21-gene assay recurrence score (RS) (research version of Oncotype DX), calculated from component scores relating to proliferation, ER, HER2, and invasion. Using data from 899 patients in the Carolina Breast Cancer Study, a population-based study of diverse patients including 50% Black women and 50% women under age 50, we constructed training (n=714), validation (n=93), and test (n=92) sets. Genomic data were assayed by Nanostring, and digital pathology was based on H&E-stained whole slide images (WSIs). Models were assessed using Pearson correlation between measured and predicted ROR-P or RS. The model with the highest correlations on the test set used H&E WSIs and ER/PR/HER2 expression status as inputs and generated 10 independent outputs: ROR-P, four PAM50 centroid correlations (i.e., Basal, HER2-enriched, Luminal A, and Luminal B), RS, and four RS component scores. Prediction was strong when considering all participants in the test set (ROR-P, 0.78; RS, 0.83, N, 92) but was reduced among more clinically homogeneous ER+/HER2– patients (ROR-P, 0.55; RS, 0.49; N, 49). For RS, correlations were higher in models that included ER/PR/HER2 (Overall, 0.83; ER+/HER2–, 0.50) than in those that relied on pathology alone (Overall, 0.60, ER+/HER2–, 0.34). Future models will consider strategies to enhance performance in ER+/HER2– patients, such as oversampling of ER+/HER2– patients and use of loss functions designed to focus learning on mis-predicted examples. In summary, multi-task, tissue-based regression deep learning models recapitulate transcription-based risk assays with high correlations and, with optimization (especially integration of additional clinicopathologic data), hold promise for personalized treatment decisions from early, routinely-collected biopsy images. Citation Format: Jakub R Kaczmarzyk, Luke A Torre-Healy, Richard A Moffitt, Rajarsi Gupta, Alina M Hamilton, Tahsin M Kurc, Katherine A Hoadley, Melissa A Troester, Joel H Saltz. Early risk stratification of ER+/HER2– breast cancer patients using digital pathology and multi-task, weakly-supervised deep learning [abstract]. In: Proceedings of the AACR Special Conference in Cancer Research: Advances in Breast Cancer Research; 2023 Oct 19-22; San Diego, California. Philadelphia (PA): AACR; Cancer Res 2024;84(3 Suppl_1):Abstract nr B082.

Objective. Subtype classification plays a guiding role in the clinical diagnosis and treatment of non-small-cell lung cancer (NSCLC). However, due to the gigapixel of whole slide images (WSIs) and the absence of definitive morphological features, most automatic subtype classification methods for NSCLC require manually delineating the regions of interest (ROIs) on WSIs. Approach. In this paper, a weakly supervised framework is proposed for accurate subtype classification while freeing pathologists from pixel-level annotation. With respect to the characteristics of histopathological images, we design a two-stage structure with ROI localization and subtype classification. We first develop a method called multi-resolution expectation-maximization convolutional neural network (MR-EM-CNN) to locate ROIs for subsequent subtype classification. The EM algorithm is introduced to select the discriminative image patches for training a patch-wise network, with only WSI-wise labels available. A multi-resolution mechanism is designed for fine localization, similar to the coarse-to-fine process of manual pathological analysis. In the second stage, we build a novel hierarchical attention multi-scale network (HMS) for subtype classification. HMS can capture multi-scale features flexibly driven by the attention module and implement hierarchical features interaction. Results. Experimental results on the 1002-patient Cancer Genome Atlas dataset achieved an AUC of 0.9602 in the ROI localization and an AUC of 0.9671 for subtype classification. Significance. The proposed method shows superiority compared with other algorithms in the subtype classification of NSCLC. The proposed framework can also be extended to other classification tasks with WSIs.

1577 Background: Only a subset of non-small cell lung cancer (NSCLC) patients experiences durable benefit from immune checkpoint inhibitors (ICIs), and precise biomarkers remain scarce. Meanwhile, computational pathology, based on digital pathology, has led to significant advancements in NSCLC prognosis prediction. Yet limited generalization and interpretation remain critical challenges in current clinical practice. Self-supervised learning, pretrained on unlabeled data to comprehensively capture underlying biological information in pathological images, may enable expandable, interpretable pathomics models for outcome prediction of ICIs. Methods: H&E-stained slides from pan-cancer patients were digitized as whole slide images (WSIs), then segmented for preprocessing. A self-supervised foundation model (patho-GPT) were performed to extract WSI features, and further fine-tuned for outcome prediction of immunotherapy in NSCLC with progression-free survival (PFS) labels. Its performance was evaluated by accuracy, sensitivity, specificity, and AUC in internal and external validation. Patients were classified as immunotherapy-resistant (R) or immunotherapy-sensitive (S), and ROC/survival curves were generated. The model was also tested in an operable cohort on neoadjuvant immunotherapy. Finally, single-cell RNA (scRNA) sequencing analyses were performed to provide biological insights. Results: A total of 13770 whole slide images (WSIs) from 6589 patients were included to construct the self-supervised patho-GPT model, which utilizes a context encoder, target encoder, and predictor based on the Vision Transformer architecture. There were 771 WSIs from 511 NSCLC patients receiving immunotherapy, labeled using 5.23 months as the cut-off value. All patients were divided into training and validation sets at a 7:3 ratio. For downstream outcome prediction of immunotherapy, the weight of patho-GPT was fine-tuned in the training set, and performance was evaluated in internal and independent external validation sets. The accuracy was 0.828 (AUC 0.774) in the internal set and 0.758 (AUC 0.752) externally. Survival analyses showed the model’s risk group was significantly associated with survival after immunotherapy. In contrast, the ViT-ViT model using initial weights achieved 0.677 accuracy (AUC 0.547) in the external set, which was significantly inferior. ScRNA sequencing and differential analysis between R and S groups were performed, and a high level of H1.2hi Teffs was found in R, linked to dysfunction of cytotoxicity-related genes and immune pathways. Conclusions: The self-supervised patho-GPT can be used for the accurate prediction of immunotherapy outcome, with well generalization and biological interpretation.

Accurate prediction of the likelihood of recurrence is important in the selection of postoperative treatment for patients with early-stage breast cancer. In this study, we investigated whether deep learning algorithms can predict patients’ risk of recurrence by analyzing the pathology images of their cancer histology. We analyzed 125 hematoxylin and eosin-stained whole slide images (WSIs) from 125 patients across two institutions (National Cancer Center and Korea University Medical Center Guro Hospital) to predict breast cancer recurrence risk using deep learning. Sensitivity reached 0.857, 0.746, and 0.529 for low, intermediate, and high-risk categories, respectively, with specificity of 0.816, 0.803, and 0.972, and a Pearson correlation of 0.61 with histological grade. Class activation maps highlighted features like tubule formation and mitotic rate, suggesting a cost-effective approach to risk stratification, pending broader validation. These findings suggest that deep learning models trained exclusively on hematoxylin and eosin stained whole slide images can approximate genomic assay results, offering a cost-effective and scalable tool for breast cancer recurrence risk assessment. However, further validation using larger and more balanced datasets is needed to confirm the clinical applicability of our approach.

Deep learning has received increasing attention in recent years and is used in many different areas. Since image analysis is a strength of deep learning, it would be obvious to use it for histopathological questions too. Our goal is to identify possible deep learning approaches from general pathology which could be used in ophthalmic pathology. In addition, the data of the past year will be used to estimate the proportion of potentially interesting cases and the necessary technical effort. Firstly, a literature search for deep learning models and their possible applications in the field of pathology was carried out. In order to estimate the potential benefit, technical challenges and feasibility, the number of suitable ophthalmopathology cases in our lab in 2019 for the identified models was determined and put in relation to the resulting amount of data and the scanning time. We identified 7 areas of particular interest: determination of regions of interest (ROI), classification of histological images in scoring systems, mapping of tumor fractions, differentiation of different types of inflammation, differentiation of various cutaneous tumors, classification of lymphomas and prediction of patient outcome-based on tumor histology. Within one year, a total of 831 cases (43%) would have been suitable for the above models. The creation of whole slide images (WSI) for all histological cases would have required a storage capacity of 630 GB with a scanning time of 35 h. There are several deep learning approaches which are also interesting for ophthalmic pathology. Most of them would have to be specially trained for the ophthalmopathological aspects. To be able to apply deep learning approaches, it is necessary to have a good IT infrastructure with the possibility to create and permanently store WSI, and this seems to be technically feasible. Future studies should focus on the specific practical implementation of current deep learning possibilities for ophthalmic pathology.

The analysis of digital pathology images, catalyzes research and automate diagnosis for improving clinical care. Regardless of the advances in high-resolution and speedy scanning devices, the clinical analysis of Whole Slide Image (WSI) remains mainly as the work of an image analyst. The complications when working on these images are primarily, its enormous image size, as well as the heterogeneous nature of tissue regions. These reasons may lead to inefficient computer analysis. We propose a comprehensive algorithm to analyse, these large images by using small tile image analysis. The small tile images can be learnt individually in detail, rather than processing the whole high resolution WSI. The analysis of these small tile images helps us configure, the localised pathology image characteristics. The work flow involves extracting Perceptual Features (PF), Fourier Local Binary Patterns (FLBP), Histogram, Gabor and Gray Level Co-occurrence Matrix (GLCM) features from the tiled WSI. Then, the multidimensional feature set is provided to the classifiers like K-Nearest Neighbour (KNN), Support Vector Machine (SVM), Naive Bayes (NB) and Logistic Regression (LR) for performance analysis. We evaluated our predictive models by automatically classifying image datasets (186 and 124) of brain tumour cases, using 10 fold cross validation. The model classified the images into Low Grade Glioma (LGG) and Glioblastoma Multiforme (GBM). It gives the highest accuracy of 93.5% for a feature combination of PF+FLBP+GLCM+GABOR with linear Support Vector Machine (linSVM) and Area under the Curve (AUC) of 0.9741 for the 186 image dataset. Also linSVM gives an accuracy of 96.8% with various feature combinations for the 124 image dataset. We did an exhaustive study on the performance of various sets of texture descriptors and classification models. This resulted in finding an optimised classification approach with a combined feature set that outperformed the original feature set classification. The proposed work is unique because it combines a wide range of feature extracting methods and an exhaustive list of classifiers showcasing the best method. The idea that, rather than selecting a region of interest from WSI for grade designation, all the regions in a slide are considered. This enables the involvement of each tile in making the final diagnostic decision, considering the heterogeneous nature of WSI. Our work can be extended for different tumour subtypes. Also, the experiment can be scaled with different sample sizes and finally the method works without a pathologist input.

Combining big data and deep learning can analyze large-scale breast cancer pathology images for auxiliary diagnosis. Furthermore, Whole Slide Images (WSIs) of breast cancer pathology offer detailed tissue feature information, which supports the accurate identification of malignant lesions. Current approaches combine Self-Supervised Learning (SSL) and Multiple Instance Learning (MIL) for WSI analysis, aiming to address the issues of billion-level pixels in a single WSI and the lack of precise annotations. However, pseudo-labels produced by SSL frequently lack accuracy, and MIL fails to effectively integrate global information at the WSI level, resulting in performance bottlenecks. This paper proposes the Multi-granularity Self-supervised Second-order MIL (MS2M) to tackle these issues. MS2M first achieves instance-level fine-grained feature learning through multi-granularity SSL and optimizes instance-level representations using bag-level labels within the MIL framework. Then, the transformer captures long-range dependencies between instances. When combined with second-order (covariance) pooling, it also captures high-order relational information. This process generates a robust bag-level representation. MS2M achieves accuracies of 0.9845 and 0.9719 on the CAMELYON16 and private breast cancer WSI datasets, respectively, outperforming existing methods.

Recent advances in programmed death-1 (PD-1) and programmed death-ligand 1 (PD-L1) immune checkpoint inhibitor (ICI) revolutionized the clinical practice in lung cancer treatment. The PD-L1 immunohistochemistry (IHC) test is a widely used biomarker for ICI responder in lung cancer. However, the accuracy of PD-L1 IHC test for selecting ICI responder is unsatisfactory, especially due to its low specificity. Therefore, methods which could effectively predict the efficacy of ICI are crucial for patient selection. In this article, we apply a deep neural network (DNN) to predict responders for anti-PD1 blockade on the basis of histopathological images of hematoxylin and eosin (H&E) stained tissue. The main difficulty to train DNN for responder prediction is the inability to accurately label at the image patch level. We employed a semi-supervised multi-instance learning (MIL) framework with adaptive positive patch selection in each region-of-interest (ROI). Using a dataset of 250 whole slide images (WSIs) of non-small cell lung cancer (111 responders and 139 nonresponders), we train a DNN-based MIL classifier on a case-level partition of the dataset (150 WSIs) and obtain an area under curve (AUC) of 0.773 for the test dataset (50 WSIs), which outperforms the PD-L1 IHC test (AUC=0.636). These results suggest that a DNN model can be used for assisting clinicians to make decision for treatment plan. We also confirm that the locations of adaptively selected positive patches can give valuable insights into the histological features associated with drug response.

Deep learning is a powerful tool for whole slide image (WSI) analysis. Typically, when performing supervised deep learning, a WSI is divided into small patches, trained and the outcomes are aggregated to estimate disease grade. However, patch-based methods introduce label noise during training by assuming that each patch is independent with the same label as the WSI and neglect overall WSI-level information that is significant in disease grading. Here we present a Graph-Transformer (GT) that fuses a graph-based representation of an WSI and a vision transformer for processing pathology images, called GTP, to predict disease grade. We selected 4,818 WSIs from the Clinical Proteomic Tumor Analysis Consortium (CPTAC), the National Lung Screening Trial (NLST), and The Cancer Genome Atlas (TCGA), and used GTP to distinguish adenocarcinoma (LUAD) and squamous cell carcinoma (LSCC) from adjacent non-cancerous tissue (normal). First, using NLST data, we developed a contrastive learning framework to generate a feature extractor. This allowed us to compute feature vectors of individual WSI patches, which were used to represent the nodes of the graph followed by construction of the GTP framework. Our model trained on the CPTAC data achieved consistently high performance on three-label classification (normal versus LUAD versus LSCC: mean accuracy = 91.2 ± 2.5%) based on five-fold cross-validation, and mean accuracy = 82.3 ± 1.0% on external test data (TCGA). We also introduced a graph-based saliency mapping technique, called GraphCAM, that can identify regions that are highly associated with the class label. Our findings demonstrate GTP as an interpretable and effective deep learning framework for WSI-level classification.

Simple SummaryFor the effective treatment of lung cancer patients, correct tumor subtyping is of utmost importance, but it is often challenging in clinical routine. Using artificial intelligence and combining information from digital microscopy and matrix-assisted laser desorption/ionization mass spectrometry imaging data have the potential to support the pathologist’s decision-making process. We present a classification algorithm to distinguish between adenocarcinoma and squamous cell carcinoma of the lung based on the automatic detection of tumor areas in whole tissue sections and the determination of the tumor subtype with high accuracy.Artificial intelligence (AI) has shown potential for facilitating the detection and classification of tumors. In patients with non-small cell lung cancer, distinguishing between the most common subtypes, adenocarcinoma (ADC) and squamous cell carcinoma (SqCC), is crucial for the development of an effective treatment plan. This task, however, may still present challenges in clinical routine. We propose a two-modality, AI-based classification algorithm to detect and subtype tumor areas, which combines information from matrix-assisted laser desorption/ionization (MALDI) mass spectrometry imaging (MSI) data and digital microscopy whole slide images (WSIs) of lung tissue sections. The method consists of first detecting areas with high tumor cell content by performing a segmentation of the hematoxylin and eosin-stained (H&E-stained) WSIs, and subsequently classifying the tumor areas based on the corresponding MALDI MSI data. We trained the algorithm on six tissue microarrays (TMAs) with tumor samples from N = 232 patients and used 14 additional whole sections for validation and model selection. Classification accuracy was evaluated on a test dataset with another 16 whole sections. The algorithm accurately detected and classified tumor areas, yielding a test accuracy of 94.7% on spectrum level, and correctly classified 15 of 16 test sections. When an additional quality control criterion was introduced, a 100% test accuracy was achieved on sections that passed the quality control (14 of 16). The presented method provides a step further towards the inclusion of AI and MALDI MSI data into clinical routine and has the potential to reduce the pathologist’s work load. A careful analysis of the results revealed specific challenges to be considered when training neural networks on data from lung cancer tissue.

Pathology report generation from whole slide images with knowledge retrieval and multi-level regional feature selection.

Lung cancer management challenges amidst COVID-19 pandemic: hope lives here.