Automated invasive ductal carcinoma detection based using deep transfer learning with whole-slide images

The tumor immune microenvironment (TIME) encompasses many heterogeneous cell types that engage in extensive crosstalk among the cancer, immune, and stromal components. The spatial organization of these different cell types in TIME could be used as biomarkers for predicting drug responses, prognosis and metastasis. Recently, deep learning approaches have been widely used for digital histopathology images for cancer diagnoses and prognoses. Furthermore, some recent approaches have attempted to integrate spatial and molecular omics data to better characterize the TIME. In this review we focus on machine learning-based digital histopathology image analysis methods for characterizing tumor ecosystem. In this review, we will consider three different scales of histopathological analyses that machine learning can operate within: whole slide image (WSI)-level, region of interest (ROI)-level, and cell-level. We will systematically review the various machine learning methods in these three scales with a focus on cell-level analysis. We will provide a perspective of workflow on generating cell-level training data sets using immunohistochemistry markers to “weakly-label” the cell types. We will describe some common steps in the workflow of preparing the data, as well as some limitations of this approach. Finally, we will discuss future opportunities of integrating molecular omics data with digital histopathology images for characterizing tumor ecosystem.

Pioneering effort has been made to facilitate the recognition of pathology in malignancies based on whole-slide images (WSIs) through deep learning approaches. It remains unclear whether we can accurately detect and locate basal cell carcinoma (BCC) using smartphone-captured images. To develop deep neural network frameworks for accurate BCC recognition and segmentation based on smartphone-captured microscopic ocular images (MOIs). We collected a total of 8046 MOIs, 6610 of which had binary classification labels and the other 1436 had pixelwise annotations. Meanwhile, 128 WSIs were collected for comparison. Two deep learning frameworks were created. The 'cascade' framework had a classification model for identifying hard cases (images with low prediction confidence) and a segmentation model for further in-depth analysis of the hard cases. The 'segmentation' framework directly segmented and classified all images. Sensitivity, specificity and area under the curve (AUC) were used to evaluate the overall performance of BCC recognition. The MOI- and WSI-based models achieved comparable AUCs around 0·95. The 'cascade' framework achieved 0·93 sensitivity and 0·91 specificity. The 'segmentation' framework was more accurate but required more computational resources, achieving 0·97 sensitivity, 0·94 specificity and 0·987 AUC. The runtime of the 'segmentation' framework was 15·3 ± 3·9 s per image, whereas the 'cascade' framework took 4·1 ± 1·4 s. Additionally, the 'segmentation' framework achieved 0·863 mean intersection over union. Based on the accessible MOIs via smartphone photography, we developed two deep learning frameworks for recognizing BCC pathology with high sensitivity and specificity. This work opens a new avenue for automatic BCC diagnosis in different clinical scenarios. What's already known about this topic? The diagnosis of basal cell carcinoma (BCC) is labour intensive due to the large number of images to be examined, especially when consecutive slide reading is needed in Mohs surgery. Deep learning approaches have demonstrated promising results on pathological image-related diagnostic tasks. Previous studies have focused on whole-slide images (WSIs) and leveraged classification on image patches for detecting and localizing breast cancer metastases. What does this study add? Instead of WSIs, microscopic ocular images (MOIs) photographed from microscope eyepieces using smartphone cameras were used to develop neural network models for recognizing BCC automatically. The MOI- and WSI-based models achieved comparable areas under the curve around 0·95. Two deep learning frameworks for recognizing BCC pathology were developed with high sensitivity and specificity. Recognizing BCC through a smartphone could be considered a future clinical choice.

Detection of Invasive Ductal Carcinoma (IDC) is a difficult, time taking and mandatory job. Moreover, the precise detection of IDC is important to give proper medical treatment to the patients. In this medical situation, Deep Learning (DL) approach is much reliable. The evolution of Deep Learning is taken off in the past few years due to a rise in more and more data, better hardware and it supports with robust computational modelling algorithms to train a framework. Based on the advantage of DL, a framework is introduced to automatically detect IDC in Breast Histopathology Images (BHI) of Breast Cancer (BC) either as a malign or benign. This ResNet is provided with labelled BHI data to extract common features and trained to classify the future images thus able to detect the presence of IDC in breast tissue lesion image. It takes less effort, gives better accuracy within less time. Thus our framework helps to speed up a pathologist's work and provides diagnosis support.

BackgroundTargeted therapy and immunotherapy put forward higher demands for accurate lung cancer classification, as well as benign versus malignant disease discrimination. Digital whole slide images (WSIs) witnessed the transition from traditional histopathology to computational approaches, arousing a hype of deep learning methods for histopathological analysis. We aimed at exploring the potential of deep learning models in the identification of lung cancer subtypes and cancer mimics from WSIs.MethodsWe initially obtained 741 WSIs from the First Affiliated Hospital of Sun Yat-sen University (SYSUFH) for the deep learning model development, optimization, and verification. Additional 318 WSIs from SYSUFH, 212 from Shenzhen People’s Hospital, and 422 from The Cancer Genome Atlas were further collected for multi-centre verification. EfficientNet-B5- and ResNet-50-based deep learning methods were developed and compared using the metrics of recall, precision, F1-score, and areas under the curve (AUCs). A threshold-based tumour-first aggregation approach was proposed and implemented for the label inferencing of WSIs with complex tissue components. Four pathologists of different levels from SYSUFH reviewed all the testing slides blindly, and the diagnosing results were used for quantitative comparisons with the best performing deep learning model.ResultsWe developed the first deep learning-based six-type classifier for histopathological WSI classification of lung adenocarcinoma, lung squamous cell carcinoma, small cell lung carcinoma, pulmonary tuberculosis, organizing pneumonia, and normal lung. The EfficientNet-B5-based model outperformed ResNet-50 and was selected as the backbone in the classifier. Tested on 1067 slides from four cohorts of different medical centres, AUCs of 0.970, 0.918, 0.963, and 0.978 were achieved, respectively. The classifier achieved high consistence to the ground truth and attending pathologists with high intraclass correlation coefficients over 0.873.ConclusionsMulti-cohort testing demonstrated our six-type classifier achieved consistent and comparable performance to experienced pathologists and gained advantages over other existing computational methods. The visualization of prediction heatmap improved the model interpretability intuitively. The classifier with the threshold-based tumour-first label inferencing method exhibited excellent accuracy and feasibility in classifying lung cancers and confused nonneoplastic tissues, indicating that deep learning can resolve complex multi-class tissue classification that conforms to real-world histopathological scenarios.

Automated detection and grading of Invasive Ductal Carcinoma breast cancer using ensemble of deep learning models

Breast cancer is one of the most common types of cancer in women, and histopathological imaging is considered the gold standard for its diagnosis. However, the great complexity of histopathological images and the considerable workload make this work extremely time-consuming, and the results may be affected by the subjectivity of the pathologist. Therefore, the development of an accurate, automated method for analysis of histopathological images is critical to this field. In this article, we propose a deep learning method guided by the attention mechanism for fast and effective classification of haematoxylin and eosin-stained breast biopsy images. First, this method takes advantage of DenseNet and uses the feature map's information. Second, we introduce dilated convolution to produce a larger receptive field. Finally, spatial attention and channel attention are used to guide the extraction of the most useful visual features. With the use of fivefold cross-validation, the best model obtained an accuracy of 96.47% on the BACH2018 dataset. We also evaluated our method on other datasets, and the experimental results demonstrated that our model has reliable performance. This study indicates that our histopathological image classifier with a soft attention-guided deep learning model for breast cancer shows significantly better results than the latest methods. It has great potential as an effective tool for automatic evaluation of digital histopathological microscopic images for computer-aided diagnosis.

Artificial intelligence (AI) based computer-aided diagnostic (CAD) systems can effectively diagnose critical disease. AI-based detection of breast cancer (BC) through images data is more efficient and accurate than professional radiologists. However, the existing AI-based BC diagnosis methods have complexity in low prediction accuracy and high computation time. Due to these reasons, medical professionals are not employing the current proposed techniques in E-Healthcare to effectively diagnose the BC. To diagnose the breast cancer effectively need to incorporate advanced AI techniques based methods in diagnosis process. In this work, we proposed a deep learning based diagnosis method (StackBC) to detect breast cancer in the early stage for effective treatment and recovery. In particular, we have incorporated deep learning models including Convolutional neural network (CNN), Long short term memory (LSTM), and Gated recurrent unit (GRU) for the classification of Invasive Ductal Carcinoma (IDC). Additionally, data augmentation and transfer learning techniques have been incorporated for data set balancing and for effective training the model. To further improve the predictive performance of model we used stacking technique. Among the three base classifiers (CNN, LSTM, GRU) the predictive performance of GRU are better as compared to individual model. The GRU is selected as a meta classifier to distinguish between Non-IDC and IDC breast images. The method Hold-Out has been incorporated and the data set is split into 90% and 10% for training and testing of the model, respectively. Model evaluation metrics have been computed for model performance evaluation. To analyze the efficacy of the model, we have used breast histology images data set. Our experimental results demonstrated that the proposed StackBC method achieved improved performance by gaining 99.02% accuracy and 100% area under the receiver operating characteristics curve (AUC-ROC) compared to state-of-the-art methods. Due to the high performance of the proposed method, we recommend it for early recognition of breast cancer in E-Healthcare.

The COVID-19 pandemic has highlighted the need for a robust medical infrastructure and crisis management strategy as part of smart-city applications, with technology playing a crucial role. The Internet of Things (IoT) has emerged as a promising solution, leveraging sensor arrays, wireless communication networks, and artificial intelligence (AI)-driven decision-making. Advancements in edge computing (EC), deep learning (DL), and deep transfer learning (DTL) have made IoT more effective in healthcare and pandemic-resilient infrastructures. DL architectures are particularly suitable for integration into a pandemic-compliant medical infrastructures when combined with medically oriented IoT setups. The development of an intelligent pandemic-compliant infrastructure requires combining IoT, edge and cloud computing, image processing, and AI tools to monitor adherence to social distancing norms, mask-wearing protocols, and contact tracing. The proliferation of 4G and beyond systems including 5G wireless communication has enabled ultra-wide broadband data-transfer and efficient information processing, with high reliability and low latency, thereby enabling seamless medical support as part of smart-city applications. Such setups are designed to be ever-ready to deal with virus-triggered pandemic-like medical emergencies. This study presents a pandemic-compliant mechanism leveraging IoT optimized for healthcare applications, edge and cloud computing frameworks, and a suite of DL tools. The framework uses a composite attention-driven framework incorporating various DL pre-trained models (DPTMs) for protocol adherence and contact tracing, and can detect certain cyber-attacks when interfaced with public networks. The results confirm the effectiveness of the proposed methodologies.

Coal rock fractures play a vital role as key channels for gas migration and storage in the exploration and development of coalbed methane. The characteristics of these fractures, such as porosity distribution, specific surface area, and connectivity, directly impact the adsorption, diffusion, and permeability behaviors of coal reservoirs. Therefore, a detailed analysis of coal rock fracture characteristics is vital in assessing the extractability of coalbed methane and during its exploration and development process. Currently, techniques like imaging logging, seismic detection, and micro-CT are employed for characterizing and studying fractures at various scales. Among these, high-resolution non-destructive CT scanning technology, by constructing three-dimensional data models of core samples, enables a quantitative analysis of fracture distribution features and size parameters. Traditional digital image processing methods, such as threshold segmentation and binary segmentation, though advantageous in computational speed, have limitations in the accuracy of fracture identification. With the advancement of artificial intelligence technology, deep learning-based image processing techniques have increasingly become significant tools for fracture detection due to their powerful feature extraction capabilities. This study conducted micrometer-level CT scanning experiments on coal rock samples, using CT image processing technology to construct a digitalized coal rock core model. Deep learning methods were applied for precise fracture identification, establishing an appropriate deep learning architecture and optimized parameters for coal rock fracture recognition. Furthermore, this paper provides a detailed quantitative characterization of the precisely identified fractures, offering robust technical support for the exploration and development of coalbed methane. The high-resolution non-destructive CT scanning technology and three-dimensional fracture network reconstruction method used in this study have been proven to be effective tools for investigating the micro-features of rocks, enabling the three-dimensional visualization and detailed characterization of rock microstructures.

Breast cancer is the most prevalent cancer globally and presents a significant threat to human health. The low survival rate for advanced stages highlights the importance of early and accurate diagnosis. Pathology images, providing insights into cancer's cellular aggressiveness, are crucial for breast cancer diagnosis. Advances in deep learning have transformed automated medical image diagnosis. However, traditional deep learning methods, reliant on large labeled datasets, face challenges with the typically small size of medical image datasets. To overcome this, our study combines deep transfer learning (DTL) with convolutional neural networks(CNN). We utilize and enhance a ResNet50 model pretrained on a large dataset. Additionally, we apply four different deep transfer learning methods to feature extraction and classification of the BreakHis dataset, aiming to determine if breast cancer is benign or malignant. Ultimately, our system achieved the best average accuracy rates under four different magnifications: 98.57%, 96.83%, 97.76%, and 97.16%. These results demonstrate the accuracy and potential of deep transfer learning in classifying breast cancer histopathological images and its promising application in tumor prediction.

Convolutional neural networks (CNNs) are state-of-the-art computer vision techniques for various tasks, particularly for image classification. However, there are domains where the training of classification models that generalize on several datasets is still an open challenge because of the highly heterogeneous data and the lack of large datasets with local annotations of the regions of interest, such as histopathology image analysis. Histopathology concerns the microscopic analysis of tissue specimens processed in glass slides to identify diseases such as cancer. Digital pathology concerns the acquisition, management and automatic analysis of digitized histopathology images that are large, having in the order of 100′0002 pixels per image. Digital histopathology images are highly heterogeneous due to the variability of the image acquisition procedures. Creating locally labeled regions (required for the training) is time-consuming and often expensive in the medical field, as physicians usually have to annotate the data. Despite the advances in deep learning, leveraging strongly and weakly annotated datasets to train classification models is still an unsolved problem, mainly when data are very heterogeneous. Large amounts of data are needed to create models that generalize well. This paper presents a novel approach to train CNNs that generalize to heterogeneous datasets originating from various sources and without local annotations. The data analysis pipeline targets Gleason grading on prostate images and includes two models in sequence, following a teacher/student training paradigm. The teacher model (a high-capacity neural network) automatically annotates a set of pseudo-labeled patches used to train the student model (a smaller network). The two models are trained with two different teacher/student approaches: semi-supervised learning and semi-weekly supervised learning. For each of the two approaches, three student training variants are presented. The baseline is provided by training the student model only with the strongly annotated data. Classification performance is evaluated on the student model at the patch level (using the local annotations of the Tissue Micro-Arrays Zurich dataset) and at the global level (using the TCGA-PRAD, The Cancer Genome Atlas-PRostate ADenocarcinoma, whole slide image Gleason score). The teacher/student paradigm allows the models to better generalize on both datasets, despite the inter-dataset heterogeneity and the small number of local annotations used. The classification performance is improved both at the patch-level (up to κ=0.6127±0.0133 from κ=0.5667±0.0285), at the TMA core-level (Gleason score) (up to κ=0.7645±0.0231 from κ=0.7186±0.0306) and at the WSI-level (Gleason score) (up to κ=0.4529±0.0512 from κ=0.2293±0.1350). The results show that with the teacher/student paradigm, it is possible to train models that generalize on datasets from entirely different sources, despite the inter-dataset heterogeneity and the lack of large datasets with local annotations.

Breast cancer is one of the most common cancers occurring in women and is the main cause of mortality among women aged 20 to 59. Detection of breast cancer in the early stage increases survival rate to 80%, and the rate decreases to around 40% if diagnosed in its later stage. There exists significant need to diagnose breast cancer in its early stage itself among women suspected with the disease. The hematoxylin and eosin (H&E)–stained histopathology tissue samples taken by pathologists from suspected breast areas are viewed with whole slide imaging (WSI) and can be stored as digital histopathology images. Manual analysis of histopathology images is time-consuming and leads to inter- and intraobserver variability. Hence, automated classification methods are in research focus. Breast tumors are classified as benign and malignant tumors, which are non-cancerous and cancerous tumors, respectively. Currently, deep learning through convolutional neural networks is an emerging technology and has proven to be successful for image segmentation and classification applications. These deep CNNs require large labeled datasets, which are lacking in the field of medicine, especially for breast cancer images. Also, high computational power, time, and memory are required if CNN is to be trained from scratch. To overcome this, transfer learning from natural images like ImageNet, comprising of 1.2 million natural images belonging to 1,000 classes, is used. Pretrained models like VGG16, ResNet50, Inception, AlexNet, and DenseNet are employed for transfer learning, and fine-tuning is to be performed for refinement. In the proposed method, four different network architectures, namely, AlexNet, VGG16, Inception v3, and DenseNet121, are used for classification of histopathological images using transfer learning. The top dense layers of these architectures are removed, and custom dense layers are added for classification. Also, the weights of the last two layers are fine-tuned with the histopathology input images dataset. The publicly available breast cancer hisopathology images dataset BreakHis is used for validation of the proposed method. The experimental results prove that transfer learning–based CNN gives better classification accuracy compared to networks learned from scratch. Also, it is inferred from the experimental results that the pretrained model of DenseNet121 gives the highest classification accuracy of 0.9520 compared to the other models and state-of-the-art methods.

Direct Gene Expression Profile Prediction for Uveal Melanoma from Digital Cytopathology Images via Deep Learning and Salient Image Region Identification

Electricity load forecasting is a crucial undertaking within all the deregulated markets globally. Among the research challenges on a global scale, the investigation of deep transfer learning (DTL) in the field of electricity load forecasting represents a fundamental effort that can inform artificial intelligence applications in general. In this paper, a comprehensive study is reported regarding day-ahead electricity load forecasting. For this purpose, three sequence-to-sequence (Seq2seq) deep learning (DL) models are used, namely the multilayer perceptron (MLP), the convolutional neural network (CNN) and the ensemble learning model (ELM), which consists of the weighted combination of the outputs of MLP and CNN models. Also, the study focuses on the development of different forecasting strategies based on DTL, emphasizing the way the datasets are trained and fine-tuned for higher forecasting accuracy. In order to implement the forecasting strategies using deep learning models, load datasets from three Greek islands, Rhodes, Lesvos, and Chios, are used. The main purpose is to apply DTL for day-ahead predictions (1–24 h) for each month of the year for the Chios dataset after training and fine-tuning the models using the datasets of the three islands in various combinations. Four DTL strategies are illustrated. In the first strategy (DTL Case 1), each of the three DL models is trained using only the Lesvos dataset, while fine-tuning is performed on the dataset of Chios island, in order to create day-ahead predictions for the Chios load. In the second strategy (DTL Case 2), data from both Lesvos and Rhodes concurrently are used for the DL model training period, and fine-tuning is performed on the data from Chios. The third DTL strategy (DTL Case 3) involves the training of the DL models using the Lesvos dataset, and the testing period is performed directly on the Chios dataset without fine-tuning. The fourth strategy is a multi-task deep learning (MTDL) approach, which has been extensively studied in recent years. In MTDL, the three DL models are trained simultaneously on all three datasets and the final predictions are made on the unknown part of the dataset of Chios. The results obtained demonstrate that DTL can be applied with high efficiency for day-ahead load forecasting. Specifically, DTL Case 1 and 2 outperformed MTDL in terms of load prediction accuracy. Regarding the DL models, all three exhibit very high prediction accuracy, especially in the two cases with fine-tuning. The ELM excels compared to the single models. More specifically, for conducting day-ahead predictions, it is concluded that the MLP model presents the best monthly forecasts with MAPE values of 6.24% and 6.01% for the first two cases, the CNN model presents the best monthly forecasts with MAPE values of 5.57% and 5.60%, respectively, and the ELM model achieves the best monthly forecasts with MAPE values of 5.29% and 5.31%, respectively, indicating the very high accuracy it can achieve.

Deep learning has become widely used in image analysis. Transfer learning can make use of information from other datasets for the analysis of the chosen dataset. When there is a small number of images at hand, transfer learning using pre-trained models with coefficients already estimated from other datasets is recommended. This is in contrast to deep learning, where most model parameters are re-estimated. Deep transfer learning uses pre-trained models with fixed weight parameters in the lower layers; thus, deep learning can be viewed as a two-stage approach: (1) feature extraction from a lower neural network layer and (2) estimation of a neural network using the extracted features as inputs. Since deep transfer learning is feature extraction, we can extend the two-stage approach to a more general two-stage framework: (1) feature extraction using multiple methods and (2) machine learning methods taking extracted features as inputs. We evaluate the performance of methods with different Stage 1 and Stage 2 approaches in predicting the phenotype leaf numbers based on a multi-view plant imaging dataset. This paper contains an evaluation of different two-stage machine learning methods for multi-view image data in plant image phenotyping.