Explainable Deep Learning Methods in Medical Image Classification: A Survey

Due to the complexity of medical images, traditional medical image classification methods have been unable to meet the actual application needs. In recent years, the rapid development of deep learning theory has provided a technical approach for solving medical image classification. However, deep learning has the following problems in the application of medical image classification. First, it is impossible to construct a deep learning model with excellent performance according to the characteristics of medical images. Second, the current deep learning network structure and training strategies are less adaptable to medical images. Therefore, this paper first introduces the visual attention mechanism into the deep learning model so that the information can be extracted more effectively according to the problem of medical images, and the reasoning is realized at a finer granularity. It can increase the interpretability of the model. Additionally, to solve the problem of matching the deep learning network structure and training strategy to medical images, this paper will construct a novel multiscale convolutional neural network model that can automatically extract high-level discriminative appearance features from the original image, and the loss function uses the Mahalanobis distance optimization model to obtain a better training strategy, which can improve the robust performance of the network model. The medical image classification task is completed by the above method. Based on the above ideas, this paper proposes a medical classification algorithm based on a visual attention mechanism-multiscale convolutional neural network. The lung nodules and breast cancer images were classified by the method in this paper. The experimental results show that the accuracy of medical image classification in this paper is not only higher than that of traditional machine learning methods but also improved compared with other deep learning methods, and the method has good stability and robustness.

Due to the complexity of medical images, traditional medical image classification methods have been unable to meet actual application needs. In recent years, the rapid development of deep learning theory has provided a technical approach for solving medical image classification tasks. However, deep learning has the following problems in medical image classification. First, it is impossible to construct a deep learning model hierarchy for medical image properties; second, the network initialization weights of deep learning models are not well optimized. Therefore, this paper starts from the perspective of network optimization and improves the nonlinear modeling ability of the network through optimization methods. A new network weight initialization method is proposed, which alleviates the problem that existing deep learning model initialization is limited by the type of the nonlinear unit adopted and increases the potential of the neural network to handle different visual tasks. Moreover, through an in‐depth study of the multicolumn convolutional neural network framework, this paper finds that the number of features and the convolution kernel size at different levels of the convolutional neural network are different. In contrast, the proposed method can construct different convolutional neural network models that adapt better to the characteristics of the medical images of interest and thus can better train the resulting heterogeneous multicolumn convolutional neural networks. Finally, using the adaptive sliding window fusion mechanism proposed in this paper, both methods jointly complete the classification task of medical images. Based on the above ideas, this paper proposes a medical classification algorithm based on a weight initialization/sliding window fusion for multilevel convolutional neural networks. The methods proposed in this study were applied to breast mass, brain tumor tissue, and medical image database classification experiments. The results show that the proposed method not only achieves a higher average accuracy than that of traditional machine learning and other deep learning methods but also is more stable and more robust.

Area under the curve may hide poor generalisation to external datasets

Deep Learning applications have recently emerged for processing and analyzing medical image content. DICOM files are the raw data or the input data for Deep Learning (DL) models used to achieve various tasks such as segmentation, classification, and detection in medical diagnosis. However, such files cannot be used as specific devices produce them; they need multiple pre-processing actions due to the complexity of the problem definition when designing a DL model for medical image research. This paper introduces the innovative PreProcMed framework for data curation, medical image processing, and feature exploration. PreProcMed framework, a pioneering solution, has innovative features chained in an automated workflow using only Python technology, namely i). a new approach in data curation by automated de-identification and anonymization with the option of ii). Selection of the required MRI sequence for the DL model, followed by iii). automated conversion of 2D images in 3D volumes that are easy to use in segmentation models and fully iv). integrates annotation tools like ITKSnap and visualization tools like 3D Slicer. PreProcMed is a modular and flexible framework ready to be modified and adapted according to DL models used for medical imaging research.

Deep learning-based diagnosis of disc degenerative diseases using MRI: A comprehensive review

Colorectal cancer has a high incidence rate in all countries around the world, and the survival rate of patients is improved by early detection. With the development of object detection technology based on deep learning, computer-aided diagnosis of colonoscopy medical images becomes a reality, which can effectively reduce the occurrence of missed diagnosis and misdiagnosis. In medical image recognition, the assumption that training samples follow independent identical distribution (IID) is the key to the high accuracy of deep learning. However, the classification of medical images is unbalanced in most cases. This paper proposes a new loss function named the double-balanced loss function for the deep learning model, to improve the impact of datasets on classification accuracy. It introduces the effects of sample size and sample difficulty to the loss calculation and deals with both sample size imbalance and sample difficulty imbalance. And it combines with deep learning to build the medical diagnosis model for colorectal cancer. Experimentally verified by three colorectal white-light endoscopic image datasets, the double-balanced loss function proposed in this paper has better performance on the imbalance classification problem of colorectal medical images.

Industrial Internet of Things (IIoT) refers to the extension and utilization of the Internet of Things (IoT) in industrial sectors and applications. Medical image fusion and classification has been utilized to get valuable significant multimodality medical image data. Recently, deep learning methods offer an effective way to design an end-to-end model that can determine the final classification labels. This chapter introduces a Multimodality Image Fusion Classification (MMIFC) by the incorporation of image fusion, feature extraction, and classification techniques. Initially, the input medical images were fused by the use of optimal shearlet transform where the coefficients of shearlets are optimized by the use of Enhanced Monarch Butterfly Optimization (EMBO) algorithm. In the next stage, the fused images were classified based on the feature extractor and deep learning model to check the test image is malignant and benign. To further optimize the performance of the deep learning model, hyperparameter tuning process takes place by the use of Bayesian optimization model to optimize the weights of structure. Finally, this ODNN model classifies the MMFI as class 1 or 0. A detailed simulation result takes place to ensure the effective performance of the proposed method. From the experimental results, the proposed strategy accomplishes better fusion rate and classification results compared with other supervised and non-supervised learning techniques.

The rapid development of medical practices and imaging technology tools creates substantial growth in the amount of medical image data each year in our present era. This research aims to develop a hybrid approach that integrates Machine Learning (ML) and Deep Learning (DL) techniques to enhance the accuracy and reliability of medical image classification for diagnostic purposes. Medical imaging data complexity and growing volume serve as the research motivation, which leads to an investigation of standalone ML or DL limitations and their combination into a single framework. The medical image processing starts with normalization, then noise reduction, and continues to grayscale conversion before performing histogram equalization. This research uses VGG16 and ResNet50 alongside MobileNet and InceptionV3 for feature extraction, then applies ten different ML algorithms, including SVM and MLP, and Ran-dom Forest, for classification. Five public medical image datasets from Kaggle are used: COVID-19 chest X-rays, melanoma skin lesions, pneumonia chest X-rays, acute stroke facial images, and various eye diseases. Hybrid models display superior performance compared to stand-alone ML or DL models based on accuracy, precision, recall, and F1-score evaluation measures. Multiple datasets demonstrate that the MobileNet+MLP combination de-livers the most accurate results, which demonstrates its reliable and efficient performance. The developed AI diagnostic tool presents a scalable system alongside accuracy and interpretability to enhance clinical decision outcomes.

Deep Learning has emerged as a strategic area in medical image diagnosis, with numerous studies exploring automation in this field. However, many existing approaches attempt to fully replace medical professionals by relying exclusively on model predictions. This thesis presents an alternative strategy, proposing Deep Learning architectures designed to support, rather than replace, physicians in diagnostic tasks. Two distinct methodologies are explored to achieve this goal: Content-Based Image Retrieval and eXplainable Artificial Intelligence. Content-Based Image Retrieval system enables physicians to compare complex cases with similar ones from a curated database, enhancing decision-making through feature-based similarities. The eXplainable Artificial Intelligence approach provides visual explanations through heatmaps, indicating the regions of interest detected by the model. The thesis introduces two novel architecturesMulti-Output Classification Autoencoder (MOC-AE) and Multi-Output Classification Variational Autoencoder (MOC-VAE). These models employ multi-objective training to generate rich image descriptors, capturing both pathological and anatomical information. These descriptors are then used to identify visually similar cases in a low-dimensional space. To address the challenge of quantifying visual similarity in Content-Based Image Retrieval, the Sliced Wasserstein distance is proposed, offering a robust metric for comparative diagnosis. Furthermore, the thesis demonstrates the utility of these architectures in generating precise, non-linear explanations for diagnoses, leveraging the factorization process inherent to the models. The simplicity and enhanced performance of the proposed approaches, compared to prior methods, offer a foundation for future research and real-world applications in medical diagnostics using Deep Learning. RESUMEN El Deep Learning ha surgido como un área estratégica en el diagnóstico de imágenes médicas, con numerosos estudios que exploran la automatización en este campo. Sin embargo, muchos enfoques actuales buscan reemplazar por completo a los profesionales médicos basándose únicamente en las predicciones de los modelos. Esta tesis presenta una estrategia alternativa, proponiendo arquitecturas de Deep Learning diseñadas para apoyar, en lugar de reemplazar, a los médicos en tareas diagnósticas. Se exploran dos metodologías distintas para alcanzar este objetivo: Sistemas de Recuperación de Imágenes Basada en Contenido e Inteligencia Artificial Explicable. Los Sistemas de Recuperación de Imágenes Basada en Contenido permiten a los médicos comparar casos complejos con otros similares de una base de datos curada, mejorando la toma de decisiones mediante similitudes basadas en características. El enfoque de la Inteligencia Artificial Explicable proporciona explicaciones visuales a través de mapas de calor, que indican las regiones de interés detectadas por el modelo. La tesis introduce dos arquitecturas novedosas: el Autoencoder de Clasificación Multi-Salida (MOC-AE) y el Autoencoder Variacional de Clasificación Multi-Salida (MOC-VAE). Estos modelos emplean un entrenamiento multiobjetivo para generar descriptores de imágenes enriquecidos que capturan tanto información patológica como anatómica. Estos descriptores se utilizan para identificar casos visualmente similares en un espacio de baja dimensionalidad. Para abordar el desafío de cuantificar la similitud visual en Sistemas de Recuperación de Imágenes Basada en Contenido, se propone la distancia de Wasserstein por secciones, ofreciendo una métrica robusta para el diagnóstico comparativo. Además, la tesis demuestra la utilidad de estas arquitecturas para generar explicaciones precisas y no lineales de los diagnósticos, aprovechando el proceso de factorización inherente a los modelos. La simplicidad y el rendimiento mejorado de los enfoques propuestos, en comparación con métodos anteriores, ofrecen una base sólida para futuras investigaciones y aplicaciones reales en diagnósticos médicos utilizando Deep Learning.

Recent advancements in Deep Learning (DL) methodologies have led to state-of-the-art performance in a wide range of applications especially in object recognition, classification, and segmentation of medical images. However, training modern DL models requires a large amount of computation and long training times due to the complex nature of network structures and the large number of training datasets involved. Moreover, it is an intensive, repetitive manual process to select the optimized configuration of hyperparameters for a given DL network. In this study, we present a novel approach to accelerate the training time of DL models via the progressive feeding of training datasets based on similarity measures for medical image segmentation. We term this approach Progressive Deep Learning (PDL). The two-stage PDL approach was tested on the auto-segmentation task for two imaging modalities: CT and MRI. The training datasets were ranked according to similarity measures between each sample based on Mean Square Error (MSE), Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM), and the Universal Quality Image Index (UQI) values. At the start of the training process, a relatively coarse sampling of training datasets with higher ranks was used to optimize the hyperparameters of the DL network. Following this, the samples with higher ranks were used in step 1 to yield accelerated loss minimization in early training epochs and the total dataset was added in step 2 for the remainder of training. Our results demonstrate that the PDL approach can reduce the training time by nearly half (∼49%) and can predict segmentations (CT U-net/DenseNet dice coefficient: 0.9506/0.9508, MR U-net/DenseNet dice coefficient: 0.9508/0.9510) without major statistical difference (Wilcoxon signed-rank test) compared to the conventional DL approach. The total training times with a fixed cutoff at 0.95 DSC for the CT dataset using DenseNet and U-Net architectures, respectively, were 17h, 20 min and 4h, 45 min in the conventional case compared to 8h, 45 min and 2h, 20 min with PDL. For the MRI dataset, the total training times using the same architectures were 2h, 54 min and 52 min in the conventional case and 1h, 14 min and 25 min with PDL. The proposed PDL training approach offers the ability to substantially reduce the training time for medical image segmentation while maintaining the performance achieved in the conventional case.

Explainable deep learning diagnostic system for prediction of lung disease from medical images

COVID-19 is highly infectious and causes acute respiratory disease. Machine learning (ML) and deep learning (DL) models are vital in detecting disease from computerized chest tomography (CT) scans. The DL models outperformed the ML models. For COVID-19 detection from CT scan images, DL models are used as end-to-end models. Thus, the performance of the model is evaluated for the quality of the extracted feature and classification accuracy. There are four contributions included in this work. First, this research is motivated by studying the quality of the extracted feature from the DL by feeding these extracted to an ML model. In other words, we proposed comparing the end-to-end DL model performance against the approach of using DL for feature extraction and ML for the classification of COVID-19 CT scan images. Second, we proposed studying the effect of fusing extracted features from image descriptors, e.g., Scale-Invariant Feature Transform (SIFT), with extracted features from DL models. Third, we proposed a new Convolutional Neural Network (CNN) to be trained from scratch and then compared to the deep transfer learning on the same classification problem. Finally, we studied the performance gap between classic ML models against ensemble learning models. The proposed framework is evaluated using a CT dataset, where the obtained results are evaluated using five different metrics The obtained results revealed that using the proposed CNN model is better than using the well-known DL model for the purpose of feature extraction. Moreover, using a DL model for feature extraction and an ML model for the classification task achieved better results in comparison to using an end-to-end DL model for detecting COVID-19 CT scan images. Of note, the accuracy rate of the former method improved by using ensemble learning models instead of the classic ML models. The proposed method achieved the best accuracy rate of 99.39%.

PurposeTo conduct a head-to-head comparison between deep learning (DL) and radiomics models across institutions for predicting microvascular invasion (MVI) in hepatocellular carcinoma (HCC) and to investigate the model robustness and generalizability through rigorous internal and external validation.MethodsThis retrospective study included 2304 preoperative images of 576 HCC lesions from two centers, with MVI status determined by postoperative histopathology. We developed DL and radiomics models for predicting the presence of MVI using B-mode ultrasound, contrast-enhanced ultrasound (CEUS) at the arterial, portal, and delayed phases, and a combined modality (B + CEUS). For radiomics, we constructed models with enlarged vs. original regions of interest (ROIs). A cross-validation approach was performed by training models on one center’s dataset and validating the other, and vice versa. This allowed assessment of the validity of different ultrasound modalities and the cross-center robustness of the models. The optimal model combined with alpha-fetoprotein (AFP) was also validated. The head-to-head comparison was based on the area under the receiver operating characteristic curve (AUC).ResultsThirteen DL models and 25 radiomics models using different ultrasound modalities were constructed and compared. B + CEUS was the optimal modality for both DL and radiomics models. The DL model achieved AUCs of 0.802–0.818 internally and 0.667–0.688 externally across the two centers, whereas radiomics achieved AUCs of 0.749–0.869 internally and 0.646–0.697 externally. The radiomics models showed overall improvement with enlarged ROIs (P < 0.05 for both CEUS and B + CEUS modalities). The DL models showed good cross-institutional robustness (P > 0.05 for all modalities, 1.6–2.1% differences in AUC for the optimal modality), whereas the radiomics models had relatively limited robustness across the two centers (12% drop-off in AUC for the optimal modality). Adding AFP improved the DL models (P < 0.05 externally) and well maintained the robustness, but did not benefit the radiomics model (P > 0.05).ConclusionCross-institutional validation indicated that DL demonstrated better robustness than radiomics for preoperative MVI prediction in patients with HCC, representing a promising solution to non-standardized ultrasound examination procedures.

1 - Congruence of deep learning in biomedical engineering: future prospects and challenges

A crucial method in medical diagnosis is using deep learning models with neuro assessment to detect and categorize brain cancers. One of the goals of this study is to use deep learning techniques, such as recurrent neural networks (RNNs) and multimodel convolutional neural networks (MCNNs), to automatically find and classify brain cancers in magnetic resonance imaging (MRI) scans. The first step is to gather a wide dataset of labeled MRI images of different tumor kinds. After that, we use preprocessing methods to eliminate noise and ensure the data is uniform. Next, we use the preprocessed dataset to train an adequate deep-learning architecture. Accuracy, precision, recall, and F1-score are some of the conventional measures used to assess the model's performance. Our model achieved a total accuracy of 99.94%, a recall of 99.98%, an F1-score of 99.945%, and a precision of 99.967%, according to the findings. These measures show that our method is successful in identifying and categorizing brain cancers using magnetic resonance imaging (MRI) scans. . Continuous monitoring and modifications enable the model's accuracy and dependability to improve. This research holds the potential to revolutionize the diagnosis of brain tumors, offering crucial insights for swift and precise patient care.