Medical image classification with convolutional neural network

Deep convolutional neural networks (CNNs) enable learning trainable, highly representative and hierarchical image feature from sufficient training data which makes rapid progress in computer vision possible. There are currently three major techniques that successfully employ CNNs to medical image classification: training the CNN from scratch, using off-the-shelf pretrained CNN features, and transfer learning , i.e., fine-tuning CNN models pretrained from natural image dataset (such as large-scale annotated natural image database: ImageNet) to medical image tasks. In this chapter, we exploit three important factors of employing deep convolutional neural networks to computer-aided detection problems. First, we exploit and evaluate several different CNN architectures including from shallower to deeper CNNs: classical CifarNet, to recent AlexNet and state-of-the-art GoogLeNet and their variants. The studied models contain five thousand to 160 million parameters and vary in the numbers of layers. Second, we explore the influence of dataset scales and spatial image context configurations on medical image classification performance. Third, when and why transfer learning from the pretrained ImageNet CNN models (via fine-tuning) can be useful for medical imaging tasks are carefully examined. We study two specific computer-aided detection (CADe) problems, namely thoracoabdominal lymph node (LN) detection and interstitial lung disease (ILD) classification. We achieve the state-of-the-art performance on the mediastinal LN detection and report the first fivefold cross-validation classification results on predicting axial CT slices with ILD categories. Our extensive quantitative evaluation, CNN model analysis, and empirical insights can be helpful to the design of high-performance CAD systems for other medical imaging tasks, without loss of generality.

Due to the complexity of medical images, traditional medical image classification methods have been unable to meet the actual application needs. In recent years, the rapid development of deep learning theory has provided a technical approach for solving medical image classification. However, deep learning has the following problems in the application of medical image classification. First, it is impossible to construct a deep learning model with excellent performance according to the characteristics of medical images. Second, the current deep learning network structure and training strategies are less adaptable to medical images. Therefore, this paper first introduces the visual attention mechanism into the deep learning model so that the information can be extracted more effectively according to the problem of medical images, and the reasoning is realized at a finer granularity. It can increase the interpretability of the model. Additionally, to solve the problem of matching the deep learning network structure and training strategy to medical images, this paper will construct a novel multiscale convolutional neural network model that can automatically extract high-level discriminative appearance features from the original image, and the loss function uses the Mahalanobis distance optimization model to obtain a better training strategy, which can improve the robust performance of the network model. The medical image classification task is completed by the above method. Based on the above ideas, this paper proposes a medical classification algorithm based on a visual attention mechanism-multiscale convolutional neural network. The lung nodules and breast cancer images were classified by the method in this paper. The experimental results show that the accuracy of medical image classification in this paper is not only higher than that of traditional machine learning methods but also improved compared with other deep learning methods, and the method has good stability and robustness.

Due to the complexity of medical images, traditional medical image classification methods have been unable to meet actual application needs. In recent years, the rapid development of deep learning theory has provided a technical approach for solving medical image classification tasks. However, deep learning has the following problems in medical image classification. First, it is impossible to construct a deep learning model hierarchy for medical image properties; second, the network initialization weights of deep learning models are not well optimized. Therefore, this paper starts from the perspective of network optimization and improves the nonlinear modeling ability of the network through optimization methods. A new network weight initialization method is proposed, which alleviates the problem that existing deep learning model initialization is limited by the type of the nonlinear unit adopted and increases the potential of the neural network to handle different visual tasks. Moreover, through an in‐depth study of the multicolumn convolutional neural network framework, this paper finds that the number of features and the convolution kernel size at different levels of the convolutional neural network are different. In contrast, the proposed method can construct different convolutional neural network models that adapt better to the characteristics of the medical images of interest and thus can better train the resulting heterogeneous multicolumn convolutional neural networks. Finally, using the adaptive sliding window fusion mechanism proposed in this paper, both methods jointly complete the classification task of medical images. Based on the above ideas, this paper proposes a medical classification algorithm based on a weight initialization/sliding window fusion for multilevel convolutional neural networks. The methods proposed in this study were applied to breast mass, brain tumor tissue, and medical image database classification experiments. The results show that the proposed method not only achieves a higher average accuracy than that of traditional machine learning and other deep learning methods but also is more stable and more robust.

This paper addresses the task of class-imbalanced medical image classification using complication labels. Medical image classification plays an essential role in improving patient care and reducing the burden on healthcare systems, along with other computer aided diagnosis tasks. However, there are two common obstacles with medical image image classification. One is that there is a high cost of acquiring annotations from doctors/experts. Two is that in many cases there is a differences in the number of incidents that appear per symptom which lead to class-imbalance. The above factors make medical image classification difficult to deal with. Additionally, there are cases where multiple symptoms appear in a mixed manner (complications), such as diabetic complications where both symptoms of infection and ischaemia could appear at the same time, which also could influence the classification result negatively. In this paper, a method that jointly tackles the obstacles mentioned above is presented. First, we conduct extensive experiment with various class-imbalanced learning methods introduced in previous works and also propose a method that improves the baseline class-imbalanced approaches by utilizing complication labels in pre-training. Second, we validate the proposed method with the diabetic foot ulcer dataset introduced in Diabetic Foot Ulcer Challenge 2021.

Deep learning models have demonstrated favorable performance on many medical image classification tasks. However, they rely on expensive hand-labeled datasets that are time-consuming to create. In this work, we explore a new supervision source to training deep learning models by using gaze data that is passively and cheaply collected during a clinician’s workflow. We focus on three medical imaging tasks, including classifying chest X-ray scans for pneumothorax and brain MRI slices for metastasis, two of which we curated gaze data for. The gaze data consists of a sequence of fixation locations on the image from an expert trying to identify an abnormality. Hence, the gaze data contains rich information about the image that can be used as a powerful supervision source. We first identify a set of gaze features and show that they indeed contain class-discriminative information. Then, we propose two methods for incorporating gaze features into deep learning pipelines. When no task labels are available, we combine multiple gaze features to extract weak labels and use them as the sole source of supervision (Gaze-WS). When task labels are available, we propose to use the gaze features as auxiliary task labels in a multi-task learning framework (Gaze-MTL). On three medical image classification tasks, our Gaze-WS method without task labels comes within 5 AUROC points (1.7 precision points) of models trained with task labels. With task labels, our Gaze-MTL method can improve performance by 2.4 AUROC points (4 precision points) over multiple baselines.

Medical supervised masked autoencoder: Crafting a better masking strategy and efficient fine-tuning schedule for medical image classification

In recent years, generative adversarial networks (GANs) have gained tremendous popularity for various imaging related tasks such as artificial image generation to support AI training. GANs are especially useful for medical imaging-related tasks where training datasets are usually limited in size and heavily imbalanced against the diseased class. We present a systematic review, following the PRISMA guidelines, of recent GAN architectures used for medical image analysis to help the readers in making an informed decision before employing GANs in developing medical image classification and segmentation models. We have extracted 54 papers that highlight the capabilities and application of GANs in medical imaging from January 2015 to August 2020 and inclusion criteria for meta-analysis. Our results show four main architectures of GAN that are used for segmentation or classification in medical imaging. We provide a comprehensive overview of recent trends in the application of GANs in clinical diagnosis through medical image segmentation and classification and ultimately share experiences for task-based GAN implementations.

Deep Learning has radically transformed the field of Medical Image Analysis by surpassing human performance in several challenging classification tasks and thus has widespread applications in industry and academia. In this work, we employ a Deep Convolutional Neural Network (DCNN) that uses Transfer Learning and is trained using the Open-access Medical Image Repositories and OpenI repository. We use DCNN to assist in a crowdsourced Big Data platform by conducting two crucial classification tasks: (1) Interclass classification of the type of the medical image—MRI, Ultrasound or X-Ray; (2) Intraclass classification of the X-Ray images into chest or abdomen X-Rays. Our approach automates the classification process of medical images, to provide certified medical experts with medical images of individuals on a crowdsourced platform, along with similar medical images of the suspected medical condition for validation, in order to enhance diagnostic effectiveness. In this work, we focus primarily on the classification tasks using DCNN; the elucidation of the privacy-preserving architecture of our cloud-based crowdsourcing platform and classification using conventional clustering approaches in the absence of large training images is deferred to future work.KeywordsBig dataDeep learningDCNNCrowdsourcingMedical image classificationObject detection

The transformation model plays a vital role in medical image processing. This paper proposed new two Mixed Transforms models that are the hybrid combination of linear and nonlinear Transformations techniques. The first mixed transform is computed in three steps: calculate 2D discrete cosine transform (DCT) of the image, and applying Arnold Transform (AT) on the DCT coefficients, and applying the discrete Wavelet Transform (DWT) on the result to get which was abbreviated as (CAW). The second mixed transform consists of firstly computing the discrete Fourier transform (DFT), net applying the Arnold Transform (AT), and finally, the computation of discrete Wavelet Transform (DWT) which was abbreviated as (FAW). These transforms have superior directional representations as compared to other multiresolution representations such as DWT or DCT and work as non-adaptive mixed transformations for multi-scale object analysis. Due to their relationship to the wavelet idea, they are finding increasing use in areas like image processing and scientific computing. These transforms are tested in medical image classification task and their performances are compared with that of the traditional transforms. CAW and FAW transforms are used in the feature extraction stage of a classification VGG16 deep learning (DNN) task of Tumor MRI medical image. The numerical findings favor CAW and FAW over the wavelet transform for estimating and classifying pictures. From the results obtained it was shown that the CAW and FAW transform gave e much higher classification rate than that achieved with the traditional transforms, namely DCT, DFT and DWT. Furthermore, this combination leads to a family of directional and multi-transformation bases for image processing.

Due to the high availability of large-scale annotated image datasets, paramount progress has been made in deep convolutional neural networks (CNNs) for image classification tasks. CNNs enable learning highly representative and hierarchical local image features directly from data. However, the availability of annotated data, especially in the medical imaging domain, remains the biggest challenge in the field. Transfer learning can provide a promising and effective solution by transferring knowledge from generic image recognition tasks to the medical image classification. However, due to irregularities in the dataset distribution, transfer learning usually fails to provide a robust solution. Class decomposition facilitates easier to learn class boundaries of a dataset, and consequently can deal with any irregularities in the data distribution. Motivated by this challenging problem, the paper presents Decompose, Transfer, and Compose (DeTraC) approach, a novel CNN architecture based on class decomposition to improve the performance of medical image classification using transfer learning and class decomposition approach. DeTraC enables learning at the subclass level that can be more separable with a prospect to faster convergence.We validated our proposed approach with three different cohorts of chest X-ray images, histological images of human colorectal cancer, and digital mammograms. We compared DeTraC with the state-of-the-art CNN models to demonstrate its high performance in terms of accuracy, sensitivity, and specificity.

Medical image classification plays an important role in disease diagnosis since it can provide important reference information for doctors. The supervised convolutional neural networks (CNNs) such as DenseNet provide the versatile and effective method for medical image classification tasks, but they require large amounts of data with labels and involve complex and time-consuming training process. The unsupervised CNNs such as principal component analysis network (PCANet) need no labels for training but cannot provide desirable classification accuracy. To realize the accurate medical image classification in the case of a small training dataset, we have proposed a light-weighted hybrid neural network which consists of a modified PCANet cascaded with a simplified DenseNet. The modified PCANet has two stages, in which the network produces the effective feature maps at each stage by convoluting inputs with various learned kernels. The following simplified DenseNet with a small number of weights will take all feature maps produced by the PCANet as inputs and employ the dense shortcut connections to realize accurate medical image classification. To appreciate the performance of the proposed method, some experiments have been done on mammography and osteosarcoma histology images. Experimental results show that the proposed hybrid neural network is easy to train and it outperforms such popular CNN models as PCANet, ResNet and DenseNet in terms of classification accuracy, sensitivity and specificity.

We previously developed two AI-based medical automatic image classification tools using a multi-layer fuzzy approach (MFA and MCM) to convert image-based abnormality into a quantity. However, there is currently limited research on using diagnostic image assessment tools to statistically predict the hazard due to the disease. The present study introduces a novel approach that addresses a substantial research gap in the identification of hazard or risk associated with a disease using an automatically quantified image-based abnormality. The method employed to ascertain hazard in an image-based quantified abnormality was the cox proportional hazard (PH) model, a unique tool in medical research for identifying hazard related to covariates. MFA was first used to quantify the abnormality in CT scan images, and hazard plots were utilized to visually represent the hazard risk over time. Hazards corresponding to image-based abnormality were then computed for the variables, ‘gender,’ ‘age,’ and ‘smoking-status’. This integrated framework potentially minimizes false negatives, identifies patients with the highest mortality risk and facilitates timely initiation of treatment. By utilizing pre-existing patient images, this method could reduce the considerable costs associated with public health research and clinical trials. Furthermore, understanding the hazard posed by widespread global diseases like COVID-19 aids medical researchers in prompt decision-making regarding treatment and preventive measures.

Convolutional neural networks (CNN) have been successfully used in the classification of medical images. However, CNNs extract the features of the image, neglecting the spatial information between the objects since they evaluate each object separately and this makes it difficult to analyse images reliably. CapsNets partially overcomes these limitations by encapsulating the objects in the image and their feature parameters without losing the spatial information between objects. However, this advantage of CapsNet can become a disadvantage due to the fact that CapsNet stores a lot of information, which can lead to an increase in the number of parameters and the complexity of the network. In this study, a new method is proposed in which residual block and CBAM (Convolutional Block Attention Module) modules are used hybridly with CapsNet in order to classify medical images with high accuracy and reliability and, at the same time, to overcome this disadvantage of CapsNet by reducing the number of parameters. In this study, more features of the image are extracted by using the residual block instead of the convolutional layer of CapsNet. The attention mechanism is used to increase the effectiveness of important features and reduce the effect of unnecessary features. In addition, the residual block and CBAM module reduced the number of parameters in CapsNet by approximately 2 million. Brain tumour images and OCT (Optical Coherence Tomography) images were classified with the proposed model. The results of the study show that the proposed model achieves significant results when compared to other CapsNet-based studies in the literature.

Deep learning and convolutional neural networks (CNNs) in particular are increasingly popular tools for segmentation and classification of medical images. CNNs were shown to be successful for segmentation of brain tumors into multiple regions or labels. However, in the environment which fosters data-sharing and collection of multi-institutional datasets, a question arises: does training with data from another institution with potentially different imaging equipment, contrast protocol, and patient population impact the segmentation performance of the CNN? Our study presents preliminary data towards answering this question. Specifically, we used MRI data of glioblastoma (GBM) patients for two institutions present in The Cancer Imaging Archive. We performed a process of training and testing CNN multiple times such that half of the time the CNN was tested on data from the same institution that was used for training and half of the time it was tested on another institution, keeping the training and testing set size constant. We observed a decrease in performance as measured by Dice coefficient when the CNN was trained with data from a different institution as compared to training with data from the same institution. The changes in performance for the entire tumor and for four different labels within the tumor were: 0.72 to 0.65 (p=0.06), 0.61 to 0.58 (p=0.49), 0.54 to 0.51 (p=0.82), 0.31 to 0.24 (p<0.03), and 0.43 to 0.31(p<0.003) respectively. In summary, we found that while data across institutions can be used for development of CNNs, this might be associated with a decrease in performance.

Deep learning methods, especially convolutional neural networks, have become more and more popular in medical image classifications. However, training a deep neural network from scratch can be a luxury for many medical image datasets as the process requires a large and well-balanced sample to output satisfactory results. Unlike natural image datasets, medical images are expensive to collect owing to labor and equipment costs. Besides, the class labels in medical image datasets are usually severely imbalanced subject to the availability of patients. Further, aggregating medical images from multiple sources can be challenging due to policy restrictions, privacy concerns, communication costs, and data heterogeneity caused by equipment differences and labeling discrepancies. In this paper, we propose to address these issues with the help of transfer learning and artificial samples created by generative models. Instead of requesting medical images from source data, our method only needs a parsimonious supplement of model parameters pre-trained on the source data. The proposed method preserves the data privacy in the source data and significantly reduces the communication cost. Our study shows transfer learning together with artificial samples can improve the pneumonia classification accuracy on a small but heavily imbalanced chest X-ray image dataset by \(11.53\%\) which performs even better than directly augmenting that source data into the training process.KeywordsDeep learningGenerative modelsMedical image classificationPrivacy preservationTransfer learning