Stable polyp-scene classification via subsampling and residual learning from an imbalanced large dataset

One of the limitations of the current computer-aided detection (CAD) of polyps in CT colonography (CTC) is a relatively large number of false-positive (FP) detections. Rectal tubes (RTs) are one of the typical sources of FPs because a portion of a RT, especially a portion of a bulbous tip, often exhibits a cap-like shape that closely mimics the appearance of a small polyp. Radiologists can easily recognize and dismiss RT-induced FPs; thus, they may lose their confidence in CAD as an effective tool if the CAD scheme generates such "obvious" FPs due to RTs consistently. In addition, RT-induced FPs may distract radiologists from less common true positives in the rectum. Therefore, removal RT-induced FPs as well as other types of FPs is desirable while maintaining a high sensitivity in the detection of polyps. We developed a three-dimensional (3D) massive-training artificial neural network (MTANN) for distinction between polyps and RTs in 3D CTC volumetric data. The 3D MTANN is a supervised volume-processing technique which is trained with input CTC volumes and the corresponding "teaching" volumes. The teaching volume for a polyp contains a 3D Gaussian distribution, and that for a RT contains zeros for enhancement of polyps and suppression of RTs, respectively. For distinction between polyps and nonpolyps including RTs, a 3D scoring method based on a 3D Gaussian weighting function is applied to the output of the trained 3D MTANN. Our database consisted of CTC examinations of 73 patients, scanned in both supine and prone positions (146 CTC data sets in total), with optical colonoscopy as a reference standard for the presence of polyps. Fifteen patients had 28 polyps, 15 of which were 5-9 mm and 13 were 10-25 mm in size. These CTC cases were subjected to our previously reported CAD scheme that included centerline-based segmentation of the colon, shape-based detection of polyps, and reduction of FPs by use of a Bayesian neural network based on geometric and texture features. Application of this CAD scheme yielded 96.4% (27/28) by-polyp sensitivity with 3.1 (224/73) FPs per patient, among which 20 FPs were caused by RTs. To eliminate the FPs due to RTs and possibly other normal structures, we trained a 3D MTANN with ten representative polyps and ten RTs, and applied the trained 3D MTANN to the above CAD true- and false-positive detections. In the output volumes of the 3D MTANN, polyps were represented by distributions of bright voxels, whereas RTs and other normal structures partly similar to RTs appeared as darker voxels, indicating the ability of the 3D MTANN to suppress RTs as well as other normal structures effectively. Application of the 3D MTANN to the CAD detections showed that the 3D MTANN eliminated all RT-induced 20 FPs, as well as 53 FPs due to other causes, without removal of any true positives. Overall, the 3D MTANN was able to reduce the FP rate of the CAD scheme from 3.1 to 2.1 FPs per patient (33% reduction), while the original by-polyp sensitivity of 96.4% was maintained.

It is critical to have accurate detection of lung nodules in CT images for the early diagnosis of lung cancer. In order to achieve this, it is necessary to reduce the false positive rate of detection. Due to the heterogeneity of lung nodules and their similarity to the background, it is difficult to distinguish true lung nodules from numerous candidate nodules. In this paper, in order to solve this challenging problem, we propose a Multi-Branch Ensemble Learning architecture based on the three-dimensional (3D) convolutional neural networks (MBEL-3D-CNN). The method combines three key ideas: 1) constructing a 3D-CNN to make the maximum utilization of spatial information of lung nodules in the 3D space; 2) embedding a multi-branch network architecture in the 3D-CNN that is well adapted to the heterogeneity of lung nodules, and; 3) using ensemble learning to effectively improve the generalization performance of the 3D-CNN model. In addition, we use offline hard mining operations to make the network capable of handling those indistinguishable positive and negative samples. The proposed method was tested on the dataset LUNA16 in our experiments. The experimental results show that MBEL-3D-CNN architecture can achieve better screening results.

Introduction: Water exchange (WE) improves polyp (PDR) and adenoma detection rate (ADR) compared with air insufflation [GIE 2021;93:1411], but still misses polyps due to human limitations [JCG 2021, in press]. A computer-aided detection (CADe) algorithm can overcome human omissions, but is limited by false positives (FPs) [Lancet Gastroenterol & Hepatol 2019;4:71], which might be reduced by salvage-cleaning effect provided by insertion WE [Tzu-Chi Med J 2021;33:108]. We hypothesized that the strengths of CADe and WE can complement the weaknesses of each other. Their combination can enhance PDR and ADR and reduce false positive rate (FPR). Methods: We used a locally developed convolutional neural network with YOLOv4 to build a CADe model. Images of polyps (n=15961) were divided in 80% and 20% for training and testing. Prior to real-time application, we validated the model in detecting polyps, using edited and coded withdrawal phase videos of the right colon from a RCT comparing right colon adenoma detection between WE and air insufflation. The polyps detected by the CADe were confirmed by the consensus of 2 blinded reviewers (experienced endoscopists). We also developed a computer-aided diagnosis (CADx) model based on YOLOv4 to characterize the detected polyps as adenomas or not. The primary and secondary outcomes were PDR and the FPR, respectively. Results: A total of 245 coded videos of colonoscopies inserted with WE (n=123) or air insufflation (n=122) were analyzed. CADe achieved significantly higher PDR in the WE group (61.8%) than in the air insufflation group (40.2%), P=0.0009. CADe also had significantly higher additional PDR in the WE group (30.1% vs 12.3%, P=0.001). The CADx assessed ADR was significantly higher in the WE group than that in the air insufflation group (46.3% vs 30.3%, P=0.018), after the detected polyps were characterized by the CADx. (Table a) The mean numbers of FPs related to feces [1.78 (1.67) vs 2.09 (2.09), P=0.007] and bubbles [0.53 (0.89) vs 1.25 (2.45), P=0.001] were significantly lower in the WE group than in the air insufflation group. (Table b). Conclusion: The data confirmed the hypothesis that the strengths of CADe and WE complemented the weaknesses of each other, enhanced PDR and ADR and reduced FPR. The locally developed CADe and CADx are validated for polyp detection and adenoma assignment for real-time application. Furthermore, WE provides an optimal platform for the computer-assisted algorithms.Table 1.: ADR, adenoma detection rate; All the percentages calculated according to respective total number of patients for each group; PDR, polyp detection rate; Total numbers of polyps were based on those found by the performing endoscopist plus those detected by the CADe; Proportion of additional polyps found by CADe equals numbers of additional polyp (adenomas) found by CADe divided by total numbers of polyps (adenomas). Others: ileocecal valve, cecum, polypectomy site, suction marks and drug pills. † Mann-Whitney test; ††Fisher exact test.

An important measurable indicator of urbanization and its environmental implications has been identified as the urban impervious surface. It presents a strategy based on three-dimensional convolutional neural networks (3D CNNs) for extracting urbanization from the LiDAR datasets using deep learning technology. Various 3D CNN parameters are tested to see how they affect impervious surface extraction. For urban impervious surface delineation, this study investigates the synergistic integration of multiple remote sensing datasets of Azad Kashmir, State of Pakistan, to alleviate the restrictions imposed by single sensor data. Overall accuracy was greater than 95% and overall kappa value was greater than 90% in our suggested 3D CNN approach, which shows tremendous promise for impervious surface extraction. Because it uses multiscale convolutional processes to combine spatial and spectral information and texture and feature maps, we discovered that our proposed 3D CNN approach makes better use of urbanization than the commonly utilized pixel-based support vector machine classifier. In the fast-growing big data era, image analysis presents significant obstacles, yet our proposed 3D CNNs will effectively extract more urban impervious surfaces.

Fluorescence lifetime imaging of the co-enzyme reduced nicotinamide adenine dinucleotide (NADH) offers a label-free approach for detecting cellular metabolic perturbations. However, the relationships between variations in NADH lifetime and metabolic pathway changes are complex, preventing robust interpretation of NADH lifetime data relative to metabolic phenotypes. Here, a three-dimensional convolutional neural network (3D CNN) trained at the cell level with 3D NAD(P)H lifetime decay images (two spatial dimensions and one time dimension) was developed to identify metabolic pathway usage by cancer cells. NADH fluorescence lifetime images of MCF7 breast cancer cells with three isolated metabolic pathways, glycolysis, oxidative phosphorylation, and glutaminolysis were obtained by a multiphoton fluorescence lifetime microscope and then segmented into individual cells as the input data for the classification models. The 3D CNN models achieved over 90% accuracy in identifying cancer cells reliant on glycolysis, oxidative phosphorylation, or glutaminolysis. Furthermore, the model trained with human breast cancer cell data successfully predicted the differences in metabolic phenotypes of macrophages from control and POLG-mutated mice. These results suggest that the integration of autofluorescence lifetime imaging with 3D CNNs enables intracellular spatial patterns of NADH intensity and temporal dynamics of the lifetime decay to discriminate multiple metabolic phenotypes. Furthermore, the use of 3D CNNs to identify metabolic phenotypes from NADH fluorescence lifetime decay images eliminates the need for time- and expertise-demanding exponential decay fitting procedures. In summary, metabolic-prediction CNNs will enable live-cell and in vivo metabolic measurements with single-cell resolution, filling a current gap in metabolic measurement technologies.

A model was developed to understand the effects of spatial resolution and Signal to Noise ratio on the detection and tracking performance of wide-field, diffraction-limited electro-optic and infrared motion imagery systems. False positive detection probability and false positive rate per frame were calculated as a function of target-to-background contrast and object size. Results showed that moving objects are fundamentally more difficult to detect than stationary objects because SNR for fixed objects increases and false positive probability detection rates diminish rapidly with successive frames whereas for moving objects the false detection rate remains constant or increases with successive frames. The model specifies that the desired performance of a detection system, measured by the false positive detection rate, can be achieved by image system designs with different combinations of SNR and spatial resolution, usually requiring several pixels resolving the object; this capability to tradeoff resolution and SNR enables system design trades and cost optimization. For operational use, detection thresholds required to achieve a particular false detection rate can be calculated. Interestingly, for moderate size images the model converges to the Johnson Criteria. Johnson found that an imaging system with an SNR >3.5 has a probability of detection >50% when the resolution on the object is 4 pixels or more. Under these conditions our model finds the false positive rate is less than one per hundred image frames, and the ratio of the probability of object detection to false positive detection is much greater than one. The model was programmed into Matlab to generate simulated images frames for visualization.

Currently, lung cancer has one of the highest mortality rates because it is often caught too late. Therefore, early detection is essential to reduce the risk of death. Pulmonary nodules are considered key indicators of primary lung cancer. Developing an efficient and accurate computer-aided diagnosis system for pulmonary nodule detection is an important goal. Typically, a computer-aided diagnosis system for pulmonary nodule detection consists of two parts: candidate nodule extraction and false-positive reduction of candidate nodules. The reduction of false positives (FPs) of candidate nodules remains an important challenge due to morphological characteristics of nodule height changes and similar characteristics to other organs. In this study, we propose a novel multi-scale heterogeneous three-dimensional (3D) convolutional neural network (MSH-CNN) based on chest computed tomography (CT) images. There are three main strategies of the design: (1) using multi-scale 3D nodule blocks with different levels of contextual information as inputs; (2) using two different branches of 3D CNN to extract the expression features; (3) using a set of weights which are determined by back propagation to fuse the expression features produced by step 2. In order to test the performance of the algorithm, we trained and tested on the Lung Nodule Analysis 2016 (LUNA16) dataset, achieving an average competitive performance metric (CPM) score of 0.874 and a sensitivity of 91.7% at two FPs/scan. Moreover, our framework is universal and can be easily extended to other candidate false-positive reduction tasks in 3D object detection, as well as 3D object classification.

The time-frequency domain features of vibration signals provide valuable information for deep learning-based rolling bearing fault diagnosis methods, where fault signal classification aiding in the identification of nominal fault types during diagnosis. The Short-Time Fourier Transform (STFT) is a widely used time-frequency transformation method, and its window length is the key parameter that determines the trade-off between time and frequency resolution. The primary motivation of this study is to address the limitation in traditional STFT-based 2D CNN methods: the inability to adapt the window length to different types of signals. To achieve accurate classification of bearing fault types, this study proposes a method based on three-dimensional convolutional neural networks (3D CNNs) to deeply explore the time-frequency domain information of one-dimensional vibration signals from faulty bearings. This method first applies STFT with multiple window sizes to perform multi-resolution time-frequency transformations on the time-domain vibration signals, yielding three-dimensional data. Subsequently, a classifier is trained based on the proposed 3D CNN. Experimental results on public datasets show that, without any sophisticated techniques, the proposed method achieves an average classification accuracy of 99.2% for six types of bearing faults using a relatively simple CNN structure. Compared to 1D CNN and 2D CNN methods that use fixed window sizes for STFT, the proposed method significantly enhances classification performance. Furthermore, it demonstrates robust classification results even on small-scaled bearing datasets.

Artificial Intelligence–Aided Colonoscopy for Characterizing and Detecting Colorectal Polyps: Required, Nice to Have, or Overhyped?

This study aimed to explore the application value of three-dimensional (3D) convolutional neural networks (3D-CNN)-based computed tomography (CT) image intelligent segmentation model in the identification of lesions of patients with hepatitis B virus-associated acute-on-chronic liver failure (HBV-ACLF). A total of 30 patients with HBV-ACLF, 30 patients with chronic HBV hospitalized in hospital, and 30 healthy volunteers were selected as subjects. Liver function and serum inflammatory factors were measured in each group, and the 3D-CNN algorithm model was applied to CT imaging. The results showed that the levels of interleukin (IL)-6, IL-26, and IL-37 in the HBV-ACLF group were the highest, which were 128.43 ± 45.16 pg/mL, 1237.47 ± 536.22 pg/mL, and 50.83 ± 7.62 pg/mL, respectively. Total bilirubin (TBIL) (P=0.035) and IL-26 (P=0.013) were independent predictors that affected the prognosis of HBV-ACLF patients. The results of lesion segmentation showed that the Dice coefficient of 3D-CNN low-density focus and enhanced focus segmentation was the highest (0.821 ± 0.07 and 0.773 ± 0.071), and the marked area was close to the area manually drawn by the doctor. 3D CNN was superior to other algorithms in the number of nodular lesions detected (533), sensitivity (97.5%), and missed detection rate (0.52%) (P < 0.05). In short, IL-26 may become a useful biomarker in the treatment of HBV-ACLF. The 3D-CNN model improved the segmentation performance of lesions in CT images of HBV-ACLF patients, which provided a reference for the diagnosis and prognosis of HBV-ACLF.

To test the hypothesis that artificial intelligence (AI) techniques can aid in identifying and assessing lesion severity in the cartilage, bone marrow, meniscus, and anterior cruciate ligament (ACL) in the knee, improving overall MRI interreader agreement. This retrospective study was conducted on 1435 knee MRI studies (n = 294 patients; mean age, 43 years ± 15 [standard deviation]; 153 women) collected within three previous studies (from 2011 to 2014). All MRI studies were acquired using high-spatial-resolution three-dimensional fast-spin-echo CUBE sequence. Three-dimensional convolutional neural networks were developed to detect the regions of interest within MRI studies and grade abnormalities of the cartilage, bone marrow, menisci, and ACL. Evaluation included sensitivity, specificity, and Cohen linear-weighted ĸ. The impact of AI-aided grading in intergrader agreement was assessed on an external dataset. Binary lesion sensitivity reported for all tissues was between 70% and 88%. Specificity ranged from 85% to 89%. The area under the receiver operating characteristic curve for all tissues ranged from 0.83 to 0.93. Deep learning-assisted intergrader Cohen ĸ agreement significantly improved in 10 of 16 comparisons among two attending physicians and two trainees for all tissues. The three-dimensional convolutional neural network had high sensitivity, specificity, and accuracy for knee-lesion-severity scoring and also increased intergrader agreement when used on an external dataset.Supplemental material is available for this article. Keywords: Bone Marrow, Cartilage, Computer Aided Diagnosis (CAD), Computer Applications-3D, Computer Applications-Detection/Diagnosis, Knee, Ligaments, MR-Imaging, Neural Networks, Observer Performance, Segmentation, Statistics © RSNA, 2021See also the commentary by Li and Chang in this issue.: An earlier incorrect version of this article appeared online. This article was corrected on April 16, 2021.

Lung cancer, one of the most lethal malignancies globally, has a high mortality rate, making early screening and accurate diagnosis crucial for improving patient survival rates. Low-dose computed tomography (LDCT) has significantly increased the detection rate of lung nodules, but the vast amount of imaging data poses a significant challenge for manual interpretation. This article proposed a lung nodule computer-aided quantitative diagnostic system based on a three-dimensional convolutional neural network (3D CNN), named 3D-DHA-MAFNet. By integrating deformable convolution network (DCN), hybrid attention mechanism (HAM), and multi-view adaptive fusion network (MAFN), this system significantly enhanced the detection and benign/malignant classification capabilities for small and irregular lung nodules. The model was trained and evaluated on the lung nodule analysis 2016 (LUNA16) and LIDC-IDRI datasets with a patient-level split in the ratio of 7:2:1. The number of training epochs was 200. Data augmentation methods included random rotation (±15°), translation (±5 pixels), and elastic deformation. The performance was compared with baseline models including 3D-UNet, AttentNet, 3DAGNet, 3D multi-scale capsule network (3D-MCN), and 3D multi-view SE CNN (3D MVSECNN). Experiments showed that on the LUNA16 dataset, the model achieved a detection sensitivity of 98.7% with a low false-positive rate of 1.2 per scan, which was a notable improvement over baseline models like 3D-UNet. On the LIDC-IDRI dataset, the classification accuracy for benign and malignant nodules reached 96.8%, with an AUC value of 0.983, and particularly, the sensitivity for malignant nodules was enhanced to 97.5%. Ablation studies verified the synergistic gains of each module, and cross-clinical dataset testing further demonstrated its strong generalization ability. This article presents a high-precision and interpretable intelligent diagnostic tool designed for clinical applications, which holds strong potential to promote the clinical translation of early lung cancer screening.

The bone marrow overproduces immature cells in the malignancy known as Acute Lymphoblastic Leukemia (ALL). In the United States, about 6500 occurrences of ALL are diagnosed each year in both children and adults, comprising nearly 25% of pediatric cancer cases. Recently, many computer-assisted diagnosis (CAD) systems have been proposed to aid hematologists in reducing workload, providing correct results, and managing enormous volumes of data. Traditional CAD systems rely on hematologists’ expertise, specialized features, and subject knowledge. Utilizing early detection of ALL can aid radiologists and doctors in making medical decisions. In this study, Deep Dilated Residual Convolutional Neural Network (DDRNet) is presented for the classification of blood cell images, focusing on eosinophils, lymphocytes, monocytes, and neutrophils. To tackle challenges like vanishing gradients and enhance feature extraction, the model incorporates Deep Residual Dilated Blocks (DRDB) for faster convergence. Conventional residual blocks are strategically placed between layers to preserve original information and extract general feature maps. Global and Local Feature Enhancement Blocks (GLFEB) balance weak contributions from shallow layers for improved feature normalization. The global feature from the initial convolution layer, when combined with GLFEB-processed features, reinforces classification representations. The Tanh function introduces non-linearity. A Channel and Spatial Attention Block (CSAB) is integrated into the neural network to emphasize or minimize specific feature channels, while fully connected layers transform the data. The use of a sigmoid activation function concentrates on relevant features for multiclass lymphoblastic leukemia classification The model was analyzed with Kaggle dataset (16,249 images) categorized into four classes, with a training and testing ratio of 80:20. Experimental results showed that DRDB, GLFEB and CSAB blocks’ feature discrimination ability boosted the DDRNet model F1 score to 0.96 with minimal computational complexity and optimum classification accuracy of 99.86% and 91.98% for training and testing data. The DDRNet model stands out from existing methods due to its high testing accuracy of 91.98%, F1 score of 0.96, minimal computational complexity, and enhanced feature discrimination ability. The strategic combination of these blocks (DRDB, GLFEB, and CSAB) are designed to address specific challenges in the classification process, leading to improved discrimination of features crucial for accurate multi-class blood cell image identification. Their effective integration within the model contributes to the superior performance of DDRNet.

Understanding human behavior and human action recognition are both essential components of effective surveillance video analysis for the purpose of guaranteeing public safety. However, existing approaches such as three-dimensional convolutional neural networks (3D CNN) and two-stream neural networks (2SNN) have computational hurdles due to the significant parameterization they require. In this paper, we offer HARNet, a specialized lightweight residual 3D CNN that is built on directed acyclic graphs and was created expressly to handle these issues and achieve effective human action detection. The suggested method presents an innovative pipeline for creating spatial motion data from raw video inputs, which makes successful latent representation learning of human motions easier to accomplish. This generated input is then supplied into HARNet, which processes spatial and motion information in a single stream in an effective manner, maximizing the benefits of both types of cues. The use of traditional machine learning classifiers is done in order to further improve the discriminative capacity of the features that have been learned. To be more specific, we use the latent representations that are stored in HARNet’s fully connected layer and use them as our deep learnt features. After that, these features are entered into the Support Vector Machine (SVM) classifier in order to accomplish action recognition. In order to evaluate the HARNet-SVM method that was developed, empirical tests were run on commonly used action recognition datasets such as UCF101, HMDB51, and the KTH dataset. These tests were carried out in order to gather data for the evaluation. The experimental results show that our method is superior to other state-of-the-art approaches, achieving considerable performance increases of 2.75% on UCF101, 10.94% on HMDB51, and 0.18% on the KTH dataset. These results were obtained by running the method on each dataset separately. Our findings demonstrate the usefulness of HARNet’s lightweight design and highlight the significance of utilizing SVM classifiers with deep learnt features for the purpose of accurate and computationally efficient human activity recognition in surveillance videos. This work helps to the advancement of surveillance technology, which in turn makes video analysis in applications that take place in the real world safer and more dependable.

BackgroundTo introduce a three-dimensional convolutional neural network (3D CNN) leveraging transfer learning for fusing PET/CT images and clinical data to predict EGFR mutation status in lung adenocarcinoma (LADC).MethodsRetrospective data from 516 LADC patients, encompassing preoperative PET/CT images, clinical information, and EGFR mutation status, were divided into training (n = 404) and test sets (n = 112). Several deep learning models were developed utilizing transfer learning, involving CT-only and PET-only models. A dual-stream model fusing PET and CT and a three-stream transfer learning model (TS_TL) integrating clinical data were also developed. Image preprocessing includes semi-automatic segmentation, resampling, and image cropping. Considering the impact of class imbalance, the performance of the model was evaluated using ROC curves and AUC values.ResultsTS_TL model demonstrated promising performance in predicting the EGFR mutation status, with an AUC of 0.883 (95%CI = 0.849–0.917) in the training set and 0.730 (95%CI = 0.629–0.830) in the independent test set. Particularly in advanced LADC, the model achieved an AUC of 0.871 (95%CI = 0.823–0.919) in the training set and 0.760 (95%CI = 0.638–0.881) in the test set. The model identified distinct activation areas in solid or subsolid lesions associated with wild and mutant types. Additionally, the patterns captured by the model were significantly altered by effective tyrosine kinase inhibitors treatment, leading to notable changes in predicted mutation probabilities.ConclusionPET/CT deep learning model can act as a tool for predicting EGFR mutation in LADC. Additionally, it offers clinicians insights for treatment decisions through evaluations both before and after treatment.