Development of a Three-Dimensional Convolutional Neural Network with Attention for Aneurysm Detection

This study aims to develop and assess an optimized three-dimensional convolutional neural network model (3D CNN) for predicting major cardiac events from coronary computed tomography angiography (CCTA) images in patients with suspected coronary artery disease. Patients undergoing CCTA with suspected coronary artery disease (CAD) were retrospectively included in this single-center study and split into training and test sets. The endpoint was defined as a composite of all-cause death, myocardial infarction, unstable angina, or revascularization events. Cardiovascular risk assessment relied on Morise score and the extent of CAD (eoCAD). An optimized 3D CNN mimicking the DenseNet architecture was trained on CCTA images to predict the clinical endpoints. The data was unannotated for presence of coronary plaque. A total of 5562 patients were assigned to the training group (66.4% male, median age 61.1 ± 11.2); 714 to the test group (69.3% male, 61.5 ± 11.4). Over a 7.2-year follow-up, the composite endpoint occurred in 760 training group and 83 test group patients. In the test cohort, the CNN achieved an AUC of 0.872 ± 0.020 for predicting the composite endpoint. The predictive performance improved in a stepwise manner: from an AUC of 0.652 ± 0.031 while using Morise score alone to 0.901 ± 0.016 when adding eoCAD and finally to 0.920 ± 0.015 when combining Morise score, eoCAD, and CNN (p < 0.001 and p = 0.012, respectively). Deep learning-based analysis of CCTA images improves prognostic risk stratification when combined with clinical and imaging risk factors in patients with suspected CAD.

Seizure detection in epilepsy monitoring units (EMUs) is essential for the clinical assessment of drug-resistant epilepsy. Automated video analysis using machine learning provides a promising aid for seizure detection, with resultant reduction in the resources required for diagnostic monitoring. We employ a three-dimensional (3D) convolutional neural network with fully fine-tuned backbone layers to identify seizures from EMU videos. A two-stream inflated 3D-ConvNet architecture (I3D) classified video clips as a seizure or not a seizure. A pretrained action classifier was fine-tuned on 11 h of video containing 49 tonic-clonic seizures from 25 patients monitored at a large academic hospital (site A) using leave-one-patient-out cross-validation. Performance was evaluated by comparing model predictions to ground-truth annotations obtained from video-electroencephalographic review by an epileptologist on videos from site A and a separate dataset from a second large academic hospital (site B). The model achieved a leave-one-patient-out cross-validation F1-score of .960 ± .007 (mean ± SD) and area under the receiver operating curve score of .988 ± .004 at site A. Evaluation on full videos detected all seizures (95% binomial exact confidence interval = 94.1%-100%), with median detection latency of 0.0 s (interquartile range = 0.0-3.0) from seizure onset. The site A model had an average false alarm rate of 1.81 alarms per hour, although 36 of the 49 videos (73.5%) had no false alarms. Evaluation at site B demonstrated generalizability of the architecture and training strategy, although cross-site evaluation (site A model tested on site B data and vice versa) resulted in diminished performance. Our model demonstrates high performance in the detection of epileptic seizures from video data using a fine-tuned I3D model and outperforms similar models identified in the literature. This study provides a foundation for future work in real-time EMU seizure monitoring and possibly for reliable, cost-effective at-home detection of tonic-clonic seizures.

Convolutional neural networks (CNN) have given rise to a new generation of video super-resolution (SR) technique. However, most existing CNN-based video SR algorithms treat the consecutive frames as a series of feature maps, just as the procedure performed in single image SR algorithms. We propose an end-to-end three-dimensional (3-D) CNN video SR framework. The input frames are considered as a cube in our framework. 3-D convolution is performed on it to extract features along spatial and temporal dimension. Image prior knowledge, such as optical flows, is introduced in reconstruction. A combination of mean square error loss and multiscale structure similarity index (MS-SSIM) loss is used to optimize the model. Experimental results show that the proposed method reconstructs high-resolution frames with more accurate and visually pleasant structures compared with state-of-the-art video SR algorithms. We also achieve comparable PSNR/SSIM results with less computation time.

To test the hypothesis that artificial intelligence (AI) techniques can aid in identifying and assessing lesion severity in the cartilage, bone marrow, meniscus, and anterior cruciate ligament (ACL) in the knee, improving overall MRI interreader agreement. This retrospective study was conducted on 1435 knee MRI studies (n = 294 patients; mean age, 43 years ± 15 [standard deviation]; 153 women) collected within three previous studies (from 2011 to 2014). All MRI studies were acquired using high-spatial-resolution three-dimensional fast-spin-echo CUBE sequence. Three-dimensional convolutional neural networks were developed to detect the regions of interest within MRI studies and grade abnormalities of the cartilage, bone marrow, menisci, and ACL. Evaluation included sensitivity, specificity, and Cohen linear-weighted ĸ. The impact of AI-aided grading in intergrader agreement was assessed on an external dataset. Binary lesion sensitivity reported for all tissues was between 70% and 88%. Specificity ranged from 85% to 89%. The area under the receiver operating characteristic curve for all tissues ranged from 0.83 to 0.93. Deep learning-assisted intergrader Cohen ĸ agreement significantly improved in 10 of 16 comparisons among two attending physicians and two trainees for all tissues. The three-dimensional convolutional neural network had high sensitivity, specificity, and accuracy for knee-lesion-severity scoring and also increased intergrader agreement when used on an external dataset.Supplemental material is available for this article. Keywords: Bone Marrow, Cartilage, Computer Aided Diagnosis (CAD), Computer Applications-3D, Computer Applications-Detection/Diagnosis, Knee, Ligaments, MR-Imaging, Neural Networks, Observer Performance, Segmentation, Statistics © RSNA, 2021See also the commentary by Li and Chang in this issue.: An earlier incorrect version of this article appeared online. This article was corrected on April 16, 2021.

In this paper, we present a new single sound source DOA estimation and tracking system based on the well-known SRP-PHAT algorithm and a three-dimensional Convolutional Neural Network. It uses SRP-PHAT power maps as input features of a fully convolutional causal architecture that uses 3D convolutional layers to accurately perform the tracking of a sound source even in highly reverberant scenarios where most of the state of the art techniques fail. Unlike previous methods, since we do not use bidirectional recurrent layers and all our convolutional layers are causal in the time dimension, our system is feasible for real-time applications and it provides a new DOA estimation for each new SRP-PHAT map. To train the model, we introduce a new procedure to simulate random trajectories as they are needed during the training, equivalent to an infinite-size dataset with high flexibility to modify its acoustical conditions such as the reverberation time. We use both acoustical simulations on a large range of reverberation times and the actual recordings of the LOCATA dataset to prove the robustness of our system and its good performance even using low-resolution SRP-PHAT maps.

Three-dimensional convolutional neural networks (3DCNNs), a rapidly evolving modality of deep learning, has gained popularity in many fields. For oral cancers, CT images are traditionally processed using two-dimensional input, without considering information between lesion slices. In this paper, we established a 3DCNNs-based image processing algorithm for the early diagnosis of oral cancers, which was compared with a 2DCNNs-based algorithm. The 3D and 2D CNNs were constructed using the same hierarchical structure to profile oral tumors as benign or malignant. Our results showed that 3DCNNs with dynamic characteristics of the enhancement rate image performed better than 2DCNNS with single enhancement sequence for the discrimination of oral cancer lesions. Our data indicate that spatial features and spatial dynamics extracted from 3DCNNs may inform future design of CT-assisted diagnosis system.

Middle cerebral artery aneurysm is a common type of intracranial aneurysm in neurosurgery, accounting for about 20% of intracranial aneurysms, and is the third most common site of intracranial aneurysms. The surgical success rate and postoperative recovery ability of today's treatment plans are not satisfactory. Therefore, this paper designs a health model based on data analysis to clinically apply clipping surgery for cerebral aneurysm. This paper studies data analytics health models in the context of big data analytics. The model combines the characteristics of cerebral aneurysms for targeted analysis, and then through the understanding of the clipping treatment of cerebral aneurysms, this paper combines the deep learning in the neural network to train the treatment plan under the data analysis health model. Finally, this paper designs a therapeutic plan for clipping treatment of cerebral aneurysm based on a data analysis health model. To verify its data analysis ability, this paper designs experiments on unbalanced data sets and experiments to improve the execution efficiency of the algorithm. After analyzing the results obtained from the experiment, this paper will apply them to the clinic. The final experiment showed that the surgical success rate of the clipping treatment for cerebral aneurysm based on the data analysis health model was increased by 21.84% compared with the traditional clipping treatment for cerebral aneurysm.

Multimodal emotion recognition has become a hot topic in human-computer interaction and intelligent healthcare fields. However, combining information from different human different modalities for emotion computation is still challenging. In this paper, we propose a three-dimensional convolutional recurrent neural network model (referred to as 3FACRNN network) based on multimodal fusion and attention mechanism. The 3FACRNN network model consists of a visual network and an EEG network. The visual network is composed of a cascaded convolutional neural network-time convolutional network (CNN-TCN). In the EEG network, the 3D feature building module was added to integrate band information, spatial information and temporal information of the EEG signal, and the band attention and self-attention modules were added to the convolutional recurrent neural network (CRNN). The former explores the effect of different frequency bands on network recognition performance, while the latter is to obtain the intrinsic similarity of different EEG samples. To investigate the effect of different frequency bands on the experiment, we obtained the average attention mask for all subjects in different frequency bands. The distribution of the attention masks across the different frequency bands suggests that signals more relevant to human emotions may be active in the high frequency bands γ (31-50 Hz). Finally, we try to use the multi-task loss function Lc to force the approximation of the intermediate feature vectors of the visual and EEG modalities, with the aim of using the knowledge of the visual modalities to improve the performance of the EEG network model. The mean recognition accuracy and standard deviation of the proposed method on the two multimodal sentiment datasets DEAP and MAHNOB-HCI (arousal, valence) were 96.75 ± 1.75, 96.86 ± 1.33; 97.55 ± 1.51, 98.37 ± 1.07, better than those of the state-of-the-art multimodal recognition approaches. The experimental results show that starting from the multimodal information, the facial video frames and electroencephalogram (EEG) signals of the subjects are used as inputs to the emotion recognition network, which can enhance the stability of the emotion network and improve the recognition accuracy of the emotion network. In addition, in future work, we will try to utilize sparse matrix methods and deep convolutional networks to improve the performance of multimodal emotion networks.

Background The primary surgical approach for removing adrenal masses is minimally invasive adrenalectomy. Recognition of anatomical landmarks during surgery is critical for minimizing complications. Artificial intelligence-based tools can be utilized to create real-time navigation systems during laparoscopic and robotic right adrenalectomy. In this study, we aimed to develop deep learning models that can identify critical anatomical structures during minimally invasive right adrenalectomy. Methods In this experimental feasibility study, intraoperative videos of 20 patients who underwent minimally invasive right adrenalectomy in a tertiary care center between 2011 and 2023 were analyzed and used to develop an artificial intelligence-based anatomical landmark recognition system. Semantic segmentation of the liver, the inferior vena cava (IVC), and the right adrenal gland were performed. Fifty random images per patient during the dissection phase were extracted from videos. The experiments on the annotated images were performed on two state-of-the-art segmentation models named SwinUNETR and MedNeXt, which are transformer and convolutional neural network (CNN)-based segmentation architectures, respectively. Two loss function combinations, Dice-Cross Entropy and Dice-Focal Loss were experimented with for both of the models. The dataset was split into training and validation subsets with an 80:20 distribution on a patient basis in a 5-fold cross-validation approach. To introduce a sample variability to the dataset, strong-augmentation techniques were performed using intensity modifications and perspective transformations to represent different surgery environment scenarios. The models were evaluated by Dice Similarity Coefficient (DSC) and Intersection over Union (IoU) which are widely used segmentation metrics. For pixelwise classification performance, accuracy, sensitivity and specificity metrics were calculated on the validation subset. Results Out of 20 videos, 1000 images were extracted, and the anatomical landmarks (liver, IVC, and right adrenal gland) were annotated. Randomly distributed 800 images and 200 images were selected for the training and validation subsets, respectively. Our benchmark results show that the utilization of Dice-Cross Entropy Loss with the transformer-based SwinUNETR model achieved 78.37%, whereas the CNN-based MedNeXt model reached a 77.09% mDSC score. Conversely, MedNeXt reaches a higher mIoU score of 63.71% than SwinUNETR by 62.10% on a three-region prediction task. Conclusion Artificial intelligence-based systems can predict anatomical landmarks with high performance in minimally invasive right adrenalectomy. Such tools can later be used to create real-time navigation systems during surgery in the near future.

Deep learning has the potential to augment clinician performance in medical imaging interpretation and reduce time to diagnosis through automated segmentation. Few studies to date have explored this topic. To develop and apply a neural network segmentation model (the HeadXNet model) capable of generating precise voxel-by-voxel predictions of intracranial aneurysms on head computed tomographic angiography (CTA) imaging to augment clinicians' intracranial aneurysm diagnostic performance. In this diagnostic study, a 3-dimensional convolutional neural network architecture was developed using a training set of 611 head CTA examinations to generate aneurysm segmentations. Segmentation outputs from this support model on a test set of 115 examinations were provided to clinicians. Between August 13, 2018, and October 4, 2018, 8 clinicians diagnosed the presence of aneurysm on the test set, both with and without model augmentation, in a crossover design using randomized order and a 14-day washout period. Head and neck examinations performed between January 3, 2003, and May 31, 2017, at a single academic medical center were used to train, validate, and test the model. Examinations positive for aneurysm had at least 1 clinically significant, nonruptured intracranial aneurysm. Examinations with hemorrhage, ruptured aneurysm, posttraumatic or infectious pseudoaneurysm, arteriovenous malformation, surgical clips, coils, catheters, or other surgical hardware were excluded. All other CTA examinations were considered controls. Sensitivity, specificity, accuracy, time, and interrater agreement were measured. Metrics for clinician performance with and without model augmentation were compared. The data set contained 818 examinations from 662 unique patients with 328 CTA examinations (40.1%) containing at least 1 intracranial aneurysm and 490 examinations (59.9%) without intracranial aneurysms. The 8 clinicians reading the test set ranged in experience from 2 to 12 years. Augmenting clinicians with artificial intelligence-produced segmentation predictions resulted in clinicians achieving statistically significant improvements in sensitivity, accuracy, and interrater agreement when compared with no augmentation. The clinicians' mean sensitivity increased by 0.059 (95% CI, 0.028-0.091; adjusted P = .01), mean accuracy increased by 0.038 (95% CI, 0.014-0.062; adjusted P = .02), and mean interrater agreement (Fleiss κ) increased by 0.060, from 0.799 to 0.859 (adjusted P = .05). There was no statistically significant change in mean specificity (0.016; 95% CI, -0.010 to 0.041; adjusted P = .16) and time to diagnosis (5.71 seconds; 95% CI, 7.22-18.63 seconds; adjusted P = .19). The deep learning model developed successfully detected clinically significant intracranial aneurysms on CTA. This suggests that integration of an artificial intelligence-assisted diagnostic model may augment clinician performance with dependable and accurate predictions and thereby optimize patient care.

To resolve the finite computing precision problem in chaotic cryptography, and the security and efficiency drawbacks of deep learning-based image cryptosystems, an image encryption framework with low computing precision is presented based on three-dimensional Boolean convolution neural network (3D-BCNN). Unlike traditional CNN, the proposed 3D-BCNN is composed of only convolutional layers, in which a cross-channel 3-D Boolean convolution operation is devised without training and parameter optimization. To strengthen the security and sensitivity of the cryptosystem, the kernels and convolutional matrices are generated by the combined prime modulo multiplicative linear congruence generators, SHA-1, and a plain-image. All operations of 3D-BCNN can be conducted on devices with 8-bit word length, so the proposed encryption scheme could work well with low computing precision. Simulation results demonstrate that, with the largest computing accuracy 2−8, the proposed encryption scheme has high security and low time complexity, and can withstand various attacks.

Understanding human behavior and human action recognition are both essential components of effective surveillance video analysis for the purpose of guaranteeing public safety. However, existing approaches such as three-dimensional convolutional neural networks (3D CNN) and two-stream neural networks (2SNN) have computational hurdles due to the significant parameterization they require. In this paper, we offer HARNet, a specialized lightweight residual 3D CNN that is built on directed acyclic graphs and was created expressly to handle these issues and achieve effective human action detection. The suggested method presents an innovative pipeline for creating spatial motion data from raw video inputs, which makes successful latent representation learning of human motions easier to accomplish. This generated input is then supplied into HARNet, which processes spatial and motion information in a single stream in an effective manner, maximizing the benefits of both types of cues. The use of traditional machine learning classifiers is done in order to further improve the discriminative capacity of the features that have been learned. To be more specific, we use the latent representations that are stored in HARNet’s fully connected layer and use them as our deep learnt features. After that, these features are entered into the Support Vector Machine (SVM) classifier in order to accomplish action recognition. In order to evaluate the HARNet-SVM method that was developed, empirical tests were run on commonly used action recognition datasets such as UCF101, HMDB51, and the KTH dataset. These tests were carried out in order to gather data for the evaluation. The experimental results show that our method is superior to other state-of-the-art approaches, achieving considerable performance increases of 2.75% on UCF101, 10.94% on HMDB51, and 0.18% on the KTH dataset. These results were obtained by running the method on each dataset separately. Our findings demonstrate the usefulness of HARNet’s lightweight design and highlight the significance of utilizing SVM classifiers with deep learnt features for the purpose of accurate and computationally efficient human activity recognition in surveillance videos. This work helps to the advancement of surveillance technology, which in turn makes video analysis in applications that take place in the real world safer and more dependable.

Evolutionary decision fusion has applications in biometric authentication and verification. Gray wolf optimizer (GWO) is one such evolutionary decision fusion approach that can be used to tune the fusion parameters in a multimodal data acquisition system. Human gait is a proven biometric trait with applications in security and authentication. However, acquiring human-gait data can be erroneous due to various factors and multimodal fusion of such erroneous gait data can be challenging. In this paper, we propose a new decision fusion-based approach to solve the above problem. Gait data is recorded simultaneously using motion sensors and visible-light camera. The signals of the motion sensors are modeled using a long short-term memory neural network and corresponding video recordings are processed using a three-dimensional convolutional neural network. GWO has been used to optimize the parameters during fusion. It has been chosen based on the underlying hunting strategy that leads to better approximation of the solution. Interestingly, in our case it converges quicker than other optimization techniques such as genetic algorithm or particle swarm optimization. To test the model, a dataset involving 23 males and females has been recorded while they perform four different types of walks, including, normal walk, fast walk, walking while listening to music, and walking while watching multimedia content on a mobile. An overall accuracy of 91.3% has been recorded across all test scenarios. Results reveal that the proposed study can further be explored to design robust gait biometric systems.

Accurate identification of proteins interacted with drugs helps reduce the time and cost of drug development. Most of previous methods focused on integrating multisource data about drugs and proteins for predicting drug-target interactions (DTIs). There are both similarity connection and interaction connection between two drugs, and these connections reflect their relationships from different perspectives. Similarly, two proteins have various connections from multiple perspectives. However, most of previous methods failed to deeply integrate these connections. In addition, multiple drug-protein heterogeneous networks can be constructed based on multiple kinds of connections. The diverse topological structures of these networks are still not exploited completely. We propose a novel model to extract and integrate multi-type neighbor topology information, diverse similarities and interactions related to drugs and proteins. Firstly, multiple drug-protein heterogeneous networks are constructed according to multiple kinds of connections among drugs and those among proteins. The multi-type neighbor node sequences of a drug node (or a protein node) are formed by random walks on each network and they reflect the hidden neighbor topological structure of the node. Secondly, a module based on graph neural network (GNN) is proposed to learn the multi-type neighbor topologies of each node. We propose attention mechanisms at neighbor node level and at neighbor type level to learn more informative neighbor nodes and neighbor types. A network-level attention is also designed to enhance the context dependency among multiple neighbor topologies of a pair of drug and protein nodes. Finally, the attribute embedding of the drug-protein pair is formulated by a proposed embedding strategy, and the embedding covers the similarities and interactions about the pair. A module based on three-dimensional convolutional neural networks (CNN) is constructed to deeply integrate pairwise attributes. Extensive experiments have been performed and the results indicate GCDTI outperforms several state-of-the-art prediction methods. The recall rate estimation over the top-ranked candidates and case studies on 5 drugs further demonstrate GCDTI's ability in discovering potential drug-protein interactions.

The current action recognition analysis method is easily affected by factors such as background, illumination, and target angle, which not only has low accuracy, but also relies on prior knowledge. Research on the identification and analysis of technical and tactical movements in football. According to the characteristics of football video, a multi-resolution three-dimensional convolutional neural network is constructed by combining the convolutional neural network and the three-dimensional neural network. The supervised training algorithm is used to update the network weights and thresholds, and the video images are input into the input layer. After the convolutional layer, sub-sampling layer and fully connected layer and other network layers to obtain action recognition results. The principal component analysis method is used to reduce the dimension to process the action data set, and the Fourier transform method is used to filter out the principal component noise. The experimental results show that the method can effectively identify the technical and tactical movements of athletes from complex football game videos, and analyze the applied technical and tactical strategies. The average value of accuracy, recall and precision of technical and tactical analysis is as high as 0.96, 0.97, and 0.95, and the recognition and analysis effect has significant advantages.