Research on deep learning image segmentation method based on attention mechanism

IntroductionNarrow Band Imaging (NBI) is an endoscopic visualization technique useful for upper aero-digestive tract (UADT) cancer detection and margins evaluation. However, NBI analysis is strongly operator-dependent and requires high expertise, thus limiting its wider implementation. Recently, artificial intelligence (AI) has demonstrated potential for applications in UADT videoendoscopy. Among AI methods, deep learning algorithms, and especially convolutional neural networks (CNNs), are particularly suitable for delineating cancers on videoendoscopy. This study is aimed to develop a CNN for automatic semantic segmentation of UADT cancer on endoscopic images.Materials and MethodsA dataset of white light and NBI videoframes of laryngeal squamous cell carcinoma (LSCC) was collected and manually annotated. A novel DL segmentation model (SegMENT) was designed. SegMENT relies on DeepLabV3+ CNN architecture, modified using Xception as a backbone and incorporating ensemble features from other CNNs. The performance of SegMENT was compared to state-of-the-art CNNs (UNet, ResUNet, and DeepLabv3). SegMENT was then validated on two external datasets of NBI images of oropharyngeal (OPSCC) and oral cavity SCC (OSCC) obtained from a previously published study. The impact of in-domain transfer learning through an ensemble technique was evaluated on the external datasets.Results219 LSCC patients were retrospectively included in the study. A total of 683 videoframes composed the LSCC dataset, while the external validation cohorts of OPSCC and OCSCC contained 116 and 102 images. On the LSCC dataset, SegMENT outperformed the other DL models, obtaining the following median values: 0.68 intersection over union (IoU), 0.81 dice similarity coefficient (DSC), 0.95 recall, 0.78 precision, 0.97 accuracy. For the OCSCC and OPSCC datasets, results were superior compared to previously published data: the median performance metrics were, respectively, improved as follows: DSC=10.3% and 11.9%, recall=15.0% and 5.1%, precision=17.0% and 14.7%, accuracy=4.1% and 10.3%.ConclusionSegMENT achieved promising performances, showing that automatic tumor segmentation in endoscopic images is feasible even within the highly heterogeneous and complex UADT environment. SegMENT outperformed the previously published results on the external validation cohorts. The model demonstrated potential for improved detection of early tumors, more precise biopsies, and better selection of resection margins.

Modeling accurate text recognizers for scanned Devanagari scripts is a challenging task, primarily due to the ingrained complexities of characters, vowels, conjuncts, and modifiers in the word formation of the script. This paper presents an attention-based convolutional recurrent neural network (CRNN) for improving the text recognition accuracy of the Devanagari script images. The proposed model is an encoder–decoder network with an attention mechanism. The encoder is a convolutional neural network (CNN), which extracts the significant and high-level features from the input images. The encoder also generates the extracted features in an ordered sequence. The decoder uses a bidirectional long short-term (BLSTM) network with an attention mechanism, followed by a connectionist temporal classification (CTC) layer to predict the text in images. The attention mechanism enables the decoder to selectively exploit the high-level features of the encoder congruently. We have trained, validated, and tested the performance of the proposed model on three datasets, viz. publicly available IIIT-H scene image dataset, synthetic dataset, and real dataset curated during this work. We have also compared the proposed model results with work already attempted in the area for the three datasets. We found that the performance of our proposed model outdoes the outcomes obtained from other models on the three datasets.

Convolutional Neural Networks (CNNs) have become the dominant paradigm in computer vision since the ImageNet breakthrough, establishing a solid engineering foundation in both cloud and edge scenarios. Addressing the current status where existing reviews often focus on a single dimension and lack an integrated engineering perspective, this paper systematically reviews the architectural evolution, key components, and application practices of modern CNNs, primarily focusing on image classification. First, the paper elucidates the design principles of key components, including convolution and receptive fields (RF) , normalization and activation functions , and attention mechanisms. It then traces the evolution path of modern hybrid backbones, from AlexNet/ResNet to MobileNet/EfficientNet, and further to ConvNeXt , following a timeline of "deepening and residualization – lightweight and automated scaling – large kernels and Transformer fusion". Second, by integrating typical application scenarios such as object detection, semantic segmentation, and super-resolution , this review distills reusable training recipes and efficiency optimization strategies involving the synergistic use of pruning, quantization, and distillation. It also provides a checklist for evaluation and deployment geared toward actual hardware. Finally, the article analyzes challenges such as long-tail categories, cross-domain distribution shift, and on-device computational constraints, and looks forward to future trends in self-supervised learning, hardware-aware design, and model robustness optimization.

Accurate segmentation is a basic and crucial step for medical image processing and analysis. In the last few years, U-Net, and its variants, have become widely adopted models in medical image segmentation tasks. However, the multiple training parameters of these models determines high computation complexity, which is impractical for further applications. In this paper, by introducing depthwise separable convolution and attention mechanism into U-shaped architecture, we propose a novel lightweight neural network (DSCA-Net) for medical image segmentation. Three attention modules are created to improve its segmentation performance. Firstly, Pooling Attention (PA) module is utilized to reduce the loss of consecutive down-sampling operations. Secondly, for capturing critical context information, based on attention mechanism and convolution operation, we propose Context Attention (CA) module instead of concatenation operations. Finally, Multiscale Edge Attention (MEA) module is used to emphasize multi-level representative scale edge features for final prediction. The number of parameters in our network is 2.2 M, which is 71.6% less than U-Net. Experiment results across four public datasets show the potential and the dice coefficients are improved by 5.49% for ISIC 2018, 4.28% for thyroid, 1.61% for lung and 9.31% for nuclei compared with U-Net.

Backgroundand Purpose:This study aimed to develop and evaluate an efficient method to automatically segment T1- and T2-weighted brain magnetic resonance imaging (MRI) images. We specifically compared the segmentation performance of individual convolutional neural network (CNN) models against an ensemble approach to advance the accuracy of MRI-guided radiotherapy (RT) planning.Materials and Methods. The evaluation was conducted on a private clinical dataset and a publicly available dataset (HaN-Seg). Anonymized MRI data from 55 brain cancer patients, including T1-weighted, T1-weighted with contrast, and T2-weighted images, were used in the clinical dataset. We employed an EDL strategy that integrated five independently trained 2D neural networks, each tailored for precise segmentation of tumors and organs at risk (OARs) in the MRI scans. Class probabilities were obtained by averaging the final layer activations (Softmax outputs) from the five networks using a weighted-average method, which were then converted into discrete labels. Segmentation performance was evaluated using the Dice similarity coefficient (DSC) and Hausdorff distance at 95% (HD95). The EDL model was also tested on the HaN-Seg public dataset for comparison.Results. The EDL model demonstrated superior segmentation performance on both the clinical and public datasets. For the clinical dataset, the ensemble approach achieved an average DSC of 0.7 ± 0.2 and HD95 of 4.5 ± 2.5 mm across all segmentations, significantly outperforming individual networks which yielded DSC values ≤0.6 and HD95 values ≥14 mm. Similar improvements were observed in the HaN-Seg public dataset.Conclusions. Our study shows that the EDL model consistently outperforms individual CNN networks in both clinical and public datasets, demonstrating the potential of ensemble learning to enhance segmentation accuracy. These findings underscore the value of the EDL approach for clinical applications, particularly in MRI-guided RT planning.

To satisfy the need to accurately monitor emotional stress, this paper explores the effectiveness of the attention mechanism based on the deep learning model CNN (Convolutional Neural Networks)-BiLSTM (Bi-directional Long Short-Term Memory) As different attention mechanisms can cause the framework to focus on different positions of the feature map, this discussion adds attention mechanisms to the CNN layer and the BiLSTM layer separately, and to both the CNN layer and BiLSTM layer simultaneously to generate different CNN–BiLSTM networks with attention mechanisms. ECG (electrocardiogram) data from 34 subjects were collected on the server platform created by the Institute of Psychology of the Chinese Academy of Science and the researches. It verifies that the average accuracy of CNN–BiLSTM is up to 0.865 without any attention mechanism, while the highest average accuracy of 0.868 is achieved using the CNN–attention–based BiLSTM.

A dual-branch hybrid dilated CNN model for the AI-assisted segmentation of meningiomas in MR images

Knowing the shape of vascular calcifications is crucial for appropriate planning and conductance of percutaneous coronary interventions. The clinical workflow can therefore benefit from automatic segmentation of calcified plaques in intravascular ultrasound (IVUS) images. To solve segmentation problems with convolutional neural networks (CNNs), large datasets are usually required. However, datasets are often rather small in the medical domain. Hence, developing and investigating methods for increasing CNN performance on small datasets can help on the way towards clinically relevant results. We compared two state-of-the-art CNN architectures for segmentation, U-Net and DeepLabV3, and investigated how incorporating auxiliary image data with vessel wall and lumen annotations improves the calcium segmentation performance by using these either for pretraining or multi-task training. DeepLabV3 outperforms U-Net with up to 6.3 % by means of the Dice coefficient and 36.5 % by means of the average Hausdorff distance. Using auxiliary data improves the segmentation performance in both cases, whereas the multi-task approach outperforms the pre-training approach. The improvements of the multi-task approach in contrast to not using auxiliary data at all is 5.7 % for the Dice coefficient and 42.9 % for the average Hausdorff distance. Automatic segmentation of calcified plaques in IVUS images is a demanding task due to their relatively small size compared to the image dimensions and due to visual ambiguities with other image structures. We showed that this problem can generally be tackled by CNNs. Furthermore, we were able to improve the performance by a multi-task learning approach with auxiliary segmentation data.

Convolutional neural networks (CNNs) show potential for delineating cancers on contrast-enhanced MRI (ce-MRI) but there are clinical scenarios in which administration of contrast is not desirable. We investigated performance of the CNN for delineating primary nasopharyngeal carcinoma (NPC) on non-contrast-enhanced images and compared the performance to that on ce-MRI. We retrospectively analyzed primary NPC in 195 patients using a well-established CNN, U-Net, for tumor delineation on the non-contrast-enhanced fat-suppressed (fs)-T2W, ce-T1W and ce-fs-T1W images. The CNN-derived delineations were compared to manual delineations to obtain Dice similarity coefficient (DSC) and average surface distance (ASD). The DSC and ASD on fs-T2W were compared to those on ce-MRI. Primary tumor volumes (PTVs) of CNN-derived delineations were compared to that of manual delineations. The CNN for NPC delineation on fs-T2W images showed similar DSC (0.71 ± 0.09) and ASD (0.21 ± 0.48cm) to those on ce-T1W images (0.71 ± 0.09 and 0.17 ± 0.19cm, respectively) (p > 0.05), and lower DSC but similar ASD to ce-fs-T1W images (0.73 ± 0.09, p < 0.001; and 0.17 ± 0.20cm, p > 0.05). The CNN overestimated PTVs on all sequences (p < 0.001). The CNN showed promise for NPC delineation on fs-T2W images in cases where it is desirable to avoid contrast agent injection. The CNN overestimated PTVs on all sequences.

To assess whether octree representation and octree-based convolutional neural networks (CNNs) improve segmentation accuracy of three-dimensional images. Cardiac CT angiographic examinations from 100 patients (mean age, 67 years ± 17 [standard deviation]; 60 men) performed between June 2012 and June 2018 with semantic segmentations of the left ventricular (LV) and left atrial (LA) blood pools at the end-diastolic and end-systolic cardiac phases were retrospectively evaluated. Image quality (root mean square error [RMSE]) and segmentation fidelity (global Dice and border Dice coefficients) metrics of the octree representation were compared with spatial downsampling for a range of memory footprints. Fivefold cross-validation was used to train an octree-based CNN and CNNs with spatial downsampling at four levels of image compression or spatial downsampling. The semantic segmentation performance of octree-based CNN (OctNet) was compared with the performance of U-Nets with spatial downsampling. Octrees provided high image and segmentation fidelity (median RMSE, 1.34 HU; LV Dice coefficient, 0.970; LV border Dice coefficient, 0.843) with a reduced memory footprint (87.5% reduction). Spatial downsampling to the same memory footprint had lower data fidelity (median RMSE, 12.96 HU; LV Dice coefficient, 0.852; LV border Dice coefficient, 0.310). OctNet segmentation improved the border segmentation Dice coefficient (LV, 0.612; LA, 0.636) compared with the highest performance among U-Nets with spatial downsampling (Dice coefficients: LV, 0.579; LA, 0.592). Octree-based representations can reduce the memory footprint and improve segmentation border accuracy.Keywords CT, Cardiac, Segmentation, Supervised Learning, Convolutional Neural Network (CNN), Deep Learning Algorithms, Machine Learning Algorithms© RSNA, 2021.

Identifying the massage techniques of the masseuse is a prerequisite for guiding robotic massage. It is difficult to recognize multiple consecutive massage maps with a time series for current human action recognition algorithms. To solve the problem, a method combining a convolutional neural network, long-term neural network, and attention mechanism is proposed to identify the massage techniques in this paper. First, the pressure distribution massage map is collected by a massage glove, and the data are enhanced by the conditional variational auto-encoder. Then, the features of the massage map group in the spatial domain and timing domain are extracted through the convolutional neural network and the long- and short-term memory neural network, respectively. The attention mechanism is introduced into the neural network, giving each massage map a different weight value to enhance the network extraction of data features. Finally, the massage haptic dataset is collected by a massage data acquisition system. The experimental results show that a classification accuracy of 100% is achieved. The results demonstrate that the proposed method could identify sequential massage maps, improve the network overfitting phenomenon, and enhance the network generalization ability effectively.

People deploy top-down, goal-directed attention to accomplish tasks, such as finding lost keys. By tuning the visual system to relevant information sources, object recognition can become more efficient (a benefit) and more biased toward the target (a potential cost). Motivated by selective attention in categorisation models, we developed a goal-directed attention mechanism that can process naturalistic (photographic) stimuli. Our attention mechanism can be incorporated into any existing deep convolutional neural networks (DCNNs). The processing stages in DCNNs have been related to ventral visual stream. In that light, our attentional mechanism incorporates top-down influences from prefrontal cortex (PFC) to support goal-directed behaviour. Akin to how attention weights in categorisation models warp representational spaces, we introduce a layer of attention weights to the mid-level of a DCNN that amplify or attenuate activity to further a goal. We evaluated the attentional mechanism using photographic stimuli, varying the attentional target. We found that increasing goal-directed attention has benefits (increasing hit rates) and costs (increasing false alarm rates). At a moderate level, attention improves sensitivity (i.e. increases d^{prime }) at only a moderate increase in bias for tasks involving standard images, blended images and natural adversarial images chosen to fool DCNNs. These results suggest that goal-directed attention can reconfigure general-purpose DCNNs to better suit the current task goal, much like PFC modulates activity along the ventral stream. In addition to being more parsimonious and brain consistent, the mid-level attention approach performed better than a standard machine learning approach for transfer learning, namely retraining the final network layer to accommodate the new task.

Heterogeneity is a hallmark of cancer and perhaps one of the most important features associated with resistance to therapies and likelihood of recurrence and/or metastasis. Genomic instability contributes to genetic diversity, which leads to high levels of intratumoural heterogeneity. While genetic diversity is one contributor to phenotypic heterogeneity in tumors, the spatial variation in the tumor microenvironment (TME) also drives emergent phenotypic heterogeneity and intrinsic variation in resistance to drugs and the immune system. Ultimately, heterogeneity leads to higher likelihood of metastasis. Traditionally, pathology allowed for the assessment of heterogeneity. However, this was limited to gross assessment of the TME, through measurement of mitotic levels and severity of cancer invasion. We present SimBioSys PhenoScope, a multi-scale analysis and visualization platform that integrates cancer data across scales to extract cross modality trends that drive cancer invasion. As a demonstrated use case for the platform, we present a vignette of using the platform to analyze pathology slides at three scales. Three convolutional neural networks (CNN) are developed and validated. The outputs of these networks were combined with 2D simulations of the metabolic behavior and growth of cells within the TME. Two CNNs were developed and one implemented: one that identifies cells undergoing mitosis, one that segments individual cells and classifies their type, and one that segments five tissues from pathology slides. Additionally, transcriptional data was used to generate patient specific metabolic models. Each CNN was developed using training images and validated on the test images. The mitosis detection CNN was found to have an accuracy of 76.2% (precision=83%, recall=76%) in the test set. The classification CNN was found to have a Dice Similarity Coefficient (DSC) of 0.821 for segmenting cells and an F1 score for classifying cells ranging from 0.559 (F1i) to 0.756 (F1d). The segmentation CNN was found to have an accuracy of 78.1% with DSC ranging from 0.66 and 0.86 depending on tissue. The segmentations were input into a proprietary simulation framework along with patient-specific metabolic models to predict the spatial gradients of nutrients, and spatial organization of growth and metabolism. Simulations show multiple behaviors such as regions of high lactate production or consumption by cancer, and regions that differ by lactate production and alanine uptake in cancers. These behaviors were correlated with local cell mitoses and invasion of Tumor Infiltrating Lymphocytes. Tools to examine cancers across scales and within the TME are currently lacking. We demonstrate a proof-of-principle approach of combining data across scales in a fashion that allows for novel predictions of TME behavior. Citation Format: Joseph R. Peterson, Matthew Shin, Patricia Carrigan, Michael J. Hallock, Snehal Patel, The SimBioSys Team. A multi-scale analysis and visualization platform for cancer data - deriving tumor microenvironment behavior from pathology and transcriptomics [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2022; 2022 Apr 8-13. Philadelphia (PA): AACR; Cancer Res 2022;82(12_Suppl):Abstract nr 469.

BackgroundThis study aimed to (1) develop a fully residual deep convolutional neural network (CNN)-based segmentation software for computed tomography image segmentation of the male pelvic region and (2) demonstrate its efficiency in the male pelvic region.MethodsA total of 470 prostate cancer patients who had undergone intensity-modulated radiotherapy or volumetric-modulated arc therapy were enrolled. Our model was based on FusionNet, a fully residual deep CNN developed to semantically segment biological images. To develop the CNN-based segmentation software, 450 patients were randomly selected and separated into the training, validation and testing groups (270, 90, and 90 patients, respectively). In Experiment 1, to determine the optimal model, we first assessed the segmentation accuracy according to the size of the training dataset (90, 180, and 270 patients). In Experiment 2, the effect of varying the number of training labels on segmentation accuracy was evaluated. After determining the optimal model, in Experiment 3, the developed software was used on the remaining 20 datasets to assess the segmentation accuracy. The volumetric dice similarity coefficient (DSC) and the 95th-percentile Hausdorff distance (95%HD) were calculated to evaluate the segmentation accuracy for each organ in Experiment 3.ResultsIn Experiment 1, the median DSC for the prostate were 0.61 for dataset 1 (90 patients), 0.86 for dataset 2 (180 patients), and 0.86 for dataset 3 (270 patients), respectively. The median DSCs for all the organs increased significantly when the number of training cases increased from 90 to 180 but did not improve upon further increase from 180 to 270. The number of labels applied during training had a little effect on the DSCs in Experiment 2. The optimal model was built by 270 patients and four organs. In Experiment 3, the median of the DSC and the 95%HD values were 0.82 and 3.23 mm for prostate; 0.71 and 3.82 mm for seminal vesicles; 0.89 and 2.65 mm for the rectum; 0.95 and 4.18 mm for the bladder, respectively.ConclusionsWe have developed a CNN-based segmentation software for the male pelvic region and demonstrated that the CNN-based segmentation software is efficient for the male pelvic region.

A novel network with multiple attention mechanisms for aspect-level sentiment analysis