ASFNet: Adaptive multiscale segmentation fusion network for real‐time semantic segmentation

The purpose of crowd counting is to estimate the number of pedestrians in crowd images. Crowd counting or density estimation is an extremely challenging task in computer vision, due to large scale variations and dense scene. Current methods solve these issues by compounding multi-scale Convolutional Neural Network with different receptive fields. In this paper, a novel end-to-end architecture based on Multi-Scale Adversarial Convolutional Neural Network (MSA-CNN) is proposed to generate crowd density and estimate the amount of crowd. Firstly, a multi-scale network is used to extract the globally relevant features in the crowd image, and then fractionally-strided convolutional layers are designed for up-sampling the output to recover the loss of crucial details caused by the earlier max pooling layers. An adversarial loss is directly employed to shrink the estimated value into the realistic subspace to reduce the blurring effect of density estimation. Joint training is performed in an end-to-end fashion using a combination of Adversarial loss and Euclidean loss. The two losses are integrated via a joint training scheme to improve density estimation performance.We conduct some extensive experiments on available datasets to show the significant improvements and supremacy of the proposed approach over the available state-of-the-art approaches.

In view of the insufficient illumination of low-light images in semantic segmentation, there are phenomena such as low image brightness, contrast, and signal-to-noise ratio, which makes semantic segmentation of low-light images more difficult. A low-light image semantic segmentation method based on vector symbolic architectures domain adaptation network is proposed. The method first integrates the image translation network based on vector symbolic architectures and the domain adaptive semantic segmentation network. The image translation network translates normal illumination images into low-light images, reducing style differences between domains and improving the segmentation accuracy of low-light images. Then, a layered feature mapping (LFM) module is proposed in the image translation network. The LFM module can better map the multi-layer feature vectors of the semantic segmentation network encoder to the VSA space, making the translated image more similar to the low-light image, thereby improving the segmentation accuracy of the segmentation network. Finally, a cross domain rare class mixing (CDRCM) method is proposed in the domain adaptive semantic segmentation network. CDRCM obtains rare classes in the low-light domain based on the pseudo- label distribution of low-light images during domain adaptation, and biases towards these rare classes during cross-domain mixing, thereby improving the segmentation accuracy of rare classes in the low-light domain. The experimental results on Cityscapes→Dark Zurich and Cityscapes→ACDC-night indicated that the proposed method in the paper achieved improvements of 8.2 percentage points and 7.2 percentage points, respectively, in mean intersection-over-union compared to the baseline DAFormer, the experimental results show the effectiveness of the method.

Augmenting RGB data with measured depth has been shown to improve the performance of a range of tasks in computer vision, including object detection and semantic segmentation. Although depth sensors such as the Microsoft Kinect have facilitated easy acquisition of such depth information, the vast majority of images used in vision tasks do not contain depth information. In this paper, we show that augmenting RGB images with estimated depth can also improve the accuracy of both object detection and semantic segmentation. Specifically, we first exploit the recent success of depth estimation from monocular images and learn a deep depth estimation model. Then, we learn deep depth features from the estimated depth and combine with RGB features for object detection and semantic segmentation. In addition, we propose an RGB-D semantic segmentation method, which applies a multi-task training scheme: semantic label prediction and depth value regression. We test our methods on several data sets and demonstrate that incorporating information from estimated depth improves the performance of object detection and semantic segmentation remarkably.

Semantic segmentation is an extremely important task in computer vision. At present, the related methods have achieved high performance. Nevertheless, Semantic segmentation still faces the challenge of localization accuracy due to DCNN invariance and existence of objects at multi-scale. In order to improve the accuracy of segmentation, this paper proposes a U-SEM encoder-decoder network. Firstly, in the encoding stage, it down-samples through the ResNet. Secondly, in the decoding stage, in order to filter and utilize the useful features, the SE-Mobile Block is proposed and fused to the network. The SE block adopts the idea of attention mechanism to focus on useful features and ignore those redundant features. Mobile blocks use deep separable convolutions to replace traditional convolutions, speeding up operations and reducing parameters. Finally, it adopts the skip structure where the feature information of different scales are merged to produce accurate and detailed segmentation. Experimental results show that the proposed network achieves good performance on multiple datasets which reaches the accuracy of 78.4% mIOU on PASCAL VOC 2012 and 75.7% mIOU on Cityscapes dataset.

Semantic segmentation is a classical task in computer vision. In this paper, we target to address the low confidence regions which traditional CNN can not solve very well in semantic segmentation task. Depending on different characteristics of low confidence regions, an adaptive and robust attention mechanism is important to focus on the informative regions but ignore the noisy parts in the image. Intuitively, one attention map only is not sufficient to model the interaction between the low confidence regions and its surrounding patches. Thus, in this paper, we propose an Attention Forest structure, a novel and robust attention mechanism, to handle the low confidence regions. Each Attention Tree structure can capture more interactions between current patches with its adjacent regions. Experiments on PASCAL VOC 2012 Dataset validate the effectiveness of our proposed algorithm.

As a cultural heritage, traditional dance must be protected and preserved. Pendet dance is a traditional dance from Bali, Indonesia. Dance recognition raises a complex problem for computer vision research because the features representing the dancer must focus on the dancer's entire body. This can be done by performing a segmentation task process. One type of segmentation task in computer vision is the semantic segmentation. Mask R-CNN and U-NET were employed in this task. Since it was first introduced in 2015, semantic segmentation using the U-Net architecture has been widely adopted, developed, and modified. One of the new architectures applied is the MultiRes UNet. This study carries out a semantic segmentation task on the Balinese Pendet dance image using the MultiRes UNet architecture by changing the value of α (alpha) to obtain the best results. This architectural is evaluated by DC score, Jaccard index, and MSE. In this dataset, the alpha value of 1.9 resulted in the best score for DC and the Jaccard index with 98.47% and 99.23% respectively. On the other hand, an alpha value of 1.8 obtained the best score of MSE with 8.20E-04.

SegCFT: Context-aware Fourier Transform for efficient semantic segmentation

Video object detection is a fundamental yet challenging task in computer vision. Recently, DETR-based methods have gained prominence in this domain owing to their powerful global modeling capabilities. However, these methods are still confronted with two key limitations: frame-agnostic initialization of object queries and scale-agnostic attention mechanisms, which hinder their capability to capture the appearance variations of dynamic objects and model the temporal consistency across frames. To alleviate these limitations, we propose a multiscale-aware transformer diffusion network (MSTDiff), a novel framework designed for the video object detection task, including two technical improvements over existing methods. First, we design a diffusion-driven adaptive query module, which models the object query distribution through a diffusion process conditioned on input frames, enabling an adaptive and content-aware initialization of object queries. Second, we develop a multiscale-aware transformer encoder module, which combines multi-head convolutional units with attention mechanisms to enhance multi-scale feature representations while preserving global dependence modeling. We conduct extensive experiments on the public ImageNet VID dataset, and the results demonstrate that our MSTDiff achieves 87.7% mAP with ResNet-101, outperforming most previous state-of-the-art video object detection methods.

Drone semantic segmentation is a challenging task in computer vision, mainly due to inherent complexities associated with aerial imagery. This paper presents a comprehensive methodology for drone semantic segmentation and evaluates its performance using the ICG dataset. The proposed method leverages hierarchical multi-scale feature extraction and efficient channel-based attention Atrous Spatial Pyramid Pooling (ASPP) to address the unique challenges encountered in this domain. In this study, the performance of the proposed method is compared to several state-of-the-art models. The findings of this research highlight the effectiveness of the proposed method in tackling the challenges of drone semantic segmentation. The outcomes demonstrate its superiority over the state-of-the-art models, showcasing its potential for accurate and efficient segmentation of aerial imagery. The results contribute to the advancement of drone-based applications, such as surveillance, object tracking, and environmental monitoring, where precise semantic segmentation is crucial. The obtained experimental results demonstrate that the proposed method outperforms these existing approaches regarding Dice, mIOU, and accuracy metrics. Specifically, the proposed method achieves an impressive performance with Dice, mIOU, and accuracy scores of 86.51%, 76.23%, and 91.74%, respectively.

Semantic segmentation is a core task in computer vision, but the high computational demands of state-of-the-art models hinder deployment in real-time and resource-constrained environments. Multi-exit architectures offer a potential remedy by enabling adaptive inference through early exits at intermediate layers, yet their application to dense prediction tasks remains underexplored. In this work, we extend multi-exit architectures to semantic segmentation using a SegNet backbone with two additional exits and systematically evaluate training strategies, including joint optimization, final exit distillation, and exit-ensemble distillation. Our experiments on the CamVid dataset show that knowledge distillation consistently improves both accuracy and stability, with the exit-ensemble strategy achieving the highest accuracy and final exit distillation offering the most stable convergence. We conduct a systematic evaluation of confidence metrics and identify Maximum Prediction as the most reliable estimator of prediction quality in our setting. Based on this, we apply a prediction recycling strategy that reuses intermediate outputs that would otherwise be discarded. This approach reduces computational waste and improves efficiency, achieving up to a 21.6% speedup over a single-exit baseline with only a 1.87 percentage-point drop in accuracy. These findings demonstrate the viability of multi-exit semantic segmentation for real-time applications and provide a principled foundation for future research on adaptive inference in dense prediction tasks.

Semantic segmentation is a fundamental task in computer vision and is widely used in industry. However, current state-of-the-art architectures usually bring heavy computation complexity, making it hard to meet the demand for real-time, and can not be implemented in industry. In this paper, we propose a lightweight network to complete fast segmentation. Our network follows encoder-decoder style, which encodes rich spatial information at shallow layers and gains sufficient semantic information at deep layers. At the decoder part, we use attention mechanism to re-weight features and gradually fuse high-level features back to low-level features. We evaluate our network on Cityscapes dataset. Our method achieves an accuracy of 68.0 % mean intersection over union, and runs at 50.7 frames per second at full resolution (1024x2048) on one NVIDIA GeForce GTX 1080Ti card.

Aim: Neurovascular abnormalities are extremely complex, due to the multitude of factors acting simultaneously on cerebral hemodynamics. Cerebral Arteriovenous Malformation (CAVM) hemo-dynamic in one of the vascular abnormality condition results changes in the vessels structures and hemodynamics in blood vessels. The challenge is segmenting accurate vessel region to measure hemodynamics of CAVM patients. The clinical procedure is in-vivo method to measure hemodynamics. The catheter-based procedure is difficult, as it is sometimes difficult to reach vessels sub-structures. Methods: In this paper, we have proposed adaptive vessel segmentation based on threshold technique for CAVM patients. We have compared different adaptive methods for vessel segmentation of CAVM structures. The sub-structures are modeled using lumped model to measure hemodynamics non-invasively. Results: Twenty-three CAVM patients with 150 different vessel locations of DSA datasets were studied as part of the adaptive segmentation. 30 simulated data has been evaluated for more than 150 vessels locations for sub-segmentation of vessels. The segmentation results are evaluated with accuracy of 93%. The computed p-value is smaller than the significance level 0.05. Conclusion: The adaptive segmentation using threshold based produces accurate vessel segmentation, results in better accuracy of hemodynamic measurements for DSA images for CAVM patients. The proposed adaptive segmentation helps clinicians to measure hemodynamic non-invasively for the segmented sub-structures of vessels.

In the human brain, the incoming light to the retina is transformed into meaningful representations that allow us to interact with the world. In a similar vein, the RGB pixel values are transformed by a deep neural network (DNN) into meaningful representations relevant to solving a computer vision task it was trained for. Therefore, in my research, I aim to reveal insights into the visual representations in the human visual cortex and DNNs solving vision tasks. In the previous decade, DNNs have emerged as the state-of-the-art models for predicting neural responses in the human and monkey visual cortex. Research has shown that training on a task related to a brain region’s function leads to better predictivity than a randomly initialized network. Based on this observation, we proposed that we can use DNNs trained on different computer vision tasks to identify functional mapping of the human visual cortex. To validate our proposed idea, we first investigate a brain region occipital place area (OPA) using DNNs trained on scene parsing task and scene classification task. From the previous investigations about OPA’s functions, we knew that it encodes navigational affordances that require spatial information about the scene. Therefore, we hypothesized that OPA’s representation should be closer to a scene parsing model than a scene classification model as the scene parsing task explicitly requires spatial information about the scene. Our results showed that scene parsing models had representation closer to OPA than scene classification models thus validating our approach. We then selected multiple DNNs performing a wide range of computer vision tasks ranging from low-level tasks such as edge detection, 3D tasks such as surface normals, and semantic tasks such as semantic segmentation. We compared the representations of these DNNs with all the regions in the visual cortex, thus revealing the functional representations of different regions of the visual cortex. Our results highly converged with previous investigations of these brain regions validating the feasibility of the proposed approach in finding functional representations of the human brain. Our results also provided new insights into underinvestigated brain regions that can serve as starting hypotheses and promote further investigation into those brain regions. We applied the same approach to find representational insights about the DNNs. A DNN usually consists of multiple layers with each layer performing a computation leading to the final layer that performs prediction for a given task. Training on different tasks could lead to very different representations. Therefore, we first investigate at which stage does the representation in DNNs trained on different tasks starts to differ. We further investigate if the DNNs trained on similar tasks lead to similar representations and on dissimilar tasks lead to more dissimilar representations. We selected the same set of DNNs used in the previous work that were trained on the Taskonomy dataset on a diverse range of 2D, 3D and semantic tasks. Then, given a DNN trained on a particular task, we compared the representation of multiple layers to corresponding layers in other DNNs. From this analysis, we aimed to reveal where in the network architecture task-specific representation is prominent. We found that task specificity increases as we go deeper into the DNN architecture and similar tasks start to cluster in groups. We found that the grouping we found using representational similarity was highly correlated with grouping based on transfer learning thus creating an interesting application of the approach to model selection in transfer learning. During previous works, several new measures were introduced to compare DNN representations. So, we identified the commonalities in different measures and unified different measures into a single framework referred to as duality diagram similarity. This work opens up new possibilities for similarity measures to understand DNN representations. While demonstrating a much higher correlation with transfer learning than previous state-of-the-art measures we extend it to understanding layer-wise representations of models trained on the Imagenet and Places dataset using different tasks and demonstrate its applicability to layer selection for transfer learning. In all the previous works, we used the task-specific DNN representations to understand the representations in the human visual cortex and other DNNs. We were able to interpret our findings in terms of computer vision tasks such as edge detection, semantic segmentation, depth estimation, etc. however we were not able to map the representations to human interpretable concepts. Therefore in our most recent work, we developed a new method that associates individual artificial neurons with human interpretable concepts. Overall, the works in this thesis revealed new insights into the representation of the visual cortex and DNNs...

As the core technologies represented by Internet of Things (IoT) technology and computer vision technology booming, object detection algorithms based on deep learning have received high attention in various fields. Object detection is not only a core problem in the field of computer vision, but also a prerequisite and basis for many computer vision tasks, and has important applications in the fields of autonomous driving and video surveillance. The task of object detection is to find all the targets of interest in an image and determine their category and location. Object detection includes two tasks: object classification and object localization. However, object classification focuses on the most discriminative part of the feature map, and object localization requires a feature map focusing on the whole region of the object. Therefore, in order to improve the detection accuracy, we propose a CBAM-RetinaNet network model.CBAM (Convolutional Block Attention Module) is a lightweight convolutional attention module, which combines channel and spatial attention mechanism modules. The channel attention module focuses on the meaningful information in the input image, which would lead to an increase in the accuracy of object classification. The spatial attention module focuses on the location information of the object, which may improve the accuracy of object localization. The experiment results show that the proposed achieves better object detection performance on the PASCAL VOC2007 dataset, and the accuracy is 2% higher than that of the RetinaNet model, which has good experimental results.

Deep learning has already been proved as a powerful state-of-the-art technique for many image understanding tasks in computer vision and other applications including remote sensing (RS) image analysis. Unmanned aircraft systems (UASs) offer a viable and economical alternative to a conventional sensor and platform for acquiring high spatial and high temporal resolution data with high operational flexibility. Coastal wetlands are among some of the most challenging and complex ecosystems for land cover prediction and mapping tasks because land cover targets often show high intra-class and low inter-class variances. In recent years, several deep convolutional neural network (CNN) architectures have been proposed for pixel-wise image labeling, commonly called semantic image segmentation. In this paper, some of the more recent deep CNN architectures proposed for semantic image segmentation are reviewed, and each model’s training efficiency and classification performance are evaluated by training it on a limited labeled image set. Training samples are provided using the hyper-spatial resolution UAS imagery over a wetland area and the required ground truth images are prepared by manual image labeling. Experimental results demonstrate that deep CNNs have a great potential for accurate land cover prediction task using UAS hyper-spatial resolution images. Some simple deep learning architectures perform comparable or even better than complex and very deep architectures with remarkably fewer training epochs. This performance is especially valuable when limited training samples are available, which is a common case in most RS applications.