Self-attentive Semantic Segmentation Model Based On Generative Adversarial Network

A Unified Architecture of Semantic Segmentation and Hierarchical Generative Adversarial Networks for Expression Manipulation

Semantic segmentation plays an increasingly important role in the field of computer vision. The current semantic segmentation algorithms are mainly based on full convolutional network downsampling for feature extraction, however, the reduction of image resolution due to downsampling will inevitably lead to the loss of image information and the process is irreversible. Although features can be reused to compensate for the loss, the problem of lost pixel correlation and inaccurate segmentation results still occur. In order to solve these problems, the attention mechanism and the generative adversarial network are fused into the semantic segmentation network, and a semantic segmentation network is proposed. The attention mechanism is added to the generator and discriminator modules and trained successively, then the generator module with rich information and global information is extracted, and its output is fused with the feature graph sampled above, in order to reduce the loss of local information and complete the task of image segmentation, an interpolation method based on image edge is introduced to sample the image, which improves the bilinear interpolation edge blur. Experiments on Cityscapes dataset show that the proposed network model is effective and reliable.

In view of the insufficient illumination of low-light images in semantic segmentation, there are phenomena such as low image brightness, contrast, and signal-to-noise ratio, which makes semantic segmentation of low-light images more difficult. A low-light image semantic segmentation method based on vector symbolic architectures domain adaptation network is proposed. The method first integrates the image translation network based on vector symbolic architectures and the domain adaptive semantic segmentation network. The image translation network translates normal illumination images into low-light images, reducing style differences between domains and improving the segmentation accuracy of low-light images. Then, a layered feature mapping (LFM) module is proposed in the image translation network. The LFM module can better map the multi-layer feature vectors of the semantic segmentation network encoder to the VSA space, making the translated image more similar to the low-light image, thereby improving the segmentation accuracy of the segmentation network. Finally, a cross domain rare class mixing (CDRCM) method is proposed in the domain adaptive semantic segmentation network. CDRCM obtains rare classes in the low-light domain based on the pseudo- label distribution of low-light images during domain adaptation, and biases towards these rare classes during cross-domain mixing, thereby improving the segmentation accuracy of rare classes in the low-light domain. The experimental results on Cityscapes→Dark Zurich and Cityscapes→ACDC-night indicated that the proposed method in the paper achieved improvements of 8.2 percentage points and 7.2 percentage points, respectively, in mean intersection-over-union compared to the baseline DAFormer, the experimental results show the effectiveness of the method.

Semantic segmentation has been proposed as a tool to accelerate the processing of natural history collection images. However, developing a flexible and resilient segmentation network requires an approach for adaptation which allows processing different datasets with minimal training and validation. This paper presents a cross-validation approach designed to determine whether a semantic segmentation network possesses the flexibility required for application across different collections and institutions. Consequently, the specific objectives of cross-validating the semantic segmentation network are to (a) evaluate the effectiveness of the network for segmenting image sets derived from collections different from the one in which the network was initially trained on; and (b) test the adaptability of the segmentation network for use in other types of collections. The resilience to data variations from different institutions and the portability of the network across different types of collections are required to confirm its general applicability. The proposed validation method is tested on the Natural History Museum semantic segmentation network, designed to process entomological microscope slides. The proposed semantic segmentation network is evaluated through a series of cross-validation experiments designed to test using data from two types of collections: microscope slides (from three institutions) and herbarium sheets (from seven institutions). The main contribution of this work is the method, software and ground truth sets created for this cross-validation as they can be reused in testing similar segmentation proposals in the context of digitization of natural history collections. The cross-validation of segmentation methods should be a required step in the integration of such methods into image processing workflows for natural history collections.

Adverse road surface conditions (RSCs) are a significant concern for traffic safety and mobility during the winter season. Dealing with this issue requires the prompt execution of maintenance operations to restore safe driving conditions. The faster maintenance personnel are made aware of the dangerous conditions, the faster they can mobilize to mitigate adverse RSCs. Therefore, the speed at which RSC information is provided is vital for reducing potential road-related incidents and maintaining mobility. Intending to improve the delivery of RSC information, this study proposes developing a deep learning-based method through the usage of road weather information system (RWIS) images to automate the RSC classification process. While the RWIS collects images that give a direct view of the road, the traditional manual way of utilizing these images for RSC monitoring is laborious and time-consuming. To overcome this challenge as well as other limitations associated with previously developed RSC recognition methods, the application of deep neural networks using imagery data is investigated here. More specifically, the application of semantic segmentation (SS) and generative adversarial network (GAN) techniques in automating RSC recognition was adopted in an attempt to predict the locations of drivable areas and indicate the snow hazard level solely based on RWIS images. Case study results from Iowa, U.S.A., show that both SS and GAN techniques perform their tasks with a high degree of accuracy.

Application of computer images in virtual simulation technology-apparel as an example

Generative Adversarial Network has proven to produce state-of-the-art results by framing a generative modeling task into a supervised learning problem. In this paper, we propose Attentively Conditioned Generative Adversarial Network (ACGAN) for semantic segmentation by designing a segmentor model that generates probability maps from images and a discriminator model which discriminates the segmentor's output from the ground truth labels. Additionally, we conditioned the discriminator's dual inputs with extra information as a conditional adversarial model such that, an attention obtained probability distribution of the segmentor's feature maps is incorporated, and the ground truth is also accompanied by a vector of the class label. We demonstrate that our proposed model can provide better semantic segmentation results while stabilizing the discriminator to model long-range dependencies as a result of the supplementary inputs to the network. The attention network particularly provides more insights by extracting cues from the feature locations, and alongside the class label vector, gives the model an advantage to inform better spectral sensitivity. Experiments on the PASCAL VOC 2012 and the CamVid datasets show that our adversarial training technique yields improved accuracy.

CSF: Closed-mask-guided semantic fusion method for semantic perception of unknown scenes

Semantic segmentation is a fundamental task in image processing and computer vision domains. Therefore, it is a core component of most emerging industry applications such as medical imaging, autonomous driving, and agriculture. The last five years have witnessed a huge growth in the computer vision domain with the introduction of Generative Adversarial Networks (GANs) due to their ability to learn deep representations by a competitive process between two neural networks called the Generator the Discriminator. Neural Architecture Search (NAS) is undergoing a revolution aimed at automating the architectural engineering process. This paper discusses implementing a novel framework for semantic image segmentation called iSegmentor using NAS and GANs. The proposed approach is the first framework that utilizes NAS in GANs for semantic image segmentation to the best of our knowledge. The iSegmentor framework was evaluated using mean Intersection over Union (mIoU) and Mean Pixel Accuracy (MPA) metrics. The iSegmentor was mainly tested and evaluated for the autonomous driving domain. With the analysis of the past five-year literature, Cityscapes and CamVid datasets were chosen. The iSegmentor achieved 81.73 MPA and 71.90 mIoU for the Cityscapes dataset, whereas CamVid achieved 80.41 MPA and 70.63 mIoU. Further, we present future avenues that we identified with the conducted experiments to assist other researchers to extend the research area of NAS in GANs in semantic segmentation.

Abstract. Semantic instance segmentation from scenes, serving as a crucial role for 3D modelling and scene understanding. Conducting semantic segmentation before grouping instances is adopted by the existing state-of-the-art methods. However, without additional refinement, semantic errors will fully propagate into the grouping stage, resulting in low overlap with the ground truth instance. Furthermore, the proposed methods focused on indoor level scenes, which are limited when directly applied to large-scale outdoor Airborne Laser Scanning (ALS) point clouds. Numerous instances, significant object density and scale variations make ALS point clouds distinct from indoor data. In order to address the problems, we proposed a geometric characterization-aware semantic instance segmentation network, which utilized both semantic and objectness score to select potential points for grouping. And in point cloud feature learning stage, hand-craft geometry features are taken as input for geometric characterization awareness. Moreover, to address errors propagated from previous modules after grouping, we have additionally designed a per-instance refinement module. To assess semantic instance segmentation, we conducted experiments on an open-source dataset. Additionally, we performed semantic segmentation experiments to evaluate the performance of our proposed point cloud feature learning method.

Accurate organ-level segmentation is essential for achieving high-throughput, non-destructive, and automated plant phenotyping. To address the challenge of intelligent acquisition of phenotypic parameters in tomato plants, we propose TSINet, an end-to-end dual-task segmentation network designed for effective and precise semantic labeling and instance recognition of tomato point clouds, based on the Pheno4D dataset. TSINet adopts an encoder–decoder architecture, where a shared encoder incorporates four Geometry-Aware Adaptive Feature Extraction Blocks (GAFEBs) to effectively capture local structures and geometric relationships in raw point clouds. Two parallel decoder branches are employed to independently decode shared high-level features for the respective segmentation tasks. Additionally, a Dual Attention-Based Feature Enhancement Module (DAFEM) is introduced to further enrich feature representations. The experimental results demonstrate that TSINet achieves superior performance in both semantic and instance segmentation, particularly excelling in challenging categories such as stems and large-scale instances. Specifically, TSINet achieves 97.00% mean precision, 96.17% recall, 96.57% F1-score, and 93.43% IoU in semantic segmentation and 81.54% mPrec, 81.69% mRec, 81.60% mCov, and 86.40% mWCov in instance segmentation. Compared with state-of-the-art methods, TSINet achieves balanced improvements across all metrics, significantly reducing false positives and false negatives while enhancing spatial completeness and segmentation accuracy. Furthermore, we conducted ablation studies and generalization tests to systematically validate the effectiveness of each TSINet component and the overall robustness of the model. This study provides an effective technological approach for high-throughput automated phenotyping of tomato plants, contributing to the advancement of intelligent agricultural management.

Semantic segmentation is a fundamental task in remote sensing image processing. It provides pixel-level classification, which is important for many applications, such as building extraction and land use mapping. The development of convolutional neural network has considerably improved the performance of semantic segmentation. Most semantic segmentation networks are the encoder–decoder structure. Bilinear interpolation is an ordinary upsampling method in the decoder, but bilinear interpolation only considers its own features and inserts three times its own features. This over-simple and data-independent bilinear upsampling may lead to suboptimal results. In this work, we propose an upsampling method based on local relations to replace bilinear interpolation. Upsampling is performed by correlating the local relationship of feature maps of adjacent stages, which can better integrate local and global information. We also design a fusion module based on local similarity. Our proposed method with ResNet101 as the backbone of the segmentation network can improve the average <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$F_{1}$ </tex-math></inline-formula> score and overall accuracy of the Vaihingen data set by 2.69% and 1.31%, respectively. Our proposed method also has fewer parameters and less inference time.

As remote sensing images have complex backgrounds and varying object sizes, their semantic segmentation is challenging. This study proposes a multiscale cascaded network (MSCNet) for semantic segmentation. The resolutions employed with respect to the input remote sensing images are 1, 1/2, and 1/4, which represent high, medium, and low resolutions. First, 3 backbone networks extract features with different resolutions. Then, using a multiscale attention network, the fused features are input into the dense atrous spatial pyramid pooling network to obtain multiscale information. The proposed MSCNet introduces multiscale feature extraction and attention mechanism modules suitable for remote sensing land-cover classification. Experiments are performed using the Deepglobe, Vaihingen, and Potsdam datasets; the results are compared with those of the existing classical semantic segmentation networks. The findings indicate that the mean intersection over union (mIoU) of the MSCNet is 4.73% higher than that of DeepLabv3+ with the Deepglobe datasets. For the Vaihingen datasets, the mIoU of the MSCNet is 15.3%, and 6.4% higher than those of a segmented network (SegNet), and DeepLabv3+, respectively. For the Potsdam datasets, the mIoU of the MSCNet is higher than those of a fully convolutional network, Res-U-Net, SegNet, and DeepLabv3+ by 11.18%, 5.89%, 4.78%, and 3.03%, respectively.

Semantic segmentation is a crucial approach for remote sensing interpretation. High-precision semantic segmentation results are obtained at the cost of manually collecting massive pixelwise annotations. Remote sensing imagery contains complex and variable ground objects and obtaining abundant manual annotations is expensive and arduous. The semi-supervised learning (SSL) strategy can enhance the generalization capability of a model with a small number of labeled samples. In this study, a novel semi-supervised adversarial semantic segmentation network is developed for remote sensing information extraction. A multiscale input convolution module (MICM) is designed to extract sufficient local features, while a Transformer module (TM) is applied for long-range dependency modeling. These modules are integrated to construct a segmentation network with a double-branch encoder. Additionally, a double-branch discriminator network with different convolution kernel sizes is proposed. The segmentation network and discriminator network are jointly trained under the semi-supervised adversarial learning (SSAL) framework to improve its segmentation accuracy in cases with small amounts of labeled data. Taking building extraction as a case study, experiments on three datasets with different resolutions are conducted to validate the proposed network. Semi-supervised semantic segmentation models, in which DeepLabv2, the pyramid scene parsing network (PSPNet), UNet and TransUNet are taken as backbone networks, are utilized for performance comparisons. The results suggest that the approach effectively improves the accuracy of semantic segmentation. The F1 and mean intersection over union (mIoU) accuracy measures are improved by 0.82–11.83% and 0.74–7.5%, respectively, over those of other methods.

Automatic crack detection in construction facilities is a challenging yet crucial task. However, existing deep learning (DL)-based semantic segmentation methods for this field are based on fully supervised learning models and pixel-level manual annotation, which are time-consuming and labor-intensive. To solve this problem, this paper proposes a novel crack semantic segmentation network using weakly supervised approach and mixed-label training strategy. Firstly, an image patch-level classifier of crack is trained to generate a coarse localization map for automatic pseudo-labeling of cracks combined with a thresholding-based method. Then, we integrated the pseudo-annotated with manual-annotated samples with a ratio of 4:1 to train the crack segmentation network with a mixed-label training strategy, in which the manual labels were assigned with a higher weight value. The experimental data on two public datasets demonstrate that our proposed method achieves a comparable accuracy with the fully supervised methods, reducing over 65% of the manual annotation workload.