Input Optimisation Network for Semantic Segmentation of Underexposed Images

Semantic segmentation is a fundamental task in remote sensing image understanding. Recently, Deep Convolutional Neural Networks (DCNNs) have considerably improved the performance of the semantic segmentation of natural scenes. However, it is still challenging for very high-resolution remote sensing images. Due to the large and complex scenes as well as the influence of illumination and imaging angle, it is particularly difficult for the existing methods to accurately obtain the category of pixels at object boundaries—the so-called boundary blur. We propose a framework called Boundary-Aware Semi-Supervised Semantic Segmentation Network (BAS <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">4</sup> Net), which obtains more accurate segmentation results without additional annotation workload, especially at the object boundaries. The Channel-weighted Multi-scale Feature (CMF) module balances semantic and spatial information, and the Boundary Attention Module (BAM) weights the features with rich semantic boundary information to alleviate the boundary blur. Additionally, to decrease the amount of difficult and tedious manual labeling of remote sensing images, a discriminator network infers pseudolabels from unlabeled images to assist semisupervised learning, and further improves the performance of the segmentation network. To validate the effectiveness of the proposed framework, extensive experiments have been performed on both the ISPRS Vaihingen dataset, and the novel remote sensing dataset AIR-SEG with more categories, and complex boundaries. The results demonstrate a significant improvement of accuracy especially on boundaries and for small objects.

A Unified Architecture of Semantic Segmentation and Hierarchical Generative Adversarial Networks for Expression Manipulation

Abstract. Semantic instance segmentation from scenes, serving as a crucial role for 3D modelling and scene understanding. Conducting semantic segmentation before grouping instances is adopted by the existing state-of-the-art methods. However, without additional refinement, semantic errors will fully propagate into the grouping stage, resulting in low overlap with the ground truth instance. Furthermore, the proposed methods focused on indoor level scenes, which are limited when directly applied to large-scale outdoor Airborne Laser Scanning (ALS) point clouds. Numerous instances, significant object density and scale variations make ALS point clouds distinct from indoor data. In order to address the problems, we proposed a geometric characterization-aware semantic instance segmentation network, which utilized both semantic and objectness score to select potential points for grouping. And in point cloud feature learning stage, hand-craft geometry features are taken as input for geometric characterization awareness. Moreover, to address errors propagated from previous modules after grouping, we have additionally designed a per-instance refinement module. To assess semantic instance segmentation, we conducted experiments on an open-source dataset. Additionally, we performed semantic segmentation experiments to evaluate the performance of our proposed point cloud feature learning method.

Semantic segmentation plays an increasingly important role in the field of computer vision. The current semantic segmentation algorithms are mainly based on full convolutional network downsampling for feature extraction, however, the reduction of image resolution due to downsampling will inevitably lead to the loss of image information and the process is irreversible. Although features can be reused to compensate for the loss, the problem of lost pixel correlation and inaccurate segmentation results still occur. In order to solve these problems, the attention mechanism and the generative adversarial network are fused into the semantic segmentation network, and a semantic segmentation network is proposed. The attention mechanism is added to the generator and discriminator modules and trained successively, then the generator module with rich information and global information is extracted, and its output is fused with the feature graph sampled above, in order to reduce the loss of local information and complete the task of image segmentation, an interpolation method based on image edge is introduced to sample the image, which improves the bilinear interpolation edge blur. Experiments on Cityscapes dataset show that the proposed network model is effective and reliable.

As one of the most important components of urban space, an outdated inventory of road-side trees may misguide managers in the assessment and upgrade of urban environments, potentially affecting urban road quality. Therefore, automatic and accurate instance segmentation of road-side trees from urban point clouds is an important task in urban ecology research. However, previous works show under- or over-segmentation effects for road-side trees due to overlapping, irregular shapes and incompleteness. In this paper, a deep learning framework that combines semantic and instance segmentation is proposed to extract single road-side trees from vehicle-mounted mobile laser scanning (MLS) point clouds. In the semantic segmentation stage, the ground points are filtered to reduce the processing time. Subsequently, a graph-based semantic segmentation network is developed to segment road-side tree points from the raw MLS point clouds. For the individual tree segmentation stage, a novel joint instance and semantic segmentation network is adopted to detect instance-level roadside trees. Two complex Chinese urban point cloud scenes are used to evaluate the individual urban tree segmentation performance of the proposed method. The proposed method accurately extract approximately 90% of the road-side trees and achieve better segmentation results than existing published methods in both two urban MLS point clouds. Living Vegetation Volume (LVV) calculation can benefit from individual tree segmentation. The proposed method provides a promising solution for ecological construction based on the LVV calculation of urban roads.

Semantic segmentation is a fundamental capability for autonomous vehicles. With the advancements of deep learning technologies, many effective semantic segmentation networks have been proposed in recent years. However, most of them are designed using RGB images from visible cameras. The quality of RGB images is prone to be degraded under unsatisfied lighting conditions, such as darkness and glares of oncoming headlights, which imposes critical challenges for the networks that use only RGB images. Different from visible cameras, thermal imaging cameras generate images using thermal radiations. They are able to see under various lighting conditions. In order to enable robust and accurate semantic segmentation for autonomous vehicles, we take the advantage of thermal images and fuse both the RGB and thermal information in a novel deep neural network. The main innovation of this letter is the architecture of the proposed network. We adopt the encoder–decoder design concept. ResNet is employed for feature extraction and a new decoder is developed to restore the feature map resolution. The experimental results prove that our network outperforms the state of the arts.

Accurate organ-level segmentation is essential for achieving high-throughput, non-destructive, and automated plant phenotyping. To address the challenge of intelligent acquisition of phenotypic parameters in tomato plants, we propose TSINet, an end-to-end dual-task segmentation network designed for effective and precise semantic labeling and instance recognition of tomato point clouds, based on the Pheno4D dataset. TSINet adopts an encoder–decoder architecture, where a shared encoder incorporates four Geometry-Aware Adaptive Feature Extraction Blocks (GAFEBs) to effectively capture local structures and geometric relationships in raw point clouds. Two parallel decoder branches are employed to independently decode shared high-level features for the respective segmentation tasks. Additionally, a Dual Attention-Based Feature Enhancement Module (DAFEM) is introduced to further enrich feature representations. The experimental results demonstrate that TSINet achieves superior performance in both semantic and instance segmentation, particularly excelling in challenging categories such as stems and large-scale instances. Specifically, TSINet achieves 97.00% mean precision, 96.17% recall, 96.57% F1-score, and 93.43% IoU in semantic segmentation and 81.54% mPrec, 81.69% mRec, 81.60% mCov, and 86.40% mWCov in instance segmentation. Compared with state-of-the-art methods, TSINet achieves balanced improvements across all metrics, significantly reducing false positives and false negatives while enhancing spatial completeness and segmentation accuracy. Furthermore, we conducted ablation studies and generalization tests to systematically validate the effectiveness of each TSINet component and the overall robustness of the model. This study provides an effective technological approach for high-throughput automated phenotyping of tomato plants, contributing to the advancement of intelligent agricultural management.

Semantic segmentation has been proposed as a tool to accelerate the processing of natural history collection images. However, developing a flexible and resilient segmentation network requires an approach for adaptation which allows processing different datasets with minimal training and validation. This paper presents a cross-validation approach designed to determine whether a semantic segmentation network possesses the flexibility required for application across different collections and institutions. Consequently, the specific objectives of cross-validating the semantic segmentation network are to (a) evaluate the effectiveness of the network for segmenting image sets derived from collections different from the one in which the network was initially trained on; and (b) test the adaptability of the segmentation network for use in other types of collections. The resilience to data variations from different institutions and the portability of the network across different types of collections are required to confirm its general applicability. The proposed validation method is tested on the Natural History Museum semantic segmentation network, designed to process entomological microscope slides. The proposed semantic segmentation network is evaluated through a series of cross-validation experiments designed to test using data from two types of collections: microscope slides (from three institutions) and herbarium sheets (from seven institutions). The main contribution of this work is the method, software and ground truth sets created for this cross-validation as they can be reused in testing similar segmentation proposals in the context of digitization of natural history collections. The cross-validation of segmentation methods should be a required step in the integration of such methods into image processing workflows for natural history collections.

Semantic segmentation of high-resolution optical remote sensing images is an important but challenging task. To solve the problem that many semantic segmentation networks fail to efficiently utilize global and local context information to improve the segmentation performance, this paper proposes a semantic segmentation network based on sparse self-attention (SDANet) to model the global context dependencies. Specifically, the feature maps are first divided into four regions in spatial and channel dimensions, respectively, and the divided feature maps are rearranged to form new regions. Second, the position and channel self-attention operations are performed on the rearranged regions. Third, the feature maps are restored to the original combination and the position together with channel self-attention operations are performed again to obtain the output feature maps. Finally, semantic segmentation is completed based on the output feature maps. Extensive experiments conducted on the ISPRS Vaihingen dataset demonstrate that the proposed method is superior to self-attention-based DANet, CCNet, and other general semantic segmentation networks, such as FCN, Deeplabv3+, HRNet, etc.

Semantic segmentation is a fundamental task in remote sensing image processing. It provides pixel-level classification, which is important for many applications, such as building extraction and land use mapping. The development of convolutional neural network has considerably improved the performance of semantic segmentation. Most semantic segmentation networks are the encoder–decoder structure. Bilinear interpolation is an ordinary upsampling method in the decoder, but bilinear interpolation only considers its own features and inserts three times its own features. This over-simple and data-independent bilinear upsampling may lead to suboptimal results. In this work, we propose an upsampling method based on local relations to replace bilinear interpolation. Upsampling is performed by correlating the local relationship of feature maps of adjacent stages, which can better integrate local and global information. We also design a fusion module based on local similarity. Our proposed method with ResNet101 as the backbone of the segmentation network can improve the average <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$F_{1}$ </tex-math></inline-formula> score and overall accuracy of the Vaihingen data set by 2.69% and 1.31%, respectively. Our proposed method also has fewer parameters and less inference time.

Road geometric information and a digital model based on light detection and ranging (LiDAR) can perform accurate geometric inventories and three-dimensional (3D) descriptions for as-built roads and infrastructures. However, unorganized point clouds and complex road scenarios would reduce the accuracy of geometric information extraction and digital modeling. There is a standardization need for information extraction and 3D model construction that integrates point cloud processing and digital modeling. This paper develops a framework from semantic segmentation to geometric information extraction and digital modeling based on LiDAR data. A semantic segmentation network is improved for the purpose of dividing the road surface and infrastructure. The road boundary and centerline are extracted by the alpha-shape and Voronoi diagram methods based on the semantic segmentation results. The road geometric information is obtained by a coordinate transformation matrix and the least square method. Subsequently, adaptive road components are constructed using Revit software. Thereafter, the road route, road entity model, and various infrastructure components are generated by the extracted geometric information through Dynamo and Revit software. Finally, a detailed digital model of the road scenario is developed. The Toronto-3D and Semantic3D datasets are utilized for analysis through training and testing. The overall accuracy (OA) of the proposed net for the two datasets is 95.3 and 95.0%, whereas the IoU of segmented road surfaces is 95.7 and 97.9%. This indicates that the proposed net could accomplish superior performance for semantic segmentation of point clouds. The mean absolute errors between the extracted and manually measured geometric information are marginal. This demonstrates the effectiveness and accuracy of the proposed extraction methods. Thus, the proposed framework could provide a reference for accurate extraction and modeling from LiDAR data.

The semantic segmentation of remote sensing images faces two major challenges: high inter-class similarity and interference from ubiquitous shadows. In order to address these issues, we develop a novel edge loss reinforced semantic segmentation network (ERN) that leverages the spatial boundary context to reduce the semantic ambiguity. The main contributions of this paper are as follows: (1) we propose a novel end-to-end semantic segmentation network for remote sensing, which involves multiple weighted edge supervisions to retain spatial boundary information; (2) the main representations of the network are shared between the edge loss reinforced structures and semantic segmentation, which means that the ERN simultaneously achieves semantic segmentation and edge detection without significantly increasing the model complexity; and (3) we explore and discuss different ERN schemes to guide the design of future networks. Extensive experimental results on two remote sensing datasets demonstrate the effectiveness of our approach both in quantitative and qualitative evaluation. Specifically, the semantic segmentation performance in shadow-affected regions is significantly improved.

In remote-sensing image (RSI) semantic segmentation, the dependence on large-scale and pixel-level annotated data has been a critical factor restricting its development. In this letter, we propose an unsupervised semantic segmentation network embedded with geometry consistency (UGCNet) for RSIs, which imports the adversarial-generative learning strategy into a semantic segmentation network. The proposed UGCNet can be trained on a source-domain dataset and achieve accurate segmentation results on a different target-domain dataset. Furthermore, for refining the remote-sensing target geometric representation such as densely distributed buildings, we propose a geometry-consistency (GC) constraint that can be embedded in both image-domain adaptation process and semantic segmentation network. Therefore, our model could achieve cross-domain semantic segmentation with target geometric property preservation. The experimental results on Massachusetts and Inria buildings datasets prove that the proposed unsupervised UGCNet could achieve a very comparable segmentation accuracy with the fully supervised model, which validates the effectiveness of the proposed method.

Vision-based perception is an import link in driverless technology, and semantic segmentation is one of the mostly uses atrous convolution with a large amount of computation and high memory consumption to extract high-resolution feature maps. As a result, the current mainstream semantic segmentation network lacks the segmentation speed and cannot be effectively applied in driverless technology. To solve this problem, a semantic segmentation network with better real-time performance is proposed. Firstly, a lightweight convolutional neural network is used as the encoder, and stride convolution and regular convolution are used to replace the time-consuming and memory-consuming atrous convolution. Secondly, in order to obtain the feature map similar to Deeplab v3+, a new joint upsampling module, Multi-scale Feature Map Joint Pyramid Upsamping(MJPU), is proposed. By fusing multiple feature maps in the encoder, a high resolution feature map with richer semantic information is generated. Experments on the Cityscapes dataset show that, compared with the popular semantic segmentation network Deeplab V3+, the proposed network can improve the segmentation speed to 31.4 FPS/s without losing a lot of performance(mIoU of 44.03%). Therefore, the network proposed in this paper has better real-time performance and is more suitable for driverless scenes.

In view of the insufficient illumination of low-light images in semantic segmentation, there are phenomena such as low image brightness, contrast, and signal-to-noise ratio, which makes semantic segmentation of low-light images more difficult. A low-light image semantic segmentation method based on vector symbolic architectures domain adaptation network is proposed. The method first integrates the image translation network based on vector symbolic architectures and the domain adaptive semantic segmentation network. The image translation network translates normal illumination images into low-light images, reducing style differences between domains and improving the segmentation accuracy of low-light images. Then, a layered feature mapping (LFM) module is proposed in the image translation network. The LFM module can better map the multi-layer feature vectors of the semantic segmentation network encoder to the VSA space, making the translated image more similar to the low-light image, thereby improving the segmentation accuracy of the segmentation network. Finally, a cross domain rare class mixing (CDRCM) method is proposed in the domain adaptive semantic segmentation network. CDRCM obtains rare classes in the low-light domain based on the pseudo- label distribution of low-light images during domain adaptation, and biases towards these rare classes during cross-domain mixing, thereby improving the segmentation accuracy of rare classes in the low-light domain. The experimental results on Cityscapes→Dark Zurich and Cityscapes→ACDC-night indicated that the proposed method in the paper achieved improvements of 8.2 percentage points and 7.2 percentage points, respectively, in mean intersection-over-union compared to the baseline DAFormer, the experimental results show the effectiveness of the method.