Multimodality semantic segmentation based on polarization and color images

Some exiting works have explored the promotion between depth estimation and semantic segmentation. These works are usually based on convolutional neural networks, which extract compact features and map the relationship between input and output according to specific tasks. In this paper, we introduce a novel adversarial training strategy, that is, generator produces the semantic segmentation map and depth map, and then the discriminator can judge the authenticity of the synthesized color image, thereby supervising the output of the network during back propagation. We use the adversarial loss combined with the reconstruction loss function to supervise the model, and find that the adversarial loss function which is seen as a global supervision can further optimize the output. We use 200 color images from Kitti dataset with semantic segmentation ground truth as the training set, and train the network in an end-to-end manner. The experimental results show that the adversarial training method is well applied to the multi-task training combining semantic segmentation and depth estimation, and can further improve the quantitative performance of depth estimation.

The polarization state can encode far more information compared to the intensity and spectrum of light signals. Polarization imaging and determination of polarization state has been a fascination of researchers in the field of optical engineering. Conventional polarization imaging involves complex methods and expensive manufacturing processes. Additionally, in recent times easily fabricate able semiconductor devices such as solution processed quantum dot (QD) LEDs 16 , quantum rod (QR) LEDs 13 ‐ 15 , QD photodetectors (PDs), etc. have gained tremendous interest and research efforts. The ability to fabricate a solution processable polarization detector can prove to be pivotal in the pursuit of facile polarization detection. Traditional polarization imaging systems make use of expensive wire‐grid polarizers and complicated patterning processes that make it challenging to realize such devices, the resulting devices are also quite expensive which limits their widespread application. In this work we present a novel system that makes use of a completely solution process fabricated QD PD using lead sulfide (PbS) QDs that can detect the whole visible spectrum and integrate a solution processable Azo‐dye polarizer (AD1) onto it to achieve patterned polarized detection. The PD achieved a response time of around 30 fis and the AD1 polarizer achieved a maximum extinction ratio of ~27:1. The fabricated AD1 polarizer also covers wavelengths in the range of 450 nm to 750 nm making it ideal for the visible spectrum The work represents a significant step forward in a facile and cheap polarized detector fabrication method that can enable a wider adoption of polarization imaging. Further optimization in pixelization and device architecture is expected to lead to a widespread implementation of such technology.

Designs of N optical filters for color and polarization imaging are found by minimizing detector noise, photon shot noise, and interpolation error for the image acquisition in a division of focal plane configuration. To minimize interpolation error, a general tiling procedure and an optimized tiling pattern for N filters are presented. For multispectral imaging, a general technique to find the transmission band is presented. For full Stokes polarization imaging, the general design with optimized retardances and fast angles of the polarizers is compared with the solution of the Thomson problem. These results are applied to the design of a three-color full Stokes imaging camera.

Road geometric information and a digital model based on light detection and ranging (LiDAR) can perform accurate geometric inventories and three-dimensional (3D) descriptions for as-built roads and infrastructures. However, unorganized point clouds and complex road scenarios would reduce the accuracy of geometric information extraction and digital modeling. There is a standardization need for information extraction and 3D model construction that integrates point cloud processing and digital modeling. This paper develops a framework from semantic segmentation to geometric information extraction and digital modeling based on LiDAR data. A semantic segmentation network is improved for the purpose of dividing the road surface and infrastructure. The road boundary and centerline are extracted by the alpha-shape and Voronoi diagram methods based on the semantic segmentation results. The road geometric information is obtained by a coordinate transformation matrix and the least square method. Subsequently, adaptive road components are constructed using Revit software. Thereafter, the road route, road entity model, and various infrastructure components are generated by the extracted geometric information through Dynamo and Revit software. Finally, a detailed digital model of the road scenario is developed. The Toronto-3D and Semantic3D datasets are utilized for analysis through training and testing. The overall accuracy (OA) of the proposed net for the two datasets is 95.3 and 95.0%, whereas the IoU of segmented road surfaces is 95.7 and 97.9%. This indicates that the proposed net could accomplish superior performance for semantic segmentation of point clouds. The mean absolute errors between the extracted and manually measured geometric information are marginal. This demonstrates the effectiveness and accuracy of the proposed extraction methods. Thus, the proposed framework could provide a reference for accurate extraction and modeling from LiDAR data.

This paper proposes a new method for simultaneous 3D reconstruction and semantic segmentation for indoor scenes. Unlike existing methods that require recording a video using a color camera and/or a depth camera, our method only needs a small number of (e.g., 3~5) color images from uncalibrated sparse views, which significantly simplifies data acquisition and broadens applicable scenarios. To achieve promising 3D reconstruction from sparse views with limited overlap, our method first recovers the depth map and semantic information for each view, and then fuses the depth maps into a 3D scene. To this end, we design an iterative deep architecture, named IterNet, to estimate the depth map and semantic segmentation alternately. To obtain accurate alignment between views with limited overlap, we further propose a joint global and local registration method to reconstruct a 3D scene with semantic information. We also make available a new indoor synthetic dataset, containing photorealistic high-resolution RGB images, accurate depth maps and pixel-level semantic labels for thousands of complex layouts. Experimental results on public datasets and our dataset demonstrate that our method achieves more accurate depth estimation, smaller semantic segmentation errors, and better 3D reconstruction results over state-of-the-art methods.

Semantic segmentation is of great importance to various vision applications. Depth information plays an important role in human visual system to help people obtain meaningful segmentation results, but it is not well considered by most existing segmentation methods. In this paper, we address the problem of semantic segmentation by incorporating depth information via deep neural Markov Random Field. In our method, the color image and its corresponding depth map are first fed to a convolutional neural network. Then, a deconvolution approach is performed on the network output to obtain the pixelwise prediction in terms of the probability of labels assigned to pixels. Finally, the dense prediction is used to design unary term and pairwise term, which are determined by pixels coordinate, color and depth. Experiments are conducted on several public datasets to illustrate the effectiveness of the proposed method. On the PASCAL VOC 2011 test dataset, experimental results show that our method can get accurate results when compared with the ground truth. On the PASCAL VOC 2012 dataset and NYUDv2 dataset, the proposed method can obtain competitive results.

Semantic segmentation is considered to be one of the basic steps in understanding image content. For semantic segmentation, if multi-spectral images are used together with color images, more successful results are obtained due to complementary information obtained from multi-spectral images. In this paper, a semantic segmentation method was developed in which the images obtained from CCD and thermal sensors were used together. In the proposed method, convolutional neural networks were used in encoder-decoder architecture. The experiments carried out show that the developed method produces better numerical and visual results than the works published in the literature.

Research efforts in linear polarization imaging have largely targeted the development of novel polarizing filters with improved performance and the monolithic integration of image sensors and polarization filter arrays. However, as pixel sizes in CMOS image sensors continue to decrease, the same limitations that have an impact on color and monochrome CMOS image sensors will undoubtedly affect polarization imagers. Issues of low signal capacity and dynamic range in small pixels will severely limit the useful polarization information that can be obtained. In this paper, we propose to leverage the benefits of the relatively new Quanta image sensor (QIS) concept to mitigate the anticipated limitations of linear polarization imaging as pixel sizes decrease. We address, by theoretical calculation and simulation, implementation issues such as alignment of polarization filters over extremely small pixels used in the QIS concept and polarization image formation from single-bit output of such pixels. We also present design innovations aimed at exploiting the benefits of this new imaging concept for simultaneous color and linear polarization imaging.

Uncrewed aerial vehicles (UAVs) aerial remote sensing (RS) images offer higher spatial resolution compared with satellite RS images, providing more detailed scene information. The flexibility of UAV-mounted sensors allows for the simultaneous acquisition of multimodal data, including RGB images and depth maps. Depth information, serving as a supplementary cue, has been extensively utilized in semantic segmentation research. Nevertheless, the current investigations on red, green, blue-depth (RGB-D) image semantic segmentation predominantly concentrate on indoor or road scenarios. In UAV RS scenes, there exist numerous objects featuring similar visual texture manifestations. The introduction of depth geometric information can efficaciously enhance the semantic segmentation accuracy of complex ground objects. Concurrently, high-resolution RS images also present challenges to the modeling of global information in large-scale space. To address these challenges, we propose the first framework for semantic segmentation of RGB-D images in UAV RS, namely, SkyMamba. This framework effectively integrates visual texture information and geometric depth information. Simultaneously, it leverages the long-sequence modeling capability of the state-space model to efficiently capture global context information in the RS space, achieving efficient semantic segmentation of RS scenes. Furthermore, we introduce a novel benchmark dataset for semantic segmentation of UAV RS RGB-D images, named comprehensive urban-scale depth, which is divided into two versions based on coarse-grained and fine-grained semantic annotation strategies to provide diverse challenges. Experimental results demonstrate that the proposed method exhibits state-of-the-art performance on both versions of the dataset, particularly in handling imbalanced scene semantic class distributions. The code for this article is available at the following URL: <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/CapyLin-G/CUS-Depth</uri>

With the continuous advancement of the construction of smart cities, the availability of large-scale and semantically enriched datasets is essential for enhancing the machine’s ability to understand urban scenes. Mesh data have a distinct advantage over point cloud data for large-scale scenes, as they can provide inherent geometric topology information and consume less memory space. However, existing publicly available large-scale scene mesh datasets are limited in scale and semantic richness and do not cover a wide range of urban semantic information. The development of 3D semantic segmentation algorithms depends on the availability of datasets. Moreover, existing large-scale 3D datasets lack various types of official annotation data, which hinders the widespread applicability of benchmark applications and may cause label errors during data conversion. To address these issues, we present a comprehensive urban-scale semantic segmentation benchmark dataset. It is suitable for various research pursuits on semantic segmentation methodologies. This dataset contains finely annotated point cloud and mesh data types for 3D, as well as high-resolution original 2D images with detailed 2D semantic annotations. It is constructed from a 3D reconstruction of 10,840 UVA aerial images and spans a vast area of approximately 2.85 square kilometers that covers both urban and rural scenes. The dataset is composed of 152,298,756 3D points and 289,404,088 triangles. Each 3D point, triangular mesh, and the original 2D image in the dataset are carefully labeled with one of the ten semantic categories. Six typical 3D semantic segmentation methods were compared on the CUS3D dataset, with KPConv demonstrating the highest overall performance. The mIoU is 59.72%, OA is 89.42%, and mAcc is 97.88%. Furthermore, the experimental results on the impact of color information on semantic segmentation suggest that incorporating both coordinate and color features can enhance the performance of semantic segmentation. The current limitations of the CUS3D dataset, particularly in class imbalance, will be the primary target for future dataset enhancements.

In order to enhance the image understanding of different regions for national costume grayscale image automatic colorization, let coloring tasks take advantage of semantic conditions, also let it can apply the human parsing semantic segmentation method to the national costume grayscale image for semantic segmentation task. This paper proposes a semantic segmentation model for context embedding based on edge perceiving. Aiming at the features of national costume grayscale image, more optimizing the model and loss function. The national costume grayscale image semantic segmentation is different from semantic segmentation of the color image, this task is more difficult for the grayscale image has no color feature. In this paper, edge information and edge consistency constraints are used to improve the national costume grayscale image coloring effect. The experimental results show that the model designed in this paper can obtain more accurate fine-grained semantic segmentation results for the national costume grayscale image.

Robust semantic scene segmentation for automotive applications is a challenging problem in two key aspects: (1) labelling every individual scene pixel and (2) performing this task under unstable weather and illumination changes (e.g., foggy weather), which results in poor outdoor scene visibility. Such visibility limitations lead to non-optimal performance of generalised deep convolutional neural network-based semantic scene segmentation. In this paper, we propose an efficient end-to-end automotive semantic scene understanding approach that is robust to foggy weather conditions. As an end-to-end pipeline, our proposed approach provides: (1) the transformation of imagery from foggy to clear weather conditions using a domain transfer approach (correcting for poor visibility) and (2) semantically segmenting the scene using a competitive encoder-decoder architecture with low computational complexity (en-abling real-time performance). Our approach incorporates RGB colour, depth and luminance images via distinct encoders with dense connectivity and features fusion to effectively exploit information from different inputs, which contributes to an optimal feature representation within the overall model. Using this architectural formulation with dense skip connections, our model achieves comparable performance to contemporary approaches at a fraction of the overall model complexity.

In this paper, we aim at the classification of the physical nature of local image structures in color images on the basis of geometrical and photometrical information. To this end, a framework is proposed to combine the local differential structure (i.e. geometrical information such as edges, corners, T-junctions etc) and color (i.e. photometrical information such as shadows, shading, illumination, highlights) in a multi-dimensional feature space. This framework is used to yield a proper classifier to classify salient image structures on the basis of their physical nature. The proposed framework is empirically verified on a set of images. From the theoretical and experimental results it is concluded that the proposed classification scheme successfully classifies local image structures robust to image translation and rotation, illumination intensity variations, and noise.

Image holistic scene understanding based on image intrinsic characteristics and conditional random fields is proposed. The model integrates image scene classification, image semantic segmentation and object detection. 1) For the scene classification, we use method of PHOW feature extraction plus KPCA dimensional reduction to obtain feature information for each image. 2) For object detection section, saliency detection and segmentation characteristics of the image object detection is useful. We propose the method by integrating image segmentation information got by the method proposed in literature [1]. 3) For the semantic segmentation: (1) For the unary potentials, we incorporating HOG, RGB color histogram and LBP features by the methods proposed in literature [2]; (2) The image manifold structural features can better reflect the importance between hyper-pixel regions and eventually boost accuracy. Therefore, we add the higher-order potential item to reflect inherent manifold image feature of each super pixel region. The experiments testify that model performance has raised on all three sub-tasks.

Semantic image segmentation is often served as an instrumental front-end preprocessing of many image processing and computer vision applications. Different from the existing semantic segmentation methods which mostly rely on database training, a simple and effective approach is proposed in this paper by employing the oriented pattern as a class-discriminative feature, besides the use of color and texture. The developed algorithm is to discriminate three specific semantic classes - sky, foliage, and building, which are commonly encountered in typical color images containing outdoor scene. In our approach, two maps will be generated from the input image: 1) the over-segmented map by using JSEG, and 2) the oriented pattern map by measuring the orientation coherence at each pixel position. Based on these maps, the region coherence of each over-segmented patch can be obtained by taking the average of all the orientation coherence values within the patch. The semantic class labeling for the patch can be simply conducted by checking the computed region coherence value against two empirically determined thresholds. Extensive simulation experiments have been conducted by testing various color images acquired from various sources, including the Microsoft Research Cambridge (MSRC) Object Recognition Image Database that contains manually-segmented regions as the ground truth. All experimental results clearly show that the proposed method is able to consistently deliver highly attractive performance.