Semantic Segmentation of Indoor Scenes Based on RGBD images Feature Fusion

Finding efficient and powerful techniques for representing 2D and 3D entities is central in many areas, such as computer vision, graphics, CAD or data visualization. The large amount of data often involved and the small processing time required place strong demands on modeling techniques. This chapter gives an overview of the possibilities wavelets offer for object modeling, specially from the point of view of object recognition in computer vision environments. For the 2D case, techniques based on segmented contours or raw grey level information are described. For 3D data, both the cases of dense, regularly sampled volumetric information and sparse, irregularly sampled geometric surface information are considered. Ideas on how Bayesian methods may enhance wavelet based techniques for object representation and recognition are outlined.

Augmenting RGB data with measured depth has been shown to improve the performance of a range of tasks in computer vision, including object detection and semantic segmentation. Although depth sensors such as the Microsoft Kinect have facilitated easy acquisition of such depth information, the vast majority of images used in vision tasks do not contain depth information. In this paper, we show that augmenting RGB images with estimated depth can also improve the accuracy of both object detection and semantic segmentation. Specifically, we first exploit the recent success of depth estimation from monocular images and learn a deep depth estimation model. Then, we learn deep depth features from the estimated depth and combine with RGB features for object detection and semantic segmentation. In addition, we propose an RGB-D semantic segmentation method, which applies a multi-task training scheme: semantic label prediction and depth value regression. We test our methods on several data sets and demonstrate that incorporating information from estimated depth improves the performance of object detection and semantic segmentation remarkably.

We address the problem of object recognition in computer vision. We represent each model and the scene in the form of Attributed Relational Graph. A multiple region representation is provided at each node of the scene ARG to increase the representation reliability. The process of matching the scene ARG against the stored models is facilitated by a novel method for identifying the most probable representation from among the multiple candidates. The scene and model graph matching is accomplished using probabilistic relaxation which has been modified to minimise the label clutter. The experimental results obtained on real data demonstrate promising performance of the proposed recognition system.

Deep neural networks inspired by the connections among biological neurons have been highly effective in the advancement of computer vision. Recent research into recurrent neural networks has taken account of forward as well as recurrent connections in a neural network architecture. In this study, we proposed a model that mirrors the architecture of the human ventral pathway with separate layers representing brain regions connected using long-range recurrent links. CIFAR-10/100 datasets were used to assess the performance of object recognition using the proposed model and the results were compared with those obtained using state-of-the-art methods. We demonstrated that the classification accuracy increased as the number of recurrences increased. Our results suggest that the proposed neuroscience-inspired model can facilitate object recognition in computer vision and may help to elucidate neurological mechanisms in the human brain.

The paper descries a novel method to extract line segments and circular arcs from the image taken by CCD camera for object recognition. Model based matching is one of the widely used techniques of the object recognition in computer vision, and the position and orientation of the objects are decided with referring the extracted line segments and circular arcs. For the precise and stable decisions of the position and orientation, the accurate extractions of characters are required. These characters are. fast and accurately extracted in the proposed method, while its validity is confirmed in some experiments.

This study examines various algorithms for computing the convex hull of a set of n points in a d-dimensional space. Convex hulls are fundamental in computational geometry and are applied in computer graphics, pattern recognition, and computational biology. Such convex hulls can also be useful in symmetry problems. For instance, when points are arranged symmetrically, the convex hull is also likely to be symmetrically shaped, which can be useful for object recognition in computer vision or pattern recognition. The focus is primarily on two-dimensional algorithms, including well-known methods like Gift Wrapping, Graham Scan, Divide and Conquer, QuickHull, TORCH, Kirkpatrick–Sediel, and Chan’s algorithms. These algorithms vary in terms of time complexity and scalability to higher dimensions. This study is extended to three-dimensional convex hull algorithms, such as NAW, randomized insertion, and parallelized versions, such as CudaHull and CudaChain. This study aimed to elucidate the operational principles, step-by-step procedures, and comparative time complexities of each algorithm. The implementation in Python facilitates a detailed comparison of the algorithmic performance through stepwise analysis and graphical outputs. The ultimate goal is to provide insights into the strengths and weaknesses of each algorithm under various scenarios, thereby offering a comprehensive guide for practical implementation.

Hessian matrix is the square matrix of second partial derivatives of a scalar-valued function and is well known for object recognition in computer vision and medical shape analysis. Previous curvature based polyp detection algorithms generate myriad of false positives. Hessian-matrix based method, however, is more sensitive to local shape features, so easily reduce false positives. Calculation of Hessian matrix on 3D CT data and Eigen decomposition of the matrix gives three Eigen values and vectors at each voxel. Using these Eigen values, we can figure out which type of intensity structures (blob, line, and sheet-like) is on the given voxel. We focus on detecting blob-like object automatically. In the inner colonic wall structures, blob-like, line-like, and sheet-like objects represent polyps, folds and wall, respectively. In addition, to improve the performance of the algorithm, Gaussian blurring factor and shape threshold parameters are optimized. Before Hessian matrix calculation, smoothing the given region using Gaussian kernel with small deviation is necessary to enhance local intensity structures. To optimize the parameters and validate this method, we have produced anthropomorphic pig phantoms. Fourteen phantoms with 103 polyps (16 polyps <6mm, 87 >= 6mm) were used. CT scan was performed with 1mm slice thickness. Our detection algorithm found 84 polyps (81.6%) correctly. Average number of false positives is 7.9 at each CT scan. This results show that our algorithm is clinically applicable for polyp detection, because of high sensitivity and relatively low false positive detections.

With the emergence of deep learning, computer vision has witnessed extensive advancement and has seen immense applications in multiple domains. Specifically, image captioning has become an attractive focal direction for most machine learning experts, which includes the prerequisite of object identification, location, and semantic understanding. In this paper, semantic segmentation and image captioning are comprehensively investigated based on traditional and state‐of‐the‐art methodologies. In this survey, we deliberate on the use of deep learning techniques on the segmentation analysis of both 2D and 3D images using a fully convolutional network and other high‐level hierarchical feature extraction methods. First, each domain’s preliminaries and concept are described, and then semantic segmentation is discussed alongside its relevant features, available datasets, and evaluation criteria. Also, the semantic information capturing of objects and their attributes is presented in relation to their annotation generation. Finally, analysis of the existing methods, their contributions, and relevance are highlighted, informing the importance of these methods and illuminating a possible research continuation for the application of semantic image segmentation and image captioning approaches.

Semantic segmentation is a fundamental task in image processing and computer vision domains. Therefore, it is a core component of most emerging industry applications such as medical imaging, autonomous driving, and agriculture. The last five years have witnessed a huge growth in the computer vision domain with the introduction of Generative Adversarial Networks (GANs) due to their ability to learn deep representations by a competitive process between two neural networks called the Generator the Discriminator. Neural Architecture Search (NAS) is undergoing a revolution aimed at automating the architectural engineering process. This paper discusses implementing a novel framework for semantic image segmentation called iSegmentor using NAS and GANs. The proposed approach is the first framework that utilizes NAS in GANs for semantic image segmentation to the best of our knowledge. The iSegmentor framework was evaluated using mean Intersection over Union (mIoU) and Mean Pixel Accuracy (MPA) metrics. The iSegmentor was mainly tested and evaluated for the autonomous driving domain. With the analysis of the past five-year literature, Cityscapes and CamVid datasets were chosen. The iSegmentor achieved 81.73 MPA and 71.90 mIoU for the Cityscapes dataset, whereas CamVid achieved 80.41 MPA and 70.63 mIoU. Further, we present future avenues that we identified with the conducted experiments to assist other researchers to extend the research area of NAS in GANs in semantic segmentation.

The field of computer vision has undergone rapid development. Starting from recognition tasks such as classification, detection, and segmentation, the focus of visual analysis has gradually shifted towards learning spatiotemporal information. This thesis presents research on spatiotemporal learning, with a particular emphasis on motion estimation and visual recognition. First, we address the problem of video object tracking. Previous methods have primarily re-lied on learning improved appearance representations, while the spatiotemporal relationships of individual objects have been underexplored. We propose leveraging optical flow features to achieve higher generalization in semi-supervised video object segmentation, directly incorporating these features into both the target representation and the decoder network. Our experiments and analysis show that enriching feature representations with spatiotemporal information improves segmentation quality and generalization capability. Next, we investigate spatiotemporal learning in 3D for motion estimation, specifically scene flow estimation. Scene flow estimation as an important research topic in 3D computer vision is crucial for applications such as robotics, autonomous driving, embodied navigation, and tracking. We investigate the problem in different perspectives: 1. What is the best formulation for solving the problem and how to learn a better spatiotemporal feature representation? 2. Can we introduce uncertainty estimation to the task, which is of crucial importance for safety-critical downstream tasks? 3. How to scale the estimation to large-scale data, e.g., autonomous scenes, and leverage the temporal information without introducing much computation overheads? To answer these questions, we explore the use of transformers for improved feature representation, diffusion models for uncertainty estimation, and efficient feature learning methods for multi-frame, large-scale autonomous driving scenarios. Finally, we extend our research to joint visual segmentation, tracking, and open-vocabulary recognition in LiDAR sequences, particularly for autonomous scenes. In such environments, precise segmentation, tracking, and recognition of objects are essential for downstream analysis and control. Current human-annotated open-source datasets allow for reasonable tracking of traffic participants such as cars and pedestrians. However, we aim to advance beyond this towards segmenting and tracking any object in LiDAR data. To this end, we propose a pseudo-labeling engine that leverages the 2D visual foundation model SAM v2 and the vision-language model CLIP to automatically label LiDAR streams. We further introduce the SAL-4D model, capable of segmenting, tracking, and recognizing any object in a zero-shot manner. In summary, we explore the learning of spatiotemporal information in both 2D image and 3D point cloud domains. In the image domain, we demonstrate that spatiotemporal information improves video object segmentation quality and generalization. In the 3D point cloud domain, we show that spatiotemporal learning enables more accurate motion estimation and facilitates the first method for zero-shot segmentation, tracking, and open-vocabulary recognition of arbitrary objects.

With the continuous development of intelligent unmanned aerial vehicles reconnaissance, single-object detection or semantic segmentation can no longer meet the diversified requirements, while simple model stacking will cause the model to be too complex, which will seriously affect the real-time running effect. We propose a visual scene understanding algorithm based on a multitask learning network with an encoder–decoder structure. First, the efficient classification network VoVNet is selected as the feature-sharing network to obtain multiscale coding features. Second, based on the one-stage anchor-free object detector, the feature screening supplementary module implements feature interaction with segmented characters to enhance the detection capability of potential targets. Then for semantic segmentation and depth estimation pixel-level classification tasks, a cascaded chained residual pooling module is used as a parameter-sharing decoder, through the parameter-sharing mechanism to reduce the repeated decoding process to ensure the running speed. Finally, to improve the generalization ability of the model, a general-purpose dataset was constructed and network training was carried out based on the idea of knowledge distillation. Experiments on the dataset show that the performance of the multitask learning network can reach the mainstream algorithm level in semantic segmentation and depth estimation branches, whereas the object detection branch is better in multitarget recall rate, and the average time of each subtask is 39 ms, which meets the requirements of real-time performance and proves the effectiveness of the multitask network.

Semantic segmentation is a fundamental task in remote sensing image processing. It provides pixel-level classification, which is important for many applications, such as building extraction and land use mapping. The development of convolutional neural network has considerably improved the performance of semantic segmentation. Most semantic segmentation networks are the encoder–decoder structure. Bilinear interpolation is an ordinary upsampling method in the decoder, but bilinear interpolation only considers its own features and inserts three times its own features. This over-simple and data-independent bilinear upsampling may lead to suboptimal results. In this work, we propose an upsampling method based on local relations to replace bilinear interpolation. Upsampling is performed by correlating the local relationship of feature maps of adjacent stages, which can better integrate local and global information. We also design a fusion module based on local similarity. Our proposed method with ResNet101 as the backbone of the segmentation network can improve the average <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$F_{1}$ </tex-math></inline-formula> score and overall accuracy of the Vaihingen data set by 2.69% and 1.31%, respectively. Our proposed method also has fewer parameters and less inference time.

TAG-fusion: Two-stage attention guided multi-modal fusion network for semantic segmentation

An ever increasing number of computer vision and image/video processing challenges are being approached using deep convolutional neural networks, obtaining state-of-the-art results in object recognition and detection, semantic segmentation, action recognition, optical flow and superresolution. Hardware acceleration of these algorithms is essential to adopt these improvements in embedded and mobile computer vision systems. We present a new architecture, design and implementation as well as the first reported silicon measurements of such an accelerator, outperforming previous work in terms of power-, area- and I/O-efficiency. The manufactured device provides up to 196 GOp/s on 3.09 mm^2 of silicon in UMC 65nm technology and can achieve a power efficiency of 803 GOp/s/W. The massively reduced bandwidth requirements make it the first architecture scalable to TOp/s performance.

We present the DeepScores dataset with the goal of advancing the state-of-the-art in small object recognition by placing the question of object recognition in the context of scene understanding. DeepScores contains high quality images of musical scores, partitioned into 300, 000 sheets of written music that contain symbols of different shapes and sizes. With close to a hundred million small objects, this makes our dataset not only unique, but also the largest public dataset. DeepScores comes with ground truth for object classification, detection and semantic segmentation. DeepScores thus poses a relevant challenge for computer vision in general, and optical music recognition (OMR) research in particular. We present a detailed statistical analysis of the dataset, comparing it with other computer vision datasets like PASCAL VOC, SUN, SVHN, ImageNet, MS-COCO, as well as with other OMR datasets. Finally, we provide baseline performances for object classification, intuition for the inherent difficulty that DeepScores poses to state-of-the-art object detectors like YOLO or R-CNN, and give pointers to future research based on this dataset.