Point Cloud Instance Segmentation of Indoor Scenes Using Learned Pairwise Patch Relations

We present two robust and novel algorithms to model a 3D environment using both intensity and range data provided by an off-the-shelf stereo camera. The main issue we need to address is that the output of the stereo system is both sparse and noisy. To overcome this limitation,, we detect planar patches in the environment by region segmentation in 2D and plane extraction in 3D. The extracted planar patches are used not only to represent the workspace, but also to fill holes in range data. We also suggest a new planar patch based scan matching algorithm to register multiple views, and to incrementally augment the description of the 3D workspace in a sequence of scenes. Experimental results on real data show that planar patch segmentation and 3D scene registration for environment modeling can be robustly achieved by the proposed approaches.

With the leaps and bounds of computer performance and the advent of the era of big data, deep learning has drawn more and more attention from all walks of life. It can combine low-level features to form more abstract high-level features and describe the data more essentially. Therefore, it is widely used in various fields such as computer vision. Image segmentation is one of the most basic research topics in the field of computer vision. The main purpose is to extract regions of interest from images for later image processing research. In 3D reconstruction based on sequence images, the segmentation accuracy and speed of sequence images determine the quality and efficiency of target reconstruction. Therefore, when facing large-scale sequence images, the biggest problem is how to improve the segmentation speed while ensuring accuracy. Based on the above background, the research content of this article is an indoor scene segmentation algorithm based on full convolutional neural network. According to the characteristics of indoor application scenes, this paper proposes a fast convolutional neural network image segmentation method to segment the indoor scene image and construct the fast fully convolutional networks (FFCN) for indoor scene image segmentation uses inter-layer fusion to reduce the amount of network calculation parameters and avoid the loss of picture feature information by continuous convolution. In order to verify the effectiveness of the network, in this paper, a basic living object data set (XAUT data set) in an indoor environment is created. The XAUT data set is used to train the FFCN network under the Caffe framework to obtain an indoor scene segmentation model. In order to compare the effectiveness of the model, the structure of the worn FCN8s, FCN16s, and FCN32s models was fine-tuned, and the corresponding algorithm model for indoor scene segmentation was obtained by training with the XAUT data set. The experimental results show that the pixel recognition accuracy of all types of networks has reached 86%, and the mean IU ratio has reached more than 63%. The mean IU of the FCN8s network is the highest at 70.38%, but its segmentation speed is only 1/5 of FFCN. On the premise that other types of indicators are not much different, the average segmentation speed on FFCN fast segmentation convolutional neural network reaches 40 fps. It can be seen that the scale fusion technology can well avoid the loss of image feature information in the network convolution and reddening process. Compared with other FCN networks, it has a faster speed and is conducive to real-time image preprocessing.

This paper presents an algorithm for extracting planar patches by integrating both intensity and range data provided by a stereo system. For dealing with noisy and sparse range data, the initial segmentation is based on intensity information, and then the resulted regions are thresholded using depth data. This new algorithm, different from the existing ones that use only range data in the segmentation process, produces accurate planar patches that are then used for building a panoramic image-based model for mobile robot navigation.

Abstract. For obtaining a full coverage of 3D scans in a large-scale urban area, the registration between point clouds acquired via terrestrial laser scanning (TLS) is normally mandatory. However, due to the complex urban environment, the automatic registration of different scans is still a challenging problem. In this work, we propose an automatic marker free method for fast and coarse registration between point clouds using the geometric constrains of planar patches under a voxel structure. Our proposed method consists of four major steps: the voxelization of the point cloud, the approximation of planar patches, the matching of corresponding patches, and the estimation of transformation parameters. In the voxelization step, the point cloud of each scan is organized with a 3D voxel structure, by which the entire point cloud is partitioned into small individual patches. In the following step, we represent points of each voxel with the approximated plane function, and select those patches resembling planar surfaces. Afterwards, for matching the corresponding patches, a RANSAC-based strategy is applied. Among all the planar patches of a scan, we randomly select a planar patches set of three planar surfaces, in order to build a coordinate frame via their normal vectors and their intersection points. The transformation parameters between scans are calculated from these two coordinate frames. The planar patches set with its transformation parameters owning the largest number of coplanar patches are identified as the optimal candidate set for estimating the correct transformation parameters. The experimental results using TLS datasets of different scenes reveal that our proposed method can be both effective and efficient for the coarse registration task. Especially, for the fast orientation between scans, our proposed method can achieve a registration error of less than around 2 degrees using the testing datasets, and much more efficient than the classical baseline methods.

HighlightsWhat are the main findings?Proposed an innovative two-channel hybrid model by integrating convolutional neural networks (CNNs) with a text-based classifier.Leveraged an extended dataset derived from multiple object recognition models, increasing input data diversity and achieving a text-based classifier accuracy of 73.30%. Achieved a significant improvement of 8.33% in accuracy compared to CNN-only models, with the hybrid model attaining an accuracy of 90.46%.What is the implication of the main finding?Efficient and Scalable Methodology: Utilized EfficientNet for CNN-based feature extraction and Bag-of-Words for text representation, ensuring computational efficiency and scalability.Application Potential: Addressed challenges in indoor scene recognition, such as complex backgrounds and object diversity, demonstrating significant potential for applications in robotics, intelligent surveillance, and assistive systems.Indoor scene recognition is a computer vision task that identifies various indoor environments, such as offices, libraries, kitchens, and restaurants. This research area is particularly significant for applications in robotics, security, and assistance for individuals with disabilities, as it enables the categorization of spaces and the provision of contextual information. Convolutional Neural Networks (CNNs) are commonly employed in this field. While CNNs perform well in outdoor scene recognition by focusing on global features such as mountains and skies, they often struggle with indoor scenes, where local features like furniture and objects are more critical. In this study, the “MIT 67 Indoor Scene” dataset is used to extract and combine features from both a CNN and a text-based model utilizing object recognition outputs, resulting in a two-channel hybrid model. The experimental results demonstrate that this hybrid approach, which integrates natural language processing and image processing techniques, improves the test accuracy of the image processing model by 8.3%, achieving a notable success rate. Furthermore, this study offers contributions to new application areas in remote sensing, particularly in indoor scene understanding and indoor mapping.

Recent advancements in 3D scanning technology have spurred extensive research on indoor scene point clouds. However, point clouds captured by 3D sensors are often sparse, noisy, and irregular, posing significant challenges for downstream tasks. While learning-based upsampling methods have achieved promising results, they often struggle with large-scale indoor scenes due to the high computational cost of K-Nearest Neighbor (KNN) search and insufficient reconstruction of fine-grained structures. To address these issues, we propose S 2 PU-Net, a sparse semantic-guided progressive point cloud upsampling network for indoor scenes. Specifically, we introduce a progressive upsampling mechanism based on sparse tensors, where a progressive Point Generation Block is designed to generate and prune intermediate point clouds iteratively. By employing multiple generative deconvolutions and adaptive pruning with different coefficients, our method progressively reconstructs the target point cloud while preserving fine-grained local structures. Furthermore, we integrate semantic information into each layer of the hourglass encoder through a Sparse Semantic Embedding (SSE) module, enhancing point feature extraction. To further strengthen feature representation, we propose a Multi-Scale Sparse Semantic Embedding (MSSE) module, which refines the fused features across spatial and channel dimensions. Extensive experiments on multiple indoor scene datasets demonstrate that our method consistently outperforms state-of-the-art approaches across most evaluation metrics.

Although 3D Gaussian Splatting (3DGS) has greatly advanced the development of novel view synthesis (NVS) and surface reconstruction tasks, it still faces serious challenges in indoor scenes with only sparse views due to the lack of initialization point cloud. Existing GS-like methods are geometrically initialized using point clouds from Structure from Motion (SfM), and their performance is heavily limited by the point cloud quality. However, in indoor scenes containing large areas of textureless or weakly textured regions, the SfM algorithm struggles to generate point clouds in these areas, especially when only a limited number of input views are available. This situation can cause the GS optimization process to overfit the photometric error, resulting in severe geometric degradation in regions lacking proper initialization. To address the above problem, we propose a novel sparse-view 3DGS method using global scale-aligned monocular depth to initialize Gaussian primitives and optimize them in view space, named VGA-GS. First, we design a global scale-aligned monocular depth initialization strategy, which provides geometric priors in textureless or weakly textured regions while avoiding the local geometric distortions in vanilla alignments. Second, we propose a novel Gaussian primitive representation based on view space and a refinement strategy based on pixel errors, which can constrain each primitive to always be within the field of view where it is observed to be fully optimized. Finally, we extract the surface prior from monocular depth to regularize the rendering depth and normal, and further propagate the constraints of the training view to the neighboring pseudo-views via image warping. Our method achieves state-of-the-art reconstruction performance in both textureless regions and local details for indoor scenes with sparse views. Moreover, it is highly scalable, supporting integration with various 3DGS rasterizers and depth estimation methods. We release the code at https://github.com/XT5un/VGA-GS.

We present a method for rendering novel views from a set of reference views under the assumption that scene surfaces can be approximated by planar patches. We use a set of sparse correspondences between the reference views to determine homographies through different planar patches using a clustering technique. We also obtain a segmentation of the scene in terms of planar regions visible from at least two views. Such segmentations explicitly resolve the visibility issues in novel view generation. We present results on rendering of realistic walk-through sequences for both indoor and outdoor scenes which demonstrate the applicability of our method.

With the increasing popularity of RGB-depth (RGB-D) sensor, research on the use of RGB-D sensors to reconstruct three-dimensional (3D) indoor scenes has gained more and more attention. In this paper, an automatic point cloud registration algorithm is proposed to efficiently handle the task of 3D indoor scene reconstruction using pan-tilt platforms on a fixed position. The proposed algorithm aims to align multiple point clouds using extrinsic parameters of the RGB-D camera obtained from every preset pan-tilt control point. A computationally efficient global registration method is proposed based on transformation matrices formed by the offline calibrated extrinsic parameters. Then, a local registration method, which is an optional operation in the proposed algorithm, is employed to refine the preliminary alignment result. Experimental results validate the quality and computational efficiency of the proposed point cloud alignment algorithm by comparing it with two state-of-the-art methods.

Global Navigation Satellite System (GNSS) can provide accurate absolute position but requires fine observational conditions. In urban canyons and indoor scenes, GNSS suffers from severe performance degradation and is even unavailable. Thereby, to provide ubiquitous positioning solutions, it is very necessary to incorporate other complementary sensors. A popular option is the 3D LiDAR sensor, which emits its own light and thus is robust to illumination variation. 3D LiDAR sensors can efficiently capture plentiful point patterns of the ambient environment. Besides perception tasks, the obtained point clouds are also used for relative localization in a registration fashion, known as the LiDAR odometry (LO). Generally, LO is based on structural information, where edge and planar features are extensively exploited.In this work, we propose a novel method aiming to efficiently extract planes from the sparse and noisy 3D LiDAR point clouds. To fully exploit the scanning pattern of this sensor, our method follows a framework of point-to-line-to-plane. The point cloud is firstly projected onto a range image by investigating the azimuth and elevation of each point. In the point-to-line stage, consecutive flat points in a column are grouped into line segments, where a new flat point detector is introduced. In the line-to-plane stage, we extend the classical line extraction method, i.e., Douglas-Peucker algorithm, to find planes in line segments. Considering the over-segmentation caused by occlusion and deformation, we finally conduct region growing and merging to acquire more complete results. Most importantly, we bridge the measurement noise model and the parameter uncertainty via error propagation to determine reasonable thresholds throughout our method.We test the proposed method on datasets collected by various LiDAR sensors. The experiments are conducted in indoor scenes and urban scenes, which contains abundant planar objects such as walls and building facades. Three point-level metrics, namely positive predictive value (PPV), true positive rate (TPR) and F1 score, are employed for quantitative evaluation. The average PPV, TPR, F1 of the proposed method are 89.92%, 86.38% and 88.11%, respectively. The results show that the proposed method is able to recover the dominant planar structure, which is valuable to LO. Moreover, compared to the rotation rate of the LiDAR sensor, which is generally set to 10 Hz, the average runtime of the proposed method is 15.6 ms/frame, so it is qualified for online operation.

Since the development of 3D applications, the point cloud, as a spatial description easily acquired by sensors, has been widely used in multiple areas such as SLAM and 3D reconstruction. Point Cloud Compression (PCC) has also attracted more attention as a primary step before point cloud transferring and saving, where the geometry compression is an important component of PCC to compress the points geometrical structures. However, existing non-learning-based geometry compression methods are often limited by manually pre-defined compression rules. Though learning-based compression methods can significantly improve the algorithm performances by learning compression rules from data, they still have some defects. Voxel-based compression networks introduce precision errors due to the voxelized operations, while point-based methods may have relatively weak robustness and are mainly designed for sparse point clouds. In this work, we propose a novel learning-based point cloud compression framework named 3D Point Cloud Geometry Quantiation Compression Network (3QNet), which overcomes the robustness limitation of existing point-based methods and can handle dense points. By learning a codebook including common structural features from simple and sparse shapes, 3QNet can efficiently deal with multiple kinds of point clouds. According to experiments on object models, indoor scenes, and outdoor scans, 3QNet can achieve better compression performances than many representative methods.

Component detection of spacecraft is significant for on-orbit operation and space situational awareness. Solar wings and main body are the major components of most spacecrafts, and can be described by geometric primitives like planes, cuboid or cylinder. Based on this prior, pipeline to automatically detect the basic components of spacecraft in 3D point clouds is presented, in which planes, cuboid and cylinder are successively detected. The planar patches are first detected as possible solar wings in point clouds of the recorded object. As for detection of the main body, inferring a cuboid main body from the detected patches is first attempted, and a further attempt to extract a cylinder main body is made if no cuboid exists. Dimensions are estimated for each component. Experiments on satellite point cloud data that are recovered by image-based reconstruction demonstrated effectiveness and accuracy of this pipeline.

Point cloud semantic segmentation in urban scenes plays a vital role in intelligent city modeling, autonomous driving, and urban planning. Point cloud semantic segmentation based on deep learning methods has achieved significant improvement. However, it is also challenging for accurate semantic segmentation in large scenes due to complex elements, variety of scene classes, occlusions, and noise. Besides, most methods need to split the original point cloud into multiple blocks before processing and cannot directly deal with the point clouds on a large scale. We propose a novel context-aware network (CAN) that can directly deal with large-scale point clouds. In the proposed network, a Local Feature Aggregation Module (LFAM) is designed to preserve rich geometric details in the raw point cloud and reduce the information loss during feature extraction. Then, in combination with a Global Context Aggregation Module (GCAM), capture long-range dependencies to enhance the network feature representation and suppress the noise. Finally, a Context-Aware Upsampling Module (CAUM) is embedded into the proposed network to capture the global perception from a broad perspective. The ensemble of low-level and high-level features facilitates the effectiveness and efficiency of 3D point cloud feature refinement. Comprehensive experiments were carried out on three large-scale point cloud datasets in both outdoor and indoor environments to evaluate the performance of the proposed network. The results show that the proposed method outperformed the state-of-the-art representative semantic segmentation networks, and the overall accuracy (OA) of Tongji-3D, Semantic3D, and S3DIS is 96.01%, 95.0%, and 88.55%, respectively.

Plane detection in a point cloud is one of the primary step for various applications, such as computer vision, ground plane detection for autonomous navigation, obstacle detection, indoor scene reconstruction, etc. In this paper, a new algorithm for simultaneous detection of multiple planes in a point cloud is proposed. The proposed method is a two-step process. In the first step, the surface normals are automatically clustered into probable plane orientations (angular clusters) within a user specified angle threshold, without a priori knowledge of number of planes. In the second step, the angular clusters are further clustered into separate planes, within a user specified distance threshold, based on the normal distances of the points in an angular cluster. In contrast to popular random sampling based methods, the proposed method uses deterministic approach to simultaneously detect all possible planes and has comparable results with the existing methods and is two times faster. The proposed method is implemented using Open3d point cloud library and evaluated on datasets having variety of indoor scenes.

Inter-frame matching of laser point clouds is the key to SLAM's autonomous positioning and 3D map construction. Aiming at the failure of inter frame matching of multi-line lidar SLAM during fast field of view conversion, this paper proposes a spatial 3D mapping method using 180 degree large-field-angle lidar to improve A-LOAM. Firstly, the nonground feature points are obtained by dual-threshold ground filtering, and a novel point cloud feature extraction method based on curvature is used to realize the feature point matching of large-field-angle point cloud; Secondly, using the matching methods of edge point to edge line and plane point to plane block, through the robust frame matching of large-field-angle point cloud data, the spatial 3D mapping is realized; Finally, three-dimensional mapping experiments are carried out on the large-field-angle point cloud data collected in different scenes. The results show that the spatial 3D mapping based on robust frame matching of large-field-angle lidar can reach the accuracy of nearly centimeter level in indoor scene and decimeter level in outdoor scene. It has application prospects in 3D mapping and other fields.