3D object detection combining semantic and geometric features from point clouds

Dynamic street-scene reconstruction from sparse viewpoints over long temporal spans is challenged by temporal instability, ghosting near occlusions, and background drift. This paper presents SPT-Gauss (Semantic Prior and Temporal constraint-enhanced Gaussian splatting), a Gaussian-splatting framework that improves dynamic reconstruction without object-level annotations by combining dense semantic priors with lightweight, parameter-level temporal regularization. SPT-Gauss distills per-pixel semantic features from a frozen 2D foundation model into 4D Gaussian primitives, estimates static and dynamic regions via a dual-evidence motion mask, and regularizes temporal parameters through a semantic-guided velocity constraint and a static-lifetime prior to suppress spurious background motion. Experiments on the Waymo Open Dataset and KITTI (Karlsruhe Institute of Technology and Toyota Technological Institute) show consistent improvements over representative baselines in both 4D reconstruction and novel-view synthesis, with reduced temporal artifacts and improved fidelity in motion-challenging regions.

Object detection for vehicles and pedestrians is extremely difficult to achieve in autopilot applications for the Internet of vehicles, and it is a task that requires the ability to locate and identify smaller targets even in complex environments. This paper proposes a single-stage object detection network (YOLOv3-promote) for the detection of vehicles and pedestrians in complex environments in cities, which improves on the traditional You Only Look Once version 3 (YOLOv3). First, spatial pyramid pooling is used to fuse local and global features in an image to better enrich the expression ability of the feature map and to more effectively detect targets with large size differences in the image; second, an attention mechanism is added to the feature map to weight each channel, thereby enhancing key features and removing redundant features, which allows for strengthening the ability of the feature network to discriminate between target objects and backgrounds; lastly, the anchor box derived from the K-means clustering algorithm is fitted to the final prediction box to complete the positioning and identification of target vehicles and pedestrians. The experimental results show that the proposed method achieved 91.4 mAP (mean average precision), 83.2 F1 score, and 43.7 frames per second (FPS) on the KITTI (Karlsruhe Institute of Technology and Toyota Technological Institute) dataset, and the detection performance was superior to the conventional YOLOv3 algorithm in terms of both accuracy and speed.

In order to solve the stereo matching problem better, the accuracy of the algorithm in the stereo matching task needs to be improved. To tackle the problem, we proposed a Dual Spatial Pyramid Pooling Stereo Matching Network with Attention Mechanism (DSANet), a stereo matching network with better multi-scale feature extraction capability based on the basic structure of Pyramid Stereo Matching Network (PSMNet). The network consists of two main modules: the dual spatial pyramid pooling module and the attention module. The former module extracts multi-scale features from different branches at the same time. After concatenating these different features, the latter module will optimize the fused features and generate the cost volume with different branches multi-scale feature. The proposed approach was evaluated on the Karlsruhe Institute of Technology and Toyota Technological Institute (KITTI) 2015 dataset. Our experiment results on the KITTI 2015 dataset show that the mismatch rate of our network is 3.02%. Compared with the basic structure of PSMNet, the proposed method improves the effect of stereo matching.

Accurate object detection is a fundamental requirement for autonomous systems to operate in dynamic urban environments. Considering the intricacy of the environment and the occlusion of objects, various point cloud based three-dimensional (3D) object detection methods have been proposed, such as point-based or voxel-based methods. In this paper, a feature selection mechanism is proposed in a voxel-based method to generate bird’s eye view (BEV) feature maps from the original point cloud. For the 3D Backbone, the U-shaped structure and single scale feature selection module are combined. After combining high-level semantics and low-level fine-grained features, the optimized features after the BEV feature maps are applied to region of interest (RoI) refinement, so that the voxel features can better serve the subsequent 3D object detection. The experimental results on KITTI dataset show higher 3D object detection accuracy compared to the state-of-the-art 3D detection methods, which reflects the effectiveness of the proposed architecture.

In the recent 3D object detection methods for point clouds, the combination of point-based methods and voxel-based methods is gradually becoming a trend. Point-based methods retain the accurate position and pose information in the raw points and voxel-based methods get multi-scale structure information through the 3D backbone. However, because of the sparsity and irregularity of point clouds, both representations ignore the context information, which is important for the detection of sparse and small objects. To solve this problem, we propose a multi-stream feature aggregation network to extract features from three representations of the point cloud for object detection. Specifically, we exploit multi-stream features extracted from point, voxel, and perspective view (PV) respectively on a parallel way, where the complementary information between different perspectives can be used to enrich the feature representations, especially for the perspective view containing rich semantic context information. Secondly, to eliminate redundant information and better exploit the correlation between different feature representations, we design an attention-based multi-stream feature fusion module (MSFF) to combine features from three information streams. Besides, we introduce a new voxel RoI pooling with the self-attention in the second refinement stage, which can further strengthen the connection between local features in the proposal to obtain accurate classification and localization predictions. Our method achieves progressive results on the KITTI dataset, especially in the cyclist category, which improves the baseline significantly by 5.56%, 4.73%, 5.16% AP in the test set for easy, moderate, and hard levels respectively. Code will be available at https://github.com/june2678/MR F.

Unmanned Surface Vehicles (USVs) are commonly equipped with multi-modality sensors. Fully utilized sensors could improve object detection of USVs. This could further contribute to better autonomous navigation. The purpose of this paper is to solve the problems of 3D object detection of USVs in complicated marine environment. We propose a 3D object detection Depth Neural Network based on multi-modality data of USVs. This model includes a modified Proposal Generation Network and Deep Fusion Detection Network. The Proposal Generation Network improves feature extraction. Meanwhile, the Deep Fusion Detection Network enhances the fusion performance and can achieve more accurate results of object detection. The model was tested on both the KITTI 3D object detection dataset (A project of Karlsruhe Institute of Technology and Toyota Technological Institute at Chicago) and a self-collected offshore dataset. The model shows excellent performance in a small memory condition. The results further prove that the method based on deep learning can give good accuracy in conditions of complicated surface in marine environment.

3D object detection from raw and sparse point clouds has been far less treated to date, compared with its 2D counterpart. In this paper, we propose a novel framework called FVNet for 3D front-view proposal generation and object detection from point clouds. It consists of two stages: generation of front-view proposals and estimation of 3D bounding box parameters. Instead of generating proposals from camera images or bird's-eye-view maps, we first project point clouds onto a cylindrical surface to generate front-view feature maps which retains rich information. We then introduce a proposal generation network to predict 3D region proposals from the generated maps and further extrude objects of interest from the whole point cloud. Finally, we present another network to extract the point-wise features from the extruded object points and regress the final 3D bounding box parameters in the canonical coordinates. Our framework achieves real-time performance with 12ms per point cloud sample. Extensive experiments on the 3D detection benchmark KITTI show that the proposed architecture outperforms state-of-the-art techniques which take either camera images or point clouds as input, in terms of accuracy and inference time.

Dynamic graph transformer for 3D object detection

Geometric information constraint 3D object detection from LiDAR point cloud for autonomous vehicles under adverse weather

Recently, deep learning methods and multisensory fusion have been applied to address odometry challenges in unmanned ground vehicles (UGVs). In this paper, we propose an end-to-end visual-lidar-inertial odometry framework to enhance the accuracy of pose estimation. Grayscale images, 3D point clouds, and inertial data are used as inputs to overcome the limitations of a single sensor. Convolutional neural network (CNN) and recurrent neural network (RNN) are employed as encoders for different sensor modalities. In contrast to previous multisensory odometry methods, our framework introduces a novel attention-based fusion module that remaps feature vectors to adapt to various scenes. Evaluations on the Karlsruhe Institute of Technology and Toyota Technological Institute at Chicago (KITTI) odometry benchmark demonstrate the effectiveness of our framework.

In recent years, 3D object detection has attracted considerable interest from researchers due to its crucial importance in scene understanding tasks for mobile robots. The task of 3D object detection is to determine the position, orientation, scale, and category of objects in a scene, with multimodal data inputs. These inputs consist of two distinct types: images and point clouds. Images provide colour information about the scene, while point clouds deliver geometric information. The fusion of these two diverse data types can enhance the detection of 3D objects by allowing them to complement each other. In this paper, a review of real-time multimodal 3D object detection from the perspective of multimodal sensor fusion is presented. It is important to note that previous reviews of multimodal 3D object detection have been limited to autonomous driving scenarios. In contrast, this paper reviews multimodal 3D object detection approaches in both indoor and outdoor scenarios. In addition, the popular datasets and assessment metrics employed for 3D object detection are described, as well as the issues and future prospects of 3D object detection.

Segmenting or detecting objects in sparse Lidar point clouds are two important tasks in autonomous driving to allow a vehicle to act safely in its 3D environment. The best performing methods in 3D semantic segmentation or object detection rely on a large amount of annotated data. Yet annotating 3D Lidar data for these tasks is tedious and costly. In this context, we propose a self-supervised pretraining method for 3D perception models that is tailored to autonomous driving data. Specifically, we leverage the availability of synchronized and calibrated image and Lidar sensors in autonomous driving setups for distilling self-supervised pre-trained image representations into 3D models. Hence, our method does not require any point cloud nor image annotations. The keyingredient of our method is the use of superpixels which are used to pool 3D point features and 2D pixel features in visually similar regions. We then train a 3D network on the self-supervised task of matching these pooled point features with the corresponding pooled image pixel features. The advantages of contrasting regions obtained by superpixels are that: (1) grouping together pixels and points of visually coherent regions leads to a more meaningful contrastive task that produces features well adapted to 3D semantic segmentation and 3D object detection; (2) all the different regions have the same weight in the contrastive loss regardless of the number of 3D points sampled in these regions; (3) it mitigates the noise produced by incorrect matching of points and pixels due to occlusions between the different sensors. Extensive experiments on autonomous driving datasets demonstrate the ability of our image-to-Lidar distillation strategy to produce 3D representations that transfer well on semantic segmentation and object detection tasks.

The 3D object detection is becoming indispensable for environmental perception in autonomous driving. Light detection and ranging (LiDAR) point clouds often fail to distinguish objects with similar structures and are quite sparse for distant or small objects, thereby introducing false and missed detections. To address these issues, LiDAR is often fused with cameras due to the rich textural information provided by images. However, current fusion methods suffer the inefficient data representation and inaccurate alignment of heterogeneous features, leading to poor precision and low efficiency. To this end, we propose a plug-and-play module termed range-image fusion (RI-Fusion) to achieve an effective fusion of LiDAR and camera data, designed to be easily accessible by existing mainstream LiDAR-based algorithms. In this process, we design an image and point cloud alignment method by converting a point cloud into a compact range-view representation through a spherical coordinate transformation. The range image is then integrated with a corresponding camera image utilizing an attention mechanism. The original range image is then concatenated with fusion features to retain point cloud information, and the results are projected onto a spatial point cloud. Finally, the feature-enhanced point cloud can be input into a LiDAR-based 3D object detector. The results of validation experiments involving the KITTI 3D object detection benchmark showed that our proposed fusion method significantly enhanced multiple mainstream LiDAR-based 3D object detectors, PointPillars, SECOND, and Part <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\text{A}{^{2}}$ </tex-math></inline-formula> , improving the 3D mAP (mean Average Precision) by 3.61%, 2.98%, and 1.27%, respectively, particularly for small objects such as pedestrians and cyclists.

Light Detection And Ranging (LiDAR) sensors can generate a number of sequential 3D point clouds, which are widely deployed in many real-world systems. 3D object detection in point clouds, is one of the most fundamental tasks. Unfortunately, the existing 3D object detection methods degrade in snowy weather, because in that situation the annotated samples are difficult to collect. To solve this issue, we propose a novel GAN-based Snowfall Point-cloud AugmentOR (GAN spaor ) to generate high-quality synthetic snowfall point clouds as augmentations. The basic idea of GAN spaor is to transfer annotated point clouds to snowfall versions by simultaneously learning the global style of real snowfall point clouds and the local details of physics-induced ones. Our framework fuses data-driven and physical modeling methods for rapidly generating data in snowy weather. To evaluate the effectiveness of GAN spaor , we employ a number of recent 3D object detection methods and train them by using the synthetic samples of GAN spaor as auxiliary augmentations. Moreover, we conduct a comparative analysis of the characteristics of the data distributions of the snowy point clouds synthesized by GAN spaor . Experimental results demonstrate that GAN spaor can improve the performance of 3D object detection methods compared with other existing snowfall point cloud simulators.

Deep neural networks have made tremendous progress in 3D object detection, which is an important task especially in autonomous driving scenarios. Benefited from the breakthroughs in deep learning and sensor technologies, 3D object detection methods based on different sensors, such as camera and LiDAR, have developed rapidly. Meanwhile, more and more researches notice that the abundant information contained in the multi-view data can be used to obtain more accurate understanding of the 3D surrounding environment. Therefore, many sensor-fusion 3D object detection methods have been proposed. As safety is critical in autonomous driving and the deep neural networks are known to be vulnerable to adversarial examples with visually imperceptible perturbations, it is significant to investigate adversarial attacks for 3D object detection. Recent works have shown that both image-based and LiDAR-based networks can be attacked by the adversarial examples while the attacks to the sensor-fusion models, which tend to be more robust, haven't been studied. To this end, we propose a simple multi-view correlation based adversarial attack method for the camera-LiDAR fusion 3D object detection models and focus on the black-box attack setting which is more practical in real-world systems. Specifically, we first design a generative network to generate image adversarial examples based on an auxiliary image semantic segmentation network. Then, we develop a cross-view perturbation projection method by exploiting the camera-LiDAR correlations to map each image adversarial example to the space of the point cloud data to form the point cloud adversarial examples in the LiDAR view. Extensive experiments on the KITTI dataset demonstrate the effectiveness of the proposed method.