Introducing an indoor object classification dataset including sparse point clouds from mmWave radar

Accurate human activity recognition (HAR) is the key to enable emerging context-aware applications that require an understanding and identification of human behavior, e.g., monitoring disabled or elderly people who live alone. Traditionally, HAR has been implemented either through ambient sensors, e.g., cameras, or through wearable devices, e.g., a smartwatch, with an inertial measurement unit (IMU). The ambient sensing approach is typically more generalizable for different environments as this does not require every user to have a wearable device. However, utilizing a camera in privacy-sensitive areas such as a home may capture superfluous ambient information that a user may not feel comfortable sharing. Radars have been proposed as an alternative modality for coarse-grained activity recognition that captures a minimal subset of the ambient information using micro-Doppler spectrograms. However, training fine-grained, accurate activity classifiers is a challenge as low-cost millimeter-wave (mmWave) radar systems produce sparse and non-uniform point clouds. In this paper, we propose RadHAR, a framework that performs accurate HAR using sparse and non-uniform point clouds. RadHAR utilizes a sliding time window to accumulate point clouds from a mmWave radar and generate a voxelized representation that acts as input to our classifiers. We evaluate RadHAR using a low-cost, commercial, off-the-shelf radar to get sparse point clouds which are less visually compromising. We evaluate and demonstrate our system on a collected human activity dataset with 5 different activities. We compare the accuracy of various classifiers on the dataset and find that the best performing deep learning classifier achieves an accuracy of 90.47%. Our evaluation shows the efficacy of using mmWave radar for accurate HAR detection and we enumerate future research directions in this space.

The fusion of laser point cloud and visual image depends on the point cloud density and the target framing effect, the traditional laser point cloud processing for sparse point cloud clustering effect is poor, it is difficult to frame small objects as well as medium and long distance objects. Then the subsequent sensor fusion is easy to miss the recognition of obstacles. In this paper, we improve the frame selection method for sparse point clouds, firstly build a deep learning framework pointpillar, use pointpillar to frame the sparse laser point clouds, then spatially calibrate the lidar coordinate system and camera coordinate system, project the lidar point clouds to the camera image, improve the late fusion method, effectively use the detection results of single sensor, and finally The late-fusion is performed with the target detection results of the camera image to output the exact distance as well as the category of the target. Experiments show that compared with the recognition effect of the traditional fusion algorithm, the number of frames is increased by 6 and the missed recognition rate is reduced from 31.41% to 12.31%.

Three-dimensional (3D) point clouds have a wide range of applications in the field of 3D vision. The quality of the acquired point cloud data considerably impacts the subsequent work of point cloud processing. Due to the sparsity and irregularity of point cloud data, processing point cloud data has always been challenging. However, existing deep learning-based point cloud dense reconstruction methods suffer from excessive smoothing of reconstruction results and too many outliers. The reason for this is that it is not possible to extract features for local and global features at different scales and provide different levels of attention to different regions in order to obtain long-distance dependence for dense reconstruction. In this paper, we use a parallel multi-scale feature extraction module based on graph convolution and an upsampling method with an added multi-head attention mechanism to process sparse and irregular point cloud data to obtain extended point clouds. Specifically, a point cloud training patch with 256 points is inputted. The PMS module uses three residual connections in the multi-scale feature extraction stage. Each PMS module consists of three parallel DenseGCN modules with different size convolution kernels and different averaging pooling sizes. The local and global feature information of the augmented receptive field is extracted efficiently. The scale information is obtained by averaging the different pooled augmented receptive fields. The scale information was obtained using the different average pooled augmented receptive fields. The upsampling stage uses an upsampling rate of r=4, The self-attentive features with a different focus on different point cloud data regions obtained by fusing different weights make the feature representation more diverse. This operation avoids the bias of one attention, and each focuses on extracting valuable fine-grained feature information. Finally, the coordinate reconstruction module obtains 1024 dense point cloud data. Experiments show that the proposed method demonstrates good evaluation metrics and performance and is able to obtain better visual quality. The problems of over-smoothing and excessive outliers are effectively mitigated, and the obtained sparse point cloud is more dense.

GSP-enhanced embedded hardware framework for real-time multi-class road target detection in ADAS using mmWave radar

It is an important task to reconstruct surfaces from 3D point clouds. Current methods are able to reconstruct surfaces by learning Signed Distance Functions (SDFs) from single point clouds without ground truth signed distances or point normals. However, they require the point clouds to be dense, which dramatically limits their performance in real applications. To resolve this issue, we propose to reconstruct highly accurate surfaces from sparse point clouds with an on-surface prior. We train a neural network to learn SDFs via projecting queries onto the surface represented by the sparse point cloud. Our key idea is to infer signed distances by pushing both the query projections to be on the surface and the projection distance to be the minimum. To achieve this, we train a neural network to capture the on-surface prior to determine whether a point is on a sparse point cloud or not, and then leverage it as a differentiable function to learn SDFs from unseen sparse point cloud. Our method can learn SDFs from a single s parse point cloud without ground truth signed distances or point normals. Our numerical evaluation under widely used benchmarks demonstrates that our method achieves state-of-the-art reconstruction accuracy, especially for sparse point clouds. Code and data are available at https://github.com/mabaorui/OnSurfacePrior.

Driver activity recognition has become crucial for intelligent transportation and automotive safety systems. However, existing studies mainly focus on fatigue-related behaviors while neglecting other activities for analyzing driver behavior and intent. In this work, we introduce a novel dataset for fine-grained driver activities, utilizing diverse sensors such as mmWave radars, RGB, and depth cameras, each of which includes three camera angles: body, face, and hands. This multi-modal and multi-angle approach allows for comprehensive driver behavior analysis, including hand gestures, head movement, and object interactions. Moreover, including mmWave radars provides significant privacy advantages, as the sparse dynamic point clouds prevent the identification of the driver's face and other personal information. This dataset is valuable for researchers and developers on driver activity recognition and behavior analysis. It enables the development and evaluation of robust, privacy-conscious solutions for improving road safety, driver assistance, and in-vehicle interaction. Furthermore, the multi-modal nature of the data enables the exploration of sensor fusion techniques, unlocking the full potential of diverse sensing modalities to understand complex driver behaviors.

Development of vehicle- recognition method on water surfaces using LiDAR data: SPD2 (spherically stratified point projection with diameter and distance)

Estimation of indoor location is a key capability for indoor location-based services. Among various solutions, single-chip low-cost millimeter-wave (mmWave) radar as an emerging technology are getting more and more attention due to their inherent advantages such as penetration ability, low-power consumption. Currently, the vast majority of work is focused on hybrid positioning based on mmWave radar with other sensors, and there is a lack of research on single radar indoor positioning. In this article, we propose a novel indoor positioning system based on single-radar sensor, which determines the trajectory by estimating the velocity and heading of moving radar from sparse point clouds. Doppler velocity of detected object can be able to feedback the radar velocity, and for another hand, the proposed Cluster Center Association (CCA) method combined with normal distributions transform (NDT) can be used for heading estimation. In addition, we proposed the motion consistency constraints to enhance the robustness of the system. The experimental results show that the traditional NDT algorithm will fail due to the sparse point cloud and severe indoor multipath, and the proposed algorithm can provide the positioning accuracy within 1 m and the heading estimation error of 3°.

Lidar has received a lot of attention due to its precise ranging accuracy. Ground points filtering is an important task in point clouds processing. It’s a challenge to model the ground surface and filter the point clouds accurately in the case of complex ground undulations, occlusions, and sparse point clouds. A novel ground surface modeling method based on a hybrid regression technique is proposed in this paper. The method integrates Gaussian process regression (GPR) and robust locally weighted regression (RLWR) by dividing the point clouds that are projected on the polar grid map into radial and circumferential filtering processes to form a hybrid regression model, which has the ability to eliminate the influence of outliers and model the ground surface robustly. First, the RLWR combined with gradient filter is applied to fit the sampled points in the radial direction, which will exclude outliers and get the fitting ground line. All radial fitting lines constitute the seed skeleton of the whole plane. Then, based on the seeds in the same circumferential of the skeleton, the GPR is applied to construct the ground surface model. The comparative experiments are implemented quantitatively and qualitatively on the simulated point clouds and measured data. The results show that the proposed method performs well in most real scenarios, even in the cases of ground undulation, occlusion, and sparse point clouds.

Registration aims at merging multiple scans to cover all scenes of a large environment. Thus, it is crucial to many civil infrastructure applications based on three-dimensional (3D) models. However, in many real-world scenarios, it is necessary to align point clouds with low-density or small overlaps. It is difficult to extract stable features and enough features for registration, whether keypoint features or overall posture features, under this condition. Existing methods cannot solve this problem well. This work proposed an end-to-end registration network that can self-adaptively focus on the overlap. The network learned to directly encode posture information from the overlapping area instead of using sparse keypoint correspondences, which makes the network more generalized and efficient. This work also proposed a self-supervised overlapping detector as an extension module to expand the use of this network to align large-scale point clouds of indoor building environments. The proposed detector is compatible with any registration approaches to promote their accuracy and efficiency further. The proposed network was experimentally demonstrated to outperform the state-of-the-art methods in registering sparse and low-overlapping point clouds, with higher robustness to point density and overlap ratio change. The proposed detector can reliably detect the overlapping area and empower the network to accurately align the sparse and low-overlapping point clouds of the large-scale indoor scene, thus simplifying and promoting laser scanning practices in civil infrastructure applications.

We study an unsupervised domain adaptation problem for the semantic labeling of 3D point clouds, with a particular focus on domain discrepancies induced by different LiDAR sensors. Based on the observation that sparse 3D point clouds are sampled from 3D surfaces, we take a Complete and Label approach to recover the underlying surfaces before passing them to a segmentation network. Specifically, we design a Sparse Voxel Completion Network (SVCN) to complete the 3D surfaces of a sparse point cloud. Unlike semantic labels, to obtain training pairs for SVCN requires no manual labeling. We also introduce local adversarial learning to model the surface prior. The recovered 3D surfaces serve as a canonical domain, from which semantic labels can transfer across different LiDAR sensors. Experiments and ablation studies with our new benchmark for cross-domain semantic labeling of LiDAR data show that the proposed approach provides 6.3-37.6% better performance than previous domain adaptation methods.

Current 3D object detection methods achieve accurate and efficient results on the standard point cloud dataset. However, in real-world applications, the point cloud samples obtained in the real-time running may be much sparser due to various reasons (occlusion, low reflectivity of objects and fewer laser beams) and existing methods do not consider the limitations of their models on sparse point clouds. To improve the robustness of an object detector to sparser point clouds, we propose PointDrop, which learns to drop the features of some key points in the point clouds to generate challenging sparse samples for data augmentation. Moreover, PointDrop is able to adjust the difficulty of the generated samples based on the capacity of the detector and thus progressively improve the performance of the detector. We create two sparse point clouds datasets from the KITTI dataset to evaluate our method, and the experimental results show that PointDrop significantly improves the robustness of the detector to sparse point clouds.

Surface reconstruction from 3D point clouds has a wide range of applications. In this paper, we focus on the reconstruction from raw, sparse point clouds. Although some existing methods work on this topic, the results often suffer from geometric defects. To solve this problem, we propose a novel method that optimizes a neural network (referred to as signed distance function) to fit the Signed Distance Field (SDF) from sparse point clouds. The signed distance function is optimized by projecting query points to its iso-surface accordingly. Our key idea is to encourage both the direction and distance of projection to be correct through the supervision provided by a nearest neighbor prior. In addition, we mitigate the error propagated from the prior function by augmenting the low-frequency components in the input. In our implementation, the nearest neighbor prior is trained with a large-scale local geometry dataset, and the positional encoding with a specified spectrum is used as a regularization for the optimization process. Experiments on the ShapeNetCore dataset demonstrate that our method achieves better accuracy than SDF-based methods while preserving smoothness.

Automated sparse 3D point cloud generation of infrastructure using its distinctive visual features

The effective representation and feature extraction of 3-D scenes from sparse and unstructured point clouds pose a significant challenge in 3-D object detection. In this article, we propose TransMRE, a network that enables fully sparse multiple observation plane feature fusion using LiDAR point clouds as single-modal input. TransMRE achieves this by sparsely factorizing a 3-D voxel scene into three separate observation planes: XY, XZ, and YZ planes. In addition, we propose Observation Plane Sparse Fusion and Interaction to explore the internal relationship between different observation planes. The Transformer mechanism is employed to realize feature attention within a single observation plane and feature attention across multiple observation planes. This recursive application of attention is done during multiple observation plane projection feature aggregation to effectively model the entire 3-D scene. This approach addresses the limitation of insufficient feature representation ability under a single bird’s-eye view (BEV) constructed by extremely sparse point clouds. Furthermore, TransMRE maintains the full sparsity property of the entire network, eliminating the need to convert sparse feature maps into dense feature maps. As a result, it can be effectively applied to LiDAR point cloud data with large scanning ranges, such as Argoverse 2, while ensuring low computational complexity. Extensive experiments were conducted to evaluate the effectiveness of TransMRE, achieving 64.9 mAP and 70.4 NDS on the nuScenes detection benchmark, and 32.3 mAP on the Argoverse 2 detection benchmark. These results demonstrate that our method outperforms state-of-the-art methods.