Efficient 3D point cloud network model for reconstruction of 3D point cloud, object classification, and part segmentation

We propose a framework for three-dimensional (3D) object recognition and classification in very low illumination environments using convolutional neural networks (CNNs). 3D images are reconstructed using 3D integral imaging (InIm) with conventional visible spectrum image sensors. After imaging the low light scene using 3D InIm, the 3D reconstructed image has a higher signal-to-noise ratio than a single 2D image, which is a result of 3D InIm being optimal in the maximum likelihood sense for read-noise dominant images. Once 3D reconstruction has been performed, the 3D image is denoised and regions of interest are extracted to detect 3D objects in a scene. The extracted regions are then inputted into a CNN, which was trained under low illumination conditions using 3D InIm reconstructed images, to perform object recognition. To the best of our knowledge, this is the first report of utilizing 3D InIm and convolutional neural networks for 3D training and 3D object classification under very low illumination conditions.

This paper proposes a method for recognition and classification of 3D objects using 2D moments and HMLP network. The 2D moments are calculated based on 2D intensity images taken from multiple cameras that have been arranged using multiple views technique. 2D moments are commonly used for 2D pattern recognition. However, the current study proves that with some adaptation to multiple views technique, 2D moments are sufficient to model 3D objects. In addition, the simplicity of 2D moment's calculation reduces the processing time for feature extraction, thus decreases the recognition time. The 2D moments were then fed into a neural network for classification of the 3D objects. In the current study, hybrid multi-layered perceptron (HMLP) network is proposed to perform the classification. Two distinct groups of objects that are polyhedral and free-form objects were used to access the performance of the proposed method. The recognition results show that the proposed method has successfully classified the 3D object with the accuracy of up to 100%.

3D Point clouds are irregular and unordered. Recently, neural networks applying on point clouds have shown superior performance on 3D object classification and segmentation. PointNet has achieved competitive performance in classification tasks, however, it ignores the important local information of point clouds, which carries the important information in 3D geometric space. The informations lack will reduce the accuracy of object classification. In this paper, a new deep neural network model is proposed. Specifically, a multiscale network is used to extract local features of points to improve the accuracy of PointNet and a deep residual network is used to avoid the vanishing gradient problem in the process of deep training network. Experiments show that the proposed network architecture achieves better performance than state-of-the-art methods on multiple challenging benchmark datasets and tasks.

Unlike 2-dimensional (2D) images, direct 3-dimensional (3D) point cloud processing using deep neural network architectures is challenging, mainly due to the lack of explicit neighbor relationships. Many researchers attempt to remedy this by performing an additional voxelization preprocessing step. However, this adds additional computational overhead and introduces quantization error issues, limiting an accurate estimate of the underlying structure of objects that appear in the scene. To this end, in this article, we propose a deep network that can directly consume raw unstructured point clouds to perform object classification and part segmentation. In particular, a Deep Feature Transformation Network (DFT-Net) has been proposed, consisting of a cascading combination of edge convolutions and a feature transformation layer that captures the local geometric features by preserving neighborhood relationships among the points. The proposed network builds a graph in which the edges are dynamically and independently calculated on each layer. To achieve object classification and part segmentation, we ensure point order invariance while conducting network training simultaneously—the evaluation of the proposed network has been carried out on two standard benchmark datasets for object classification and part segmentation. The results were comparable to or better than existing state-of-the-art methodologies. The overall score obtained using the proposed DFT-Net is significantly improved compared to the state-of-the-art methods with the ModelNet40 dataset for object categorization.

Nowadays, point clouds are frequently gathered by 3D scanners such as Lidar and Kinect, which produces thousands of point cloud models. Point cloud processing is vital to 3D vision, especially 3D object recognition, positioning, and navigation technology. Addressing uneven data density caused by coordinate frame transformations and the inherent problem of insufficient context connection in point clouds, the DiM-PCNet (Multi-scale and Multi-level Point Clouds Classification Network, and the Di is a prefix to represent double M in Multi-scale and Multi-level) is proposed in this paper. DiM-PCNet is provided for object classification with multi-scale and multi-level features. We encode the point cloud in multi-scale and fully fuse the features with the raw point cloud for keeping the context relationship. In DiM-PCNet, we sample the point clouds from eight parts for multi-scales feature extraction. The multi-scales features are fully fused by multi-level pyramid models. The multi-scale and multi-level strategies are applied in DiM-PCNet, in which the abundant and important features of point clouds are extracted and utilized in the 3D object classification. It is worth noting that the DiM-PCNet feature block can be embedded into the segmentation net, where the accuracy achieved is 87.1%. We conducted experiments on ShapeNet and ModelNet40 and the experimental results show that DiM-PCNet achieves state-of-the-art performance in 3D object classification. The experiment shows competitive performance on robustness and segmentation tasks.

Environment perception remains one of the key tasks in autonomous driving for which solutions have yet to reach maturity. Multi-modal approaches benefit from the complementary physical properties specific to each sensor technology used, boosting overall performance. The added complexity brought on by data fusion processes is not trivial to solve, with design decisions heavily influencing the balance between quality and latency of the results. In this paper we present our novel real-time, 360 enhanced perception component based on low-level fusion between geometry provided by the LiDAR-based 3D point clouds and semantic scene information obtained from multiple RGB cameras, of multiple types. This multi-modal, multi-sensor scheme enables better range coverage, improved detection and classification quality with increased robustness. Semantic, instance and panoptic segmentations of 2D data are computed using efficient deep-learning-based algorithms, while 3D point clouds are segmented using a fast, traditional voxel-based solution. Finally, the fusion obtained through point-to-image projection yields a semantically enhanced 3D point cloud that allows enhanced perception through 3D detection refinement and 3D object classification. The planning and control systems of the vehicle receives the individual sensors’ perception together with the enhanced one, as well as the semantically enhanced 3D points. The developed perception solutions are successfully integrated onto an autonomous vehicle software stack, as part of the UP-Drive project.

ABSTRACTVisual–Linguistic large models (V‐LLMs) have demonstrated strong semantic understanding capabilities in understanding visual open‐world scenes. However, there is limited research on how to effectively integrate these models with traditional deep learning networks for classify and segment 3D point clouds. This paper explores the joint training mechanisms of V‐VLLMs and Convolutional Neural Networks (CNNs), constructing a pluggable module that enhances 3D object classification and component segmentation using 2D images, 3D point clouds, and language descriptions. The idea of this pluggable module is to employ contrastive learning between the generalized features extracted by large models from text and image features extracted by the CNN to enhance the robustness and understanding accuracy of the original CNN‐based point cloud network. For experiments, we validated the effectiveness and accuracy of the proposed pluggable module using multiple point cloud classification frameworks, based on datasets such as ModelNet40, ScanObjectNN, and ShapeNetCore. The experimental results demonstrate that the multimodal contrastive learning pluggable module significantly enhances performance across various deep learning frameworks, resulting in significant enhancements in classification and segmentation precision. Additionally, for the multiscale feature PointNet++ framework, the proposed multiscale contrastive learning mechanism provides further performance gains. The Code is available at https://github.com/winbigqian/MCL‐3D‐VCNN.

This paper proposes an effective method for recognition and classification of 3D objects using multiple views technique and neural networks system. In the processing stage, we propose to use 2D moment invariants as the features for modeling 3D objects. 2D moments have been commonly used for 2D object recognition. However, we have proved that with some adaptation to multiple views technique, 2D moments are sufficient to model 3D objects. In addition, the simplicity of 2D moments calculation reduces the processing time for feature extraction, hence increases the system efficiency. In the recognition stage, we propose a cascaded multilayered perceptron (c-MLP) network for matching and classification. The c-MLP contains two MLP networks which are arranged in a serial combination. This proposed method has been tested using two groups of object, polyhedral and free-form objects. We also compare our method with standard MLP network. Our results show that the proposed method can successfully be applied to 3D object recognition. In addition, the proposed network also achieved better performance and faster convergence rate compared to the than standard MLP

A cross-entropy method for the classification of 3D objects with curved surfaces is proposed. In the proposed approach, small cross-entropy between the shape histograms of two objects indicates that the two objects are similar, where the shape histogram is defined as the distribution of the shape indices computed at pixels on the object. Experiments with real range images show that the cross-entropy method can be used to classify 3D objects with curved surfaces.

With the rapid advancement of 3D acquisition technologies, 3D sensors such as LiDARs, 3D scanners, and RGB-D cameras have become increasingly accessible and cost-effective. These sensors generate 3D point cloud data that require efficient algorithms for tasks such as 3D model classification and segmentation. While deep learning techniques have proven effective in these areas, existing models often rely on complex architectures, leading to high computational costs that are impractical for real-time applications like augmented reality and robotics. In this work, we propose the Multi-level Graph Convolutional Neural Network (MLGCN), an ultra-efficient model for 3D point cloud analysis. The MLGCN model utilizes shallow Graph Neural Network (GNN) blocks to extract features at various spatial locality levels, leveraging precomputed KNN graphs shared across GCN blocks. This approach significantly reduces computational overhead and memory usage, making the model well-suited for deployment on low-memory and low-CPU devices. Despite its efficiency, MLGCN achieves competitive performance in object classification and part segmentation tasks, demonstrating results comparable to state-of-the-art models while requiring up to a thousand times fewer floating-point operations and significantly less storage. The contributions of this paper include the introduction of a lightweight, multi-branch graph-based network for 3D shape analysis, the demonstration of the model's efficiency in both computation and storage, and a thorough theoretical and experimental evaluation of the model's performance. We also conduct ablation studies to assess the impact of different branches within the model, providing valuable insights into the role of specific components.

Along with increasingly popular virtual reality applications, the three-dimensional (3D) point cloud has become a fundamental data structure to characterize 3D objects and surroundings. To process 3D point clouds efficiently, a suitable model for the underlying structure and outlier noises is always critical. In this work, we propose a hypergraph-based new point cloud model that is amenable to efficient analysis and processing. We introduce tensor-based methods to estimate hypergraph spectrum components and frequency coefficients of point clouds in both ideal and noisy settings. We establish an analytical connection between hypergraph frequencies and structural features. We further evaluate the efficacy of hypergraph spectrum estimation in two common applications of sampling and denoising of point clouds for which we provide specific hypergraph filter design and spectral properties. Experimental results demonstrate the strength of hypergraph signal processing as a tool in characterizing the underlying properties of 3D point clouds.

Future multimedia high-quality systems will be, among all, based on improving 3D visual experience. To raise 3D visual content quality and interactivity it is necessary to enable segmentation and classification of content which involves dividing the scene into meaningful sub-regions with the same attributes. Partitioning the image into grouping objects has various different applications in a wide variety of areas, since distinctive features in raw images may appear unclear to the human eyes. Segmentation can be defined as the identification of meaningful image components. It is a fundamental task in image processing providing the basis for any kind of further high-level image analysis. There are many different ways of segmenting the 3D image, all of which can be considered as a good segmentations, depending on objects of interest on an image, and to a large extent, the user’s own subjectivity. Key issues in this chapter include different techniques for segmentation of 3D object based on classification on different regions and shapes.KeywordsGrey LevelSegmentation MethodCorner DetectionGlobal ThresholdingGrey Level IntensityThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Multimodal learning for view-based 3D object classification

In order to try to improve the yield, knowing the expanding of a leaf by measuring its dimension can be one easy way to monitor the plant growth. While rice is considered to be one of the most important agricultural plant, their leaves are quite difficult to measure the size since they are mostly small and direct measurement could harm the plant. To do so, we deploy the non-destructive system to measure the dimension of the rice leaf from a point cloud. The proposed system takes only photos from a smartphone camera to produce the 3D model of a specific leaf using structure from motion and multi-view stereo. To measure the dimension of the rice leaf in metric units, the scale reference is placed into the scene before the photo taking process to involve it in the reconstruction result. From the reconstructed point cloud, the leaf veins can be extracted in order to find the accurate width of a leaf based on thresholding technique using combination of RGB and HSV spaces. By comparing with measured length of the leaf in point cloud with length of one block in scale reference, we can eventually obtain the distant between each leaf vein in the metric system unit.

Analyzing the geometric and semantic properties of 3D point clouds through the deep networks is still challenging due to the irregularity and sparsity of samplings of their geometric structures. This paper presents a new method to define and compute convolution directly on 3D point clouds by the proposed annular convolution. This new convolution operator can better capture the local neighborhood geometry of each point by specifying the (regular and dilated) ring-shaped structures and directions in the computation. It can adapt to the geometric variability and scalability at the signal processing level. We apply it to the developed hierarchical neural networks for object classification, part segmentation, and semantic segmentation in large-scale scenes. The extensive experiments and comparisons demonstrate that our approach outperforms the state-of-the-art methods on a variety of standard benchmark datasets (e.g., ModelNet10, ModelNet40, ShapeNet-part, S3DIS, and ScanNet).