3D Object Classification and Segmentation Methods

This paper proposes a method for recognition and classification of 3D objects using 2D moments and HMLP network. The 2D moments are calculated based on 2D intensity images taken from multiple cameras that have been arranged using multiple views technique. 2D moments are commonly used for 2D pattern recognition. However, the current study proves that with some adaptation to multiple views technique, 2D moments are sufficient to model 3D objects. In addition, the simplicity of 2D moment's calculation reduces the processing time for feature extraction, thus decreases the recognition time. The 2D moments were then fed into a neural network for classification of the 3D objects. In the current study, hybrid multi-layered perceptron (HMLP) network is proposed to perform the classification. Two distinct groups of objects that are polyhedral and free-form objects were used to access the performance of the proposed method. The recognition results show that the proposed method has successfully classified the 3D object with the accuracy of up to 100%.

We propose a framework for three-dimensional (3D) object recognition and classification in very low illumination environments using convolutional neural networks (CNNs). 3D images are reconstructed using 3D integral imaging (InIm) with conventional visible spectrum image sensors. After imaging the low light scene using 3D InIm, the 3D reconstructed image has a higher signal-to-noise ratio than a single 2D image, which is a result of 3D InIm being optimal in the maximum likelihood sense for read-noise dominant images. Once 3D reconstruction has been performed, the 3D image is denoised and regions of interest are extracted to detect 3D objects in a scene. The extracted regions are then inputted into a CNN, which was trained under low illumination conditions using 3D InIm reconstructed images, to perform object recognition. To the best of our knowledge, this is the first report of utilizing 3D InIm and convolutional neural networks for 3D training and 3D object classification under very low illumination conditions.

This paper proposes an effective method for recognition and classification of 3D objects using multiple views technique and neural networks system. In the processing stage, we propose to use 2D moment invariants as the features for modeling 3D objects. 2D moments have been commonly used for 2D object recognition. However, we have proved that with some adaptation to multiple views technique, 2D moments are sufficient to model 3D objects. In addition, the simplicity of 2D moments calculation reduces the processing time for feature extraction, hence increases the system efficiency. In the recognition stage, we propose a cascaded multilayered perceptron (c-MLP) network for matching and classification. The c-MLP contains two MLP networks which are arranged in a serial combination. This proposed method has been tested using two groups of object, polyhedral and free-form objects. We also compare our method with standard MLP network. Our results show that the proposed method can successfully be applied to 3D object recognition. In addition, the proposed network also achieved better performance and faster convergence rate compared to the than standard MLP

Autonomous vehicles are increasingly becoming a part of urban life.However, significant challenges still need to be overcome to ensure these vehicles are both safe and efficient.One of the main challenges lies in perception, particularly in 3D object segmentation and detection, which relies on multiple sensors to enhance accuracy and operate under adverse conditions.The efficient fusion of these sensors is a crucial factor, as it directly impacts detection quality and computational cost.Modern multi-sensor fusion methods leverage Deep Learning techniques, and an emerging approach in this field is the use of attention mechanisms.These mechanisms enable more informative representations of feature maps extracted from sensors, highlighting the most relevant information while suppressing less significant data.In this context, this study explores the use of attention mechanisms to optimise the BEVFusion model, which has achieved state-of-the-art performance by employing a unified Camera-LiDAR fusion in a Bird's Eye View (BEV) representation.The key advantage of BEVFusion is its highly efficient transformation of camera-perspective views into the BEV representation.However, its sensor fusion approach is limited to a simple concatenation of extracted features, which may not be the most efficient solution.Another critical aspect of BEVFusion is its high computational cost, as it relies on deep learning models that demand powerful hardware, posing a challenge for deployment in autonomous vehicles.This is particularly relevant given the need for embedded hardware in such vehicles to be both cost-effective and energy-efficient.To address these issues, this study investigates and implements attention mechanisms to enhance the sensor fusion process in BEVFusion for 3D object detection and segmentation, while also improving computational efficiency.Modifications were made to reduce VRAM consumption and processing time, ensuring performance comparable to the original model but with lower computational demands.The results are promising, showing a 14.12% increase in IoU for the segmentation task and a 0.732% improvement in mAP for 3D object detection.Additionally, training time was reduced by a factor of 3.3, and VRAM consumption was nearly halved.

In this paper we propose a novel framework for 3D object categorization. The object is modeled it in terms of its sub-parts as an histogram of 3D visual word occurrences. We introduce an effective method for hierarchical 3D object segmentation driven by the minima rule that combines spectral clustering – for the selection of seed-regions – with region growing based on fast marching. The front propagation is driven by local geometry features, namely the Shape Index. Finally, after the coding of each object according to the Bag-of-Words paradigm, a Support Vector Machine is learnt to classify different objects categories. Several examples on two different datasets are shown which evidence the effectiveness of the proposed framework.

3D Point clouds are irregular and unordered. Recently, neural networks applying on point clouds have shown superior performance on 3D object classification and segmentation. PointNet has achieved competitive performance in classification tasks, however, it ignores the important local information of point clouds, which carries the important information in 3D geometric space. The informations lack will reduce the accuracy of object classification. In this paper, a new deep neural network model is proposed. Specifically, a multiscale network is used to extract local features of points to improve the accuracy of PointNet and a deep residual network is used to avoid the vanishing gradient problem in the process of deep training network. Experiments show that the proposed network architecture achieves better performance than state-of-the-art methods on multiple challenging benchmark datasets and tasks.

In this paper we propose to address the problem of 3D object categorization. We model the 3D object as a 2D Riemannian manifold and propose metric tensor and Christoffel symbols as a novel set of features. The proposed set of features capture the local and global geometry of 3D objects by exploiting the positional dependence of the features. The categorization of 3D objects is carried out using polynomial kernel SVM classifier. The effectiveness of the proposed framework is demonstrated on 3D objects obtained from different datasets and achieve comparable results.

Object of Interest (OoI) segmentation in video sequences is, due to its temporal and spatial complexity, still a difficult task. Thus, it cannot be done automatically, and interactive segmentation methods are inconvenient and time-consuming. Overcoming these drawbacks and performing OoI segmentation in real-time, this paper introduces a new approach for gaze-based OoI segmentation. The user fixates on the OoI throughout watching a video sequence. The users’ gaze provides relevant information on the location of the OoI, which is then used for segmenting the OoI. In comparison to present segmentation methods, our approach notably reduced user-interactions required to attain pixel-accurate segmentations. The first evaluation of our gaze-based OoI segmentation method shows promising results. These are close to, and sometimes even exceeding the quality of the polygon-based ground truth segmentation. In conclusion, these findings encourage further exploration of gaze-based OoI segmentation.

X-ray imaging technology has been used for decades in clinical tasks to reveal the internal condition of different organs, and in recent years, it has become more common in other areas such as industry, security, and geography. The recent development of computer vision and machine learning techniques has also made it easier to automatically process X-ray images and several machine learning-based object (anomaly) detection, classification, and segmentation methods have been recently employed in X-ray image analysis. Due to the high potential of deep learning in related image processing applications, it has been used in most of the studies. This survey reviews the recent research on using computer vision and machine learning for X-ray analysis in industrial production and security applications and covers the applications, techniques, evaluation metrics, datasets, and performance comparison of those techniques on publicly available datasets. We also highlight some drawbacks in the published research and give recommendations for future research in computer vision-based X-ray analysis.

Nowadays, the number of aging high-rise civil structures is growing throughout the world, and most of them use concrete as a building material. Concrete may lose its strength due to continuous loading and environmental impacts. As a result, damage (crack and spall) may occur on the exterior surface of the structure. Whenever these deformities are left unexplored and untreated, the integrity of the structure may be compromised. Therefore, regular maintenance of the structure is needed. Some previous vision-based studies have used a drone as a vehicle to capture and record the current state of the structure. Later, captured videos and images were analyzed to determine damage using object classification, localization, and segmentation methods. Also, in a few studies, drones relay the collected data to the server using a wireless medium. However, the developed systems are very complex, time-intensive, and require high bandwidth. To address these problems, this paper uses the edge computing principle to propose a real-time multi-drone damage detection system using one of the advance deep learning models called You Look Only Once-version3 (YOLO-v3) for high-rise civil structures. The proposed system uses Jetson-TX2 as a hardware platform to run YOLO-v3 and is deployed on Pixhawk’s hardware standards-based open source hexacopter drone. During a coordinated survey, if the damage is observed, the Jetson-TX2 on-board, after processing, sends only the damage-related information to the server that resides on the ground through the Wi-Fi channel. Our proposed system is evaluated on a dataset containing 800 ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$480\times480$ </tex-math></inline-formula> pixels) images of different types of damage, collected from multiple structures of CSIR-CEERI, Pilani. Manually annotated images are used for the training and testing of the modified YOLO-v3 classifier. As a result, the proposed approach offers reliable performance with an accuracy of 94.24% and can process an image in 0.033 seconds.

Nowadays, point clouds are frequently gathered by 3D scanners such as Lidar and Kinect, which produces thousands of point cloud models. Point cloud processing is vital to 3D vision, especially 3D object recognition, positioning, and navigation technology. Addressing uneven data density caused by coordinate frame transformations and the inherent problem of insufficient context connection in point clouds, the DiM-PCNet (Multi-scale and Multi-level Point Clouds Classification Network, and the Di is a prefix to represent double M in Multi-scale and Multi-level) is proposed in this paper. DiM-PCNet is provided for object classification with multi-scale and multi-level features. We encode the point cloud in multi-scale and fully fuse the features with the raw point cloud for keeping the context relationship. In DiM-PCNet, we sample the point clouds from eight parts for multi-scales feature extraction. The multi-scales features are fully fused by multi-level pyramid models. The multi-scale and multi-level strategies are applied in DiM-PCNet, in which the abundant and important features of point clouds are extracted and utilized in the 3D object classification. It is worth noting that the DiM-PCNet feature block can be embedded into the segmentation net, where the accuracy achieved is 87.1%. We conducted experiments on ShapeNet and ModelNet40 and the experimental results show that DiM-PCNet achieves state-of-the-art performance in 3D object classification. The experiment shows competitive performance on robustness and segmentation tasks.

3D object detection and isolation can be achieved algorithmically using computational integral-imaging data. The 3D scene is acquired by a multi-channel system, where each channel (elemental image) captures the scene from a shifted perspective angle. The number of these channels affects the weight, the cost, and the computational load of the segmentation process, while a lower number of channels may reduce the performance of the objects' separation in the 3D scene. This research examines the effect of the elemental images' quantity on the 3D object detection and segmentation, under both regular and noisy conditions. Moreover, based on our previous works, we perform an improvement of the 3D object segmentation quality using an adapted active-contour method.

Local features with invariant descriptions are important for many tasks in image processing and computer vision. This paper presents a new local feature descriptor for 3D object and scene representation. The new descriptor, named 3DSSIM, explores the internal geometric property of layout similarity of 3D objects to produce efficient feature representation. The 3D-SSIM is highly distinctive, quick to compute, and shows superior advantages in terms of robustness to noises, invariance to viewpoints, and tolerance to geometric distortions. We extensively evaluated performance of the new descriptor with various datasets.

A vision system is an integral component of many autonomous robots. It enables the robot to perform essential tasks such as mapping, localization, or path planning. A vision system also assists with guiding the robot's grasping and manipulation tasks. As an increased demand is placed on service robots to operate in uncontrolled environments, advanced vision systems must be created that can function effectively in visually complex and cluttered settings. This thesis presents the development of segmentation algorithms to assist in online model acquisition for guiding robotic manipulation tasks. Specifically, the focus is placed on localizing door handles to assist in robotic door opening, and on acquiring partial object models to guide robotic grasping. . First, a method for localizing a door handle of unknown geometry based on a proposed 3D segmentation method is presented. Following segmentation, localization is performed by fitting a simple box model to the segmented handle. The proposed method functions without requiring assumptions about the appearance of the handle or the door, and without a geometric model of the handle. Next, an object segmentation algorithm is developed, which combines multiple appearance (intensity and texture) and geometric (depth and curvature) cues. The algorithm is able to segment objects without utilizing any a priori appearance or geometric information in visually complex and cluttered environments. The segmentation method is based on the Conditional Random Fields (CRF) framework, and the graph cuts energy minimization technique. A simple and efficient method for initializing the proposed algorithm which overcomes graph cuts' reliance on user interaction is also developed. Finally, an improved segmentation algorithm is developed which incorporates a distance metric learning (DML) step as a means of weighing various appearance and geometric segmentation cues, allowing the method to better adapt to the available data. The improved method also models the distribution of 3D points in space as a distribution of algebraic distances from an ellipsoid fitted to the object, improving the method's ability to predict which points are likely to belong to the object or the background. Experimental validation of all methods is performed. Each method is evaluated in a realistic setting, utilizing scenarios of various complexities. Experimental results have demonstrated the effectiveness of the handle localization method, and the object segmentation methods.

This paper aims to develop an automatic 3D object segmentation method for the large-scale point clouds. Given a range image, the preprocessing is first applied to get the optimal 3D point cloud. A [Formula: see text]-nearest neighbor is built, and a segmentation algorithm based on the conditional angular clustering technique is used to segment the objects from the point cloud. The algorithm is tested on the real point cloud datasets. The experiment results demonstrated that the developed segmentation method can be used to localize the object with the relative uncertainty of 0.27%.