Selective Embedding with Gated Fusion for 6D Object Pose Estimation

Bin-picking is a practical and challenging robotic manipulation task, where accurate 6D pose estimation plays a pivotal role. The workpieces in bin-picking are typically texture-less and randomly stacked in a bin, which poses a significant challenge to 6D pose estimation. Existing solutions are typically learning-based methods, which require object-specific training. Their efficiency of practical deployment for novel workpieces is highly limited by data collection and model retraining. Zero-shot 6D pose estimation is a potential approach to address the issue of deployment efficiency. Nevertheless, existing zero-shot 6D pose estimation methods are designed to leverage feature matching to establish point-to-point correspondences for pose estimation, which is less effective for workpieces with textureless appearances and ambiguous local regions. In this paper, we propose ZeroBP, a zero-shot pose estimation frame-work designed specifically for the bin-picking task. ZeroBP learns Position-Aware Correspondence (PAC) between the scene instance and its CAD model, leveraging both local features and global positions to resolve the mismatch issue caused by ambiguous regions with similar shapes and appearances. Extensive experiments on the ROBI dataset demonstrate that ZeroBP outperforms state-of-the-art zero-shot pose estimation methods, achieving an improvement of 9.1 % in average recall of correct poses.

In traditional 3D reconstruction methods, using a single view to predict the 3D structure of an object is a very difficult task. This research mainly discusses human pose recognition and estimation based on 3D multiview basketball sports dataset. The convolutional neural network framework used in this research is VGG11, and the basketball dataset Image Net is used for pretraining. This research uses some modules of the VGG11 network. For different feature fusion methods, different modules of the VGG11 network are used as the feature extraction network. In order to be efficient in computing and processing, the multilayer perceptron in the network model is implemented by a one‐dimensional convolutional network. The input is a randomly sampled point set, and after a layer of perceptron, it outputs a feature set of n × 16. Then, the feature set is sent to two network branches, one is to continue to use the perceptron method to generate the feature set of n × 1024, and the other network is used to extract the local features of points. After the RGB basketball sports picture passes through the semantic segmentation network, a picture containing the target object is obtained, and the picture is input to the constructed feature fusion network model. After feature extraction is performed on the RGB image and the depth image, respectively, the RGB feature, the local feature of the point cloud, and the global feature are spliced and fused to form a feature vector of N × 1152. There are three branches for this vector network, which, respectively, predict the object position, rotation, and confidence. Among them, the feature dimensionality reduction is realized by one‐dimensional convolution, and the activation function is the ReLU function. After removing the feature mapping module, the accuracy of VC‐CNN_v1 dropped by 0.33% and the accuracy of VC‐CNN_v2 dropped by 0.55%. It can be seen from the research results that the addition of the feature mapping module improves the recognition effect of the network to a certain extent

It is widely shown that Convolutional Neural Networks (CNNs) are vulnerable to adversarial examples on most recognition tasks, such as image classification and segmentation. However, few work studies the more complicated task – 3D human pose estimation. This task often requires large-scale datasets, specialized network architectures, and it can be solved either from single-view RGB images or from multi-view RGB images. In this paper, we make the first attempt to investigate the robustness of current state-of-the-art 3D human pose estimation methods. To this end, we build four representative baseline models, where most of the current methods can be generally classified as one of them. Furthermore, we design targeted adversarial attacks to detect whether 3D pose estimators are robust to different camera parameters. For different types of methods, we present a comprehensive study of their robustness on the large-scale Human3.6M benchmark. Our work shows that different methods vary significantly in their resistance to adversarial attacks. Through extensive experiments, we show that multi-view 3D pose estimators can be more vulnerable to adversarial examples. We believe that our efforts can shed light on future works to design more robust 3D human pose estimators.

Template matching has been shown to accurately estimate the pose of a new object given a limited number of samples. However, pose estimation of occluded objects is still challenging. Furthermore, many robot application domains encounter texture-less objects for which depth images are more suitable than color images. In this paper, we propose a novel framework, Multi-Task Template Matching (MTTM), that finds the nearest template of a target object from a depth image while predicting segmentation masks and a pose transformation between the template and a detected object in the scene using the same feature map of the object region. The proposed feature comparison network computes segmentation masks and pose predictions by comparing feature maps of templates and cropped features of a scene. The segmentation result from this network improves the robustness of the pose estimation by excluding points that do not belong to the object. Experimental results show that MTTM outperforms baseline methods for segmentation and pose estimation of occluded objects despite using only depth images.

Real-time pose estimation of rigid objects in heavily cluttered environments

An unsupervised point cloud object retrieval and pose estimation method, called PCRP, is proposed in this work. It is assumed that there exists a gallery point cloud set that contains point cloud objects with given pose orientation information. PCRP attempts to register the unknown point cloud object with those in the gallery set so as to achieve content-based object retrieval and pose estimation jointly, where the point cloud registration task is built upon an enhanced version of the unsupervised R-PointHop method. Experiments on the ModelNet40 dataset demonstrate the superior performance of PCRP in comparison with traditional and learning based methods.

Real-world traffic is highly dynamic, with pedestrians exhibiting unpredictable movements. Pedestrians' poses are essential cues for predicting their actions, enabling vehicles to respond proactively and reduce accident risks. In autonomous driving, the distance between vehicles and pedestrians is critical, making 3D human pose estimation crucial. In this context, pedestrian pose estimation has been actively studied, and recently, light detection and ranging (LiDAR) sensors have attracted attention due to their accurate 3D depth information and privacy benefits. However, existing LiDAR-based 3D pose estimation methods mainly process 3D data directly, requiring high computational cost and memory. In this paper, we propose a lightweight LiDAR-based 3D human pose estimation method specifically designed for deployment in autonomous driving systems. Unlike conventional 3D direct processing methods, our approach strategically reduces computational complexity by projecting point clouds into 2D depth images and leveraging a lightweight MoveNet, followed by efficient 3D lifting. Furthermore, we introduce a self-occlusion correction algorithm to improve robustness under side-view and bending poses, where depth-based projections often suffer from distortion. Experimental results on benchmark datasets demonstrate that the proposed method achieves competitive pose estimation accuracy while substantially improving efficiency, highlighting its practicality and scalability for real-time autonomous vehicle applications.

Light detection and ranging (LiDAR) sensors provide accurate 3-D point clouds for noncooperative spacecraft pose estimation. Several robust methods, such as iterative closest point, exist to perform a local refinement of the pose starting from an initial estimate. However, finding the initial pose of the spacecraft is a global optimization problem, which is challenging to solve in real time. This is especially true on space hardware with limited computing power. In addition, many spacecrafts have a shape with multiple symmetries, making an unambiguous initial pose estimation impossible. This work introduces a convolutional-neural-network-based pose estimation method, accounting for potential symmetries of the target satellite. The point clouds are projected to a 2-D depth image before being processed by the network. To generate a sufficient amount of training data, a LiDAR simulator integrating multiple effects such as reflections or laser beam divergence is developed. While being trained solely on synthetic point clouds, the pose estimation method shows to be precise, efficient, and reliable when evaluated on real point clouds taken at a hardware-in-the-loop rendezvous test facility. A runtime evaluation on potential space computing hardware is also performed to demonstrate the applicability of the method to real-time onboard pose estimation.

The 6D pose estimation of objects plays a crucial role in robot sorting technology. To simultaneously improve the accuracy and efficiency of pose estimation during the grasping process, this study proposes an innovative deep learning pose estimation method based on PVN3D. Specifically, this study deeply optimized the backbone network of the image extraction model, using dense connections and learnable grouped convolutions to improve model accuracy and reduce model complexity. At the same time, the core idea of convolutional neural networks is integrated into the point cloud feature extraction process, which uses dynamic convolution techniques to process point cloud features, making the processing of point cloud data more efficient and flexible. In addition, by introducing a parameter-free attention mechanism, the accuracy of the model has been further improved. Through extensive experimental verification on the publicly available LineMOD, YCB video datasets and other three core datasets under the BOP benchmark, the method proposed in this paper demonstrates significant advantages in both computational efficiency and estimation accuracy. It not only significantly improves the accuracy of pose estimation but also significantly enhances real-time performance.

Human Pose Estimation (HPE) is defined as the problem of human joints’ localization (also known as keypoints: elbows, wrists, etc.) in images or videos. It is also defined as the search for a specific pose in space of all articulated joints. HPE has recently received significant attention from the scientific community. The main reason behind this trend is that pose estimation is considered as a key step for many computer vision tasks. Although many approaches have reported promising results, this domain remains largely unsolved due to several challenges such as occlusions, small and barely visible joints, and variations in clothing and lighting. In the last few years, the power of deep neural networks has been demonstrated in a wide variety of computer vision problems and especially the HPE task. In this context, we present in this paper a Deep Full-Body-HPE (DFB-HPE) approach from RGB images only. Based on ConvNets, fifteen human joint positions are predicted and can be further exploited for a large range of applications such as gesture recognition, sports performance analysis, or human-robot interaction. To evaluate the proposed deep pose estimation model, we apply it to recognize the daily activities of a person in an unconstrained environment. Therefore, the extracted features, represented by deep estimated poses, are fed to an SVM classifier. To validate the proposed architecture, our approach is tested on two publicly available benchmarks for pose estimation and activity recognition, namely the J-HMDBand CAD-60datasets. The obtained results demonstrate the efficiency of the proposed method based on ConvNets and SVM and prove how deep pose estimation can improve the recognition accuracy. By means of comparison with state-of-the-art methods, we achieve the best HPE performance, as well as the best activity recognition precision on the CAD-60 dataset.

A pose (relative attitude and position) estimation approach to support proximity operations such as autonomous landing of unmanned air vehicles or spacecraft on-orbit servicing is investigated in this paper. An algorithm is developed based on a feature detection capability using the point cloud data that a flash lidar (light detection and ranging) produces. The feature refers to an edge, a corner, or a distinct geometrical shape of an object. Once features are detected, orthogonal triads that are attached to the object are constructed using them. Pose estimation is then performed based on the orientation of the triads with respect to the lidar sensor frame. Since the algorithm employs detected features, it allows pose estimation of an object from a single set of point cloud data. In contrast, the conventional iterative closest point based algorithms typically require two sets of point cloud data. The estimated pose thus represents the changes in pose between the two data sets, leaving the pose of the object with respect to the sensor still to be determined. This featurebased pose estimation capability performs directly an initial pose acquisition with respect to the sensor. The algorithm may also help reduce the number of data points required for pose estimation since a small, selected feature of an object within the sensor field of view can be employed to estimate the pose. Employing the algorithm developed, pose estimation tests were conducted and their preliminary results are presented, which were obtained using a flash lidar in a laboratory testing environment.

6D pose estimation is an important branch in the field of vision measurement, and is widely used in the fields of robotics, autonomous driving and reality augmentation. The latest research trend in 6D pose estimation is to train a deep neural network to directly predict the 2D projection position of the 3D keypoint from the image, establish the corresponding relationship, and finally use the perspective-n-point (PnP) algorithm to perform pose estimation. The current challenge of pose estimation is that when objects are textureless, occluded or scene-clutterd, the detection accuracy is reduced, and most of the existing algorithm models are large and cannot accommodate real-time requirements. In this paper, we introduce a densely connected feature pyramid network (DFPN) that can efficiently integrate and utilize features. We combine the cross-stage partial network (CSPNet) with DFPN to design a new network for 6D pose estimation, DFPN-6D, a new approach for 6D object pose estimation. DFPN-6D can efficiently and accurately handle objects with textureless, occluded and scene clutter and estimate their full 6D poses in a single shot. Furthermore, we propose a new confidence calculation method and loss function for object pose estimation, which can fully consider spatial information. Finally, we propose a novel augmentation method for direct 6D pose estimation approaches to improve performance and generalization ability in the case of occlusion, which is called 6D augmentation. Our approach achieves a new state-of-the-art accuracy of 98.06 and 87.09 in terms of the ADD(-S) metric on the Linemod dataset and Occluded-Linemod dataset, and our method also achieves the best result in terms of the different metric on the MULT-I dataset, BIN-P dataset and T-LESS dataset, respectively, while still running end-to-end at over 65 FPS. The experimental results demonstrate that our algorithm is robust to textureless materials and occlusion while running more efficiently than other methods. We also deploy our proposed method to a real robot to grasp and manipulate objects based on the estimated pose.

With the advancement of intelligent manufacturing, robots are increasingly involved in industrial production, for example, replacing human beings to identify and manipulate objects. However, it is still a challenging task for object recognition and pose estimation in industrial environment, especially under large environmental noise, mutual occlusions and high-speed requirements. In this paper, we propose a new method for 6D object pose estimation, Super Key 4-Points Congruent Sets (SK-4PCS), which is a fast global method combined with local features. The proposed method includes three steps. Firstly, the key points in 3D point clouds are extracted by local features, which are invariant with respect to rotation and translation. Secondly, the smart indexing scheme is used to realize the query of 4-points congruent sets globally. Finally, the coarse pose obtained quickly is further refined by the Iteration Closest Point algorithm (ICP) [1]. The proposed method is also compared with existing methods by the evaluation indicator Visible Surface Discrepancy (VSD) on the LINEMOD dataset [2]. The experimental results show that the refined SK-4PCS results in significant speedup and satisfactory accuracy in 6D object pose estimation.

With the advantages of wide range, non-contact and high flexibility, the visual estimation technology of target pose has been widely applied in modern industry, robot guidance and other engineering practices. However, due to the influence of complicated industrial environment, outside interference factors, lack of object characteristics, restrictions of camera and other limitations, the visual estimation technology of target pose is still faced with many challenges. Focusing on the above problems, a pose estimation method of the industrial objects is developed based on 3D models of targets. By matching the extracted shape characteristics of objects with the priori 3D model database of targets, the method realizes the recognition of target. Thus a pose estimation of objects can be determined based on the monocular vision measuring model. The experimental results show that this method can be implemented to estimate the position of rigid objects based on poor images information, and provides guiding basis for the operation of the industrial robot.

Point cloud registration has a wide range of applications in 3D reconstruction, pose estimation, intelligent driving, heritage conservation, and digital cities. The traditional iterative closest point (ICP) algorithm has strong dependence on the initial position, poor robustness, and low timeliness. To address the above issues, a fast point cloud registration method that incorporates RGB image information is proposed. The SIFT algorithm is used to detect feature points of point clouds corresponding to the RGB image, followed by feature point matching. The RANSAC algorithm is applied to remove erroneous point pairs in order to calculate the initial transformation matrix. After applying a pass-through filter for noise reduction and transiting down with a voxel grid, the point cloud is subjected to rotation and translation transformation for initial registration. On the basis of initial alignment, the FR-ICP algorithm is utilized for achieving precise registration. This method not only avoids the problem of ICP easily getting stuck in local optima, but also has higher registration accuracy and efficiency. Experimental studies were conducted based on point clouds of automotive parts collected in real scenes, and the results showed that the proposed method has a registration error of only 0.487 mm. Among the same group of experimental point clouds with comparable registration error, the proposed method showed a speed improvement of 69%/48% compared to ICP/FR-ICP with regard to registration speed.