Enhanced RGB-D feature extraction for 6D pose estimation

Six degrees-of-freedom (6D) object pose estimation plays an important role in pattern recognition of fields such as robotics and augmented reality. However, there are issues with low accuracy and real-time performance of 6D object pose estimation in complex scenes. To address these challenges, in this article, RFF-PoseNet (a 6D object pose estimation network based on robust feature fusion) is proposed for complex scenes. Firstly, a more lightweight Ghost module is used to replace the convolutional blocks in the feature extraction network. Then, a pyramid pooling module is added to the semantic label branch of PoseCNN to fuse the features of different pooling layers and enhance the network’s ability to capture information about objects in complex scenes and the correlations between contextual information. Finally, a pose regression and optimization module is utilized to further improve object pose estimation in complex scenes. Simulation experiments conducted on the YCB-Video and Occlusion LineMOD datasets show that the RFF-PoseNet algorithm can strengthen the correlation of features between different levels and the recognition ability of unclear targets, thereby achieving excellent accuracy and real-time performance, as well as strong robustness.

Currently, many methods of object pose estimation use images or point clouds alone for pose estimation. This leads to their inability to accurately estimate the object pose in the case of occlusion and poor illumination. Second, these models have large parameters and cannot be deployed on mobile devices. Therefore, we propose a lightweight two-terminal feature fusion network, which can effectively use images and point clouds for accurate object pose estimation. First, Pointno problemNet network is used to extract point cloud features. Then the extracted point cloud features are combined with the images at pixel level and the features are extracted by CNN. Secondly, the extracted image features are combined with the point cloud point by point. Then feature extraction is performed on it using the improved PointNet++ network. Finally, a set of center point features are obtained and pose estimation is performed for each feature. The pose with the highest confidence is selected as the final result. Furthermore, we apply depthwise separable convolutions to reduce the amount of model parameters. Experiments show that the proposed method exhibits better performance on Linemod and Occlusion Linemod datasets. Furthermore, the model parameters are small, and it is robust in occlusion and low-light situations.

With the advancement of intelligent manufacturing, robots are increasingly involved in industrial production, for example, replacing human beings to identify and manipulate objects. However, it is still a challenging task for object recognition and pose estimation in industrial environment, especially under large environmental noise, mutual occlusions and high-speed requirements. In this paper, we propose a new method for 6D object pose estimation, Super Key 4-Points Congruent Sets (SK-4PCS), which is a fast global method combined with local features. The proposed method includes three steps. Firstly, the key points in 3D point clouds are extracted by local features, which are invariant with respect to rotation and translation. Secondly, the smart indexing scheme is used to realize the query of 4-points congruent sets globally. Finally, the coarse pose obtained quickly is further refined by the Iteration Closest Point algorithm (ICP) [1]. The proposed method is also compared with existing methods by the evaluation indicator Visible Surface Discrepancy (VSD) on the LINEMOD dataset [2]. The experimental results show that the refined SK-4PCS results in significant speedup and satisfactory accuracy in 6D object pose estimation.

The next generation of service robots is to offer services relying heavily on visually guided manipulation, besides navigation, such services as errand, logistics, appliance, home keeping, etc. For the successful introduction of these services, it is critical to establish the consumer-level dependability in 3D object recognition and pose estimation in a natural setting where a large variation of environment, e.g., perspective, texture, form factor, illumination, occlusion, etc., is common. To address this problem, we propose an approach of the two-layered particle filter to the dependability in 3D object recognition and pose estimation. In the upper layer, a set of object pose candidates is identified and maintained in the search space as a set of super-particles, each of which is assigned a probability of the true pose and evolved in time along with the accumulation of further evidences. To define the object pose candidates, first, we acquire initially weak evidences quickly and interpret them in terms of possible object poses in space. These interpretations serve as the region of interest for detailed investigation by which the pose probabilities are computed for individual interpretations based on the likelihood and unlikelihood of various features available in the corresponding regions of interest. During the process of probability computation, we select the object pose candidates to be used as super-particles in the upper layer. In the lower level, the pose uncertainties associated with the individual candidates are represented as particles that are subject to the propagation in time. Finally, the experimental results support the strength of the proposed approach in the real environment in terms of its dependability in 3D object recognition and pose estimation.

In this paper, a novel global point cloud descriptor is proposed for reliable object recognition and pose estimation, which can be effectively applied to robot grasping operation. The viewpoint feature histogram (VFH) is widely used in three-dimensional (3D) object recognition and pose estimation in real scene obtained by depth sensor because of its recognition performance and computational efficiency. However, when the object has a mirrored structure, it is often difficult to distinguish the mirrored poses relative to the viewpoint using VFH. In order to solve this difficulty, this study presents an improved feature descriptor named orthogonal viewpoint feature histogram (OVFH), which contains two components: a surface shape component and an improved viewpoint direction component. The improved viewpoint component is calculated by the orthogonal vector of the viewpoint direction, which is obtained based on the reference frame estimated for the entire point cloud. The evaluation of OVFH using a publicly available data set indicates that it enhances the ability to distinguish between mirrored poses while ensuring object recognition performance. The proposed method uses OVFH to recognize and register objects in the database and obtains precise poses by using the iterative closest point (ICP) algorithm. The experimental results show that the proposed approach can be effectively applied to guide the robot to grasp objects with mirrored poses.

6D object pose estimation from a single RGB-D image is a fundamental problem in computer vision and robot manipulation. Despite recent advancements, existing methods still suffer several limitations. First of all, the object shape representation extracted from the depth map is often less expressive because the object point cloud parsed from the depth map is highly incomplete due to the object self-occlusion and noisy due to the sensor artifacts. This shape representation issue further intensifies when lacking sufficient labeled data for model training, which unfortunately is another typical problem for object pose estimation considering the heavy annotation cost for real-world pose labeling. In this study, we propose to tackle the above issues in a unified way. First, we enhance the object shape representation from the partial point cloud with a novel canonical shape reconstruction module, in which an implicit canonical frame is established by incorporating the SE(3) equivariance, achieving implicit feature alignment of the partial point cloud inputs, leading to robust shape recovery. Second, based on the enhanced object representation, we further utilize the de-canonicalized and pose-dependent completed object shape as the training signal, and develop a novel weakly-supervised learning framework to leverage both labeled synthetic data and unlabeled real data to train the pose estimation model in a label-efficient way. Extensive experiments on three widely used benchmarks demonstrate the effectiveness, and superiority of our framework over state-of-the-art methods.

Joint object recognition and pose estimation using multiple-anchor triplet learning of canonical plane

With the continuous development of computer technology and deep learning, object detection technology has received great attention in recent years. Especially on the problem of 3D object detection in AR field, a variety of novel algorithms are in bloom. However, these algorithms are either low precision or slow speed. Therefore, we propose an object detection and pose estimation method for AR Application, which is a model for 6D pose estimation of 3D object based on 2D object detection. In our model, we firstly input the 2D projection image of the 3D model into the improved RetinaNet model, and detect the coordinates of the vertex and center point of the object on the image. Then the coordinates on the image and the coordinates on the CAD are matched by point matching. Then, the EPnP improved by Perspective-n-Point (PnP) algorithm is used to complete the pose estimation of 3D object. Finally, we test the performance of our algorithm. The results show that our accuracy has reached the average level of the mainstream algorithms.KeywordsPose estimationObject detectionDeep learningConvolutional neural network

6D object pose estimation is a longstanding computer vision problem. Existing deep learning-based methods have achieved inspiring results in this task. However, large-scale training data with annotations is extremely needed to guarantee these methods’ performance, and acquiring real 6D object pose annotations is fairly labor-intensive and time-consuming. To overcome this drawback, we propose a semi-supervised pose estimation method using labeled synthetic data and unlabeled real data. For unlabeled real data, we form a self-supervised pipeline by minimizing the distance between the input point cloud, which is under ground-truth pose, and the model points transformed based on predicted pose. The labeled synthetic data is used to supervise the network to converge correctly. And we utilize a feature mapping to eliminate the domain gap between the real and synthetic features to further enhance the network’s performance. Moreover, we propose an attention-based pose estimation network, which can concentrate more on the distinguishing features, thus improving the accuracy of pose estimation. Experiments show that our proposed semi-supervised method is able to achieve good performance without the real annotations and outperforms all other methods relying on synthetic data or self-supervision strategy, indicating that the proposed method is effective.

SkeletonPose: Exploiting human skeleton constraint for 3D human pose estimation

Pose estimation is a particularly important link in the task of robotic bin-picking. Its purpose is to obtain the 6D pose (3D position and 3D posture) of the target object. In real bin-picking scenarios, noise, overlap, and occlusion affect accuracy of pose estimation and lead to failure in robot grasping. In this paper, a new point-pair feature (PPF) descriptor is proposed, in which curvature information of point-pairs is introduced to strengthen feature description, and improves the point cloud matching rate. The proposed method also introduces an effective point cloud preprocessing, which extracts candidate targets in complex scenarios, and, thus, improves the overall computational efficiency. By combining with the curvature distribution, a weighted voting scheme is presented to further improve the accuracy of pose estimation. The experimental results performed on public data set and real scenarios show that the accuracy of the proposed method is much higher than that of the existing PPF method, and it is more efficient than the PPF method. The proposed method can be used for robotic bin-picking in real industrial scenarios.

3D object pose estimation by using a depth sensor is one of the important tasks in activities by robots. To reduce the pose ambiguity of an estimated object pose, several methods for multiple viewpoint pose estimation have been proposed. However, these methods need to select the viewpoints carefully to obtain better results. If the pose of the target object is ambiguous from the current observation, we could not decide where we should move the sensor to set as the next viewpoint. In this paper, we propose a best next viewpoint recommendation method by minimizing the pose ambiguity of the object by making use of the current pose estimation result as a latent variable. We evaluated viewpoints recommended by the proposed method and confirmed that it helps us to gain better pose estimation results than several comparative methods on a synthetic dataset.

We observe that human poses exhibit strong group-wise structural correlation and spatial coupling between keypoints due to the biological constraints of different body parts. This group-wise structural correlation can be explored to improve the accuracy and robustness of human pose estimation. In this work, we develop a self-constrained prediction-verification network to characterize and learn the structural correlation between keypoints during training. During the inference stage, the feedback information from the verification network allows us to perform further optimization of pose prediction, which significantly improves the performance of human pose estimation. Specifically, we partition the keypoints into groups according to the biological structure of human body. Within each group, the keypoints are further partitioned into two subsets, high-accuracy proximal keypoints and low-accuracy distal keypoints. We develop a self-constrained prediction-verification network to perform forward and backward predictions between these keypoint subsets. One fundamental challenge in pose estimation, as well as in generic prediction tasks, is that there is no mechanism for us to verify if the obtained pose estimation or prediction results are accurate or not, since the ground truth is not available. Once successfully learned, the verification network serves as an accuracy verification module for the forward pose prediction. During the inference stage, it can be used to guide the local optimization of the pose estimation results of low-accuracy keypoints with the self-constrained loss on high-accuracy keypoints as the objective function. Our extensive experimental results on benchmark MS COCO and CrowdPose datasets demonstrate that the proposed method can significantly improve the pose estimation results.KeywordsHuman pose estimationSelf-constrainedStructural inferencePrediction optimization

There is typically a need for physical training and assessment of the physical exercise poses of students in schools or members of fitness centers. However, few studies have been conducted on physical exercise poses, and no relevant dataset is available. We introduce a new dataset built for the particular physical fitness action of push-ups called the push-up dataset. We propose a new real-time and device-friendly pose estimation model named PUPose. The trade-off between accuracy and efficiency in human pose estimation has been challenging. In this paper, some recent studies have extended object detectors to unify person detection and pose estimation. Our PUPose combines top-down and bottom-up approaches to get bounding boxes for multiple individuals and their corresponding poses. We investigate the applicability of methods for extending object detectors to estimate human pose to lightweight pose estimation models. We strengthen the backbone structure and design an efficient neck structure, improving the feature extraction capability of the network. We optimize the loss function so that training becomes more stable and efficient. With the above optimizations, our model achieves a better trade-off between accuracy and efficiency on the push-up dataset. With only 4.02 M parameters, PUPose achieves 94.2% mAP.

This paper explores clothing virtual display technology based on human pose estimation and applies it to the Unity3D platform. This technology combines human pose estimation and virtual character-driven approaches to achieve rapid real-time clothing try-on and display effects. Firstly, key point detection of the human body in input images or videos is performed using human pose estimation techniques. This accurately infers the positions and connectivity relationships of human body keypoints, capturing subtle changes in human pose. Kalman filtering techniques are applied to enhance the stability and accuracy of pose estimation. Finally, we apply virtual character-driven technology to the Unity3D platform, developing a real-time dressing application that incorporates the estimated human pose results into virtual character models, enabling real-time clothing changes. By animating virtual characters, dynamic clothing display effects can be achieved, allowing users to instantly view and experience the effects of different clothing styles in various poses.