Joint object recognition and pose estimation using multiple-anchor triplet learning of canonical plane

This paper presents an appearance-based method for object pose recognition using single camera image. The basic idea of our method is to find the correlation between object image and its pose, and use it for object recognition and pose estimation. Canonical correlation analysis is introduced to derive such correlation and build a compact appearance model. In the approach, we first obtain a pair of training data set, i.e., object images and their pose parameters. The appearance model is given as the subspace spanned by the canonical vectors that maximize the correlation between images and poses. Pose parameters of currently observed image is predicted by finding the regression coefficient in this subspace. We also introduce the kernel methods to cope with the non-linearity lies in training data set. Experiments are conducted on object pose estimation and vehicle type classification problem. Performance of our appearance models is discussed through the comparison with conventional subspace method

The next generation of service robots is to offer services relying heavily on visually guided manipulation, besides navigation, such services as errand, logistics, appliance, home keeping, etc. For the successful introduction of these services, it is critical to establish the consumer-level dependability in 3D object recognition and pose estimation in a natural setting where a large variation of environment, e.g., perspective, texture, form factor, illumination, occlusion, etc., is common. To address this problem, we propose an approach of the two-layered particle filter to the dependability in 3D object recognition and pose estimation. In the upper layer, a set of object pose candidates is identified and maintained in the search space as a set of super-particles, each of which is assigned a probability of the true pose and evolved in time along with the accumulation of further evidences. To define the object pose candidates, first, we acquire initially weak evidences quickly and interpret them in terms of possible object poses in space. These interpretations serve as the region of interest for detailed investigation by which the pose probabilities are computed for individual interpretations based on the likelihood and unlikelihood of various features available in the corresponding regions of interest. During the process of probability computation, we select the object pose candidates to be used as super-particles in the upper layer. In the lower level, the pose uncertainties associated with the individual candidates are represented as particles that are subject to the propagation in time. Finally, the experimental results support the strength of the proposed approach in the real environment in terms of its dependability in 3D object recognition and pose estimation.

Object recognition and pose estimation are critical components in autonomous robot manipulation systems, playing a crucial role in enabling robots to interact effectively with the environment. During actual execution, the robot must recognize the object in the current scene, estimate its pose, and then select a feasible grasp pose from the pre-defined grasp configurations. While most existing methods primarily focus on pose estimation, they often neglect the graspability and reachability aspects. This oversight can lead to inefficiencies and failures during execution. In this study, we introduce an innovative graspability-aware object pose estimation framework. Our proposed approach not only estimates the poses of multiple objects in clustered scenes but also identifies graspable areas. This enables the system to concentrate its efforts on specific points or regions of an object that are suitable for grasping. It leverages both depth and color images to extract geometric and appearance features. To effectively combine these diverse features, we have developed an adaptive fusion module. In addition, the fused features are further enhanced through a graspability-aware feature enhancement module. The key innovation of our method lies in improving the discriminability and robustness of the features used for object pose estimation. We have achieved state-of-the-art results on public datasets when compared to several baseline methods. In real robot experiments conducted on a Franka Emika robot arm equipped with an Intel Realsense camera and a two-finger gripper, we consistently achieved high success rates, even in cluttered scenes.

State-of-the-art object recognition and pose estimation systems often utilize point feature algorithms, which in turn usually require the computing power of conventional PC hardware. In this paper, we describe two embedded systems for object detection and pose estimation using sophisticated point features. The feature detection step of the “Speeded-up Robust Features (SURF)” algorithm is accelerated by a special IP core. The first system performs object detection and is completely implemented in a single medium-size Virtex-5 FPGA. The second system is an augmented reality platform, which consists of an ARM-based microcontroller and intelligent FPGA-based cameras which support the main system.

There is typically a need for physical training and assessment of the physical exercise poses of students in schools or members of fitness centers. However, few studies have been conducted on physical exercise poses, and no relevant dataset is available. We introduce a new dataset built for the particular physical fitness action of push-ups called the push-up dataset. We propose a new real-time and device-friendly pose estimation model named PUPose. The trade-off between accuracy and efficiency in human pose estimation has been challenging. In this paper, some recent studies have extended object detectors to unify person detection and pose estimation. Our PUPose combines top-down and bottom-up approaches to get bounding boxes for multiple individuals and their corresponding poses. We investigate the applicability of methods for extending object detectors to estimate human pose to lightweight pose estimation models. We strengthen the backbone structure and design an efficient neck structure, improving the feature extraction capability of the network. We optimize the loss function so that training becomes more stable and efficient. With the above optimizations, our model achieves a better trade-off between accuracy and efficiency on the push-up dataset. With only 4.02 M parameters, PUPose achieves 94.2% mAP.

This paper presents a probabilistic object recognition and pose estimation method using multiple interpretation generation in cluttered indoor environments. How to handle pose ambiguity and uncertainty is the main challenge in most recognition systems. In order to solve this problem, we approach it in a probabilistic manner. First, given a three-dimensional (3D) polyhedral object model, the parallel and perpendicular line pairs, which are detected from stereo images and 3D point clouds, generate pose hypotheses as multiple interpretations, with ambiguity from partial occlusion and fragmentation of 3D lines especially taken into account. Different from the previous methods, each pose interpretation is represented as a region instead of a point in pose space reflecting the measurement uncertainty. Then, for each pose interpretation, more features around the estimated pose are further utilized as additional evidence for computing the probability using the Bayesian principle in terms of likelihood and unlikelihood. Finally, fusion strategy is applied to the top ranked interpretations with high probabilities, which are further verified and refined to give a more accurate pose estimation in real time. The experimental results show the performance and potential of the proposed approach in real cluttered domestic environments.

Reliable object recognition is a mandatory prerequisite for Service Robots in everyday environments. Typical approaches for object recognition use single algorithms or features. However, none is yet able to classify across all types of objects and the field of object recognition is thus still an open challenge. We propose an approach for object recognition and pose estimation that combines existing algorithms. Probabilistic methods are used to fuse the classification and pose estimation results, considering the error introduced by the measurements, actuators (sensor on manipulator) and algorithms. Since integration is one of the real challenges from the laboratory towards the real world, we demonstrate the approach in two fully integrated scenarios. We run the experiments on two platforms and focus on the distinction of few but similar objects.

The SIFT algorithm (Scale Invariant Feature Transform) proposed by Lowe [1] is an approach for extracting distinctive invariant features from images. It has been successfully applied to a variety of computer vision problems based on feature matching including object recognition, pose estimation, image retrieval and many others. However, in real-world applications there is still a need for improvement of the algorithm’s robustness with respect to the correct matching of SIFT features. In this paper, an improvement of the original SIFT algorithm providing more reliable feature matching for the purpose of object recognition is proposed. The main idea is to divide the features extracted from both the test and the model object image into several sub-collections before they are matched. The features are divided into several sub-collections considering the features arising from different octaves, that is from different frequency domains. The SIFT algorithm (Scale Invariant Feature Transform) proposed by Lowe [1] is an approach for extracting distinctive invariant features from images. It has been successfully applied to a variety of computer vision problems based on feature matching including object recognition, pose estimation, image retrieval and many others. However, in real-world applications there is still a need for improvement of the algorithm’s robustness with respect to the correct matching of SIFT features. In this paper, an improvement of the original SIFT algorithm providing more reliable feature matching for the purpose of object recognition is proposed. The main idea is to divide the features extracted from both the test and the model object image into several sub-collections before they are matched. The features are divided into several sub-collections considering the features arising from different octaves, that is from different frequency domains.

This paper explores clothing virtual display technology based on human pose estimation and applies it to the Unity3D platform. This technology combines human pose estimation and virtual character-driven approaches to achieve rapid real-time clothing try-on and display effects. Firstly, key point detection of the human body in input images or videos is performed using human pose estimation techniques. This accurately infers the positions and connectivity relationships of human body keypoints, capturing subtle changes in human pose. Kalman filtering techniques are applied to enhance the stability and accuracy of pose estimation. Finally, we apply virtual character-driven technology to the Unity3D platform, developing a real-time dressing application that incorporates the estimated human pose results into virtual character models, enabling real-time clothing changes. By animating virtual characters, dynamic clothing display effects can be achieved, allowing users to instantly view and experience the effects of different clothing styles in various poses.

Currently, many methods of object pose estimation use images or point clouds alone for pose estimation. This leads to their inability to accurately estimate the object pose in the case of occlusion and poor illumination. Second, these models have large parameters and cannot be deployed on mobile devices. Therefore, we propose a lightweight two-terminal feature fusion network, which can effectively use images and point clouds for accurate object pose estimation. First, Pointno problemNet network is used to extract point cloud features. Then the extracted point cloud features are combined with the images at pixel level and the features are extracted by CNN. Secondly, the extracted image features are combined with the point cloud point by point. Then feature extraction is performed on it using the improved PointNet++ network. Finally, a set of center point features are obtained and pose estimation is performed for each feature. The pose with the highest confidence is selected as the final result. Furthermore, we apply depthwise separable convolutions to reduce the amount of model parameters. Experiments show that the proposed method exhibits better performance on Linemod and Occlusion Linemod datasets. Furthermore, the model parameters are small, and it is robust in occlusion and low-light situations.

Purpose Accurate 6D object pose estimation is essential for various robotic tasks. Uncertain pose estimates can lead to task failures; however, a certain degree of error in the pose estimates is often acceptable. This paper aims to enable the robots to make informed decisions by quantifying errors in the object pose estimate and acceptable errors for task success. Design/methodology/approach In this paper, the authors introduce a framework for evaluating robotic task success under object pose uncertainty, representing both the estimated error space of the object pose and the acceptable error space for task success using multi-modal non-parametric probability distributions. The proposed framework pre-computes the acceptable error space for task success using dynamic simulations and subsequently integrates the pre-computed acceptable error space over the estimated error space of the object pose to predict the likelihood of the task succes. Findings The authors evaluated the proposed framework on two mobile manipulation tasks. Their results show that by representing the estimated and the acceptable error space using multi-modal non-parametric distributions, the authors achieve higher task success rates and fewer failures. Research limitations/implications Their proposed framework is generic and can be applied to a wide range of robotic tasks requiring object pose estimation. Hence, given the recent advancements in object pose uncertainty estimation and dynamic simulations, the proposed framework, in conjunction with these advancements, has the potential to enable robots to make reliable and informed decisions under pose uncertainty. Originality/value Unlike related works that model both acceptable error space and estimated error space using parametric uni-modal distributions, the authors model them as multi-modal distributions which is often the case in the real world.

In this paper, a novel global point cloud descriptor is proposed for reliable object recognition and pose estimation, which can be effectively applied to robot grasping operation. The viewpoint feature histogram (VFH) is widely used in three-dimensional (3D) object recognition and pose estimation in real scene obtained by depth sensor because of its recognition performance and computational efficiency. However, when the object has a mirrored structure, it is often difficult to distinguish the mirrored poses relative to the viewpoint using VFH. In order to solve this difficulty, this study presents an improved feature descriptor named orthogonal viewpoint feature histogram (OVFH), which contains two components: a surface shape component and an improved viewpoint direction component. The improved viewpoint component is calculated by the orthogonal vector of the viewpoint direction, which is obtained based on the reference frame estimated for the entire point cloud. The evaluation of OVFH using a publicly available data set indicates that it enhances the ability to distinguish between mirrored poses while ensuring object recognition performance. The proposed method uses OVFH to recognize and register objects in the database and obtains precise poses by using the iterative closest point (ICP) algorithm. The experimental results show that the proposed approach can be effectively applied to guide the robot to grasp objects with mirrored poses.

With the advancement of intelligent manufacturing, robots are increasingly involved in industrial production, for example, replacing human beings to identify and manipulate objects. However, it is still a challenging task for object recognition and pose estimation in industrial environment, especially under large environmental noise, mutual occlusions and high-speed requirements. In this paper, we propose a new method for 6D object pose estimation, Super Key 4-Points Congruent Sets (SK-4PCS), which is a fast global method combined with local features. The proposed method includes three steps. Firstly, the key points in 3D point clouds are extracted by local features, which are invariant with respect to rotation and translation. Secondly, the smart indexing scheme is used to realize the query of 4-points congruent sets globally. Finally, the coarse pose obtained quickly is further refined by the Iteration Closest Point algorithm (ICP) [1]. The proposed method is also compared with existing methods by the evaluation indicator Visible Surface Discrepancy (VSD) on the LINEMOD dataset [2]. The experimental results show that the refined SK-4PCS results in significant speedup and satisfactory accuracy in 6D object pose estimation.

We propose UniPose+, a unified framework for 2D and 3D human pose estimation in images and videos. The UniPose+ architecture leverages multi-scale feature representations to increase the effectiveness of backbone feature extractors, with no significant increase in network size and no postprocessing. Current pose estimation methods heavily rely on statistical postprocessing or predefined anchor poses for joint localization. The UniPose+ framework incorporates contextual information across scales and joint localization with Gaussian heatmap modulation at the decoder output to estimate 2D and 3D human pose in a single stage with state-of-the-art accuracy, without relying on predefined anchor poses. The multi-scale representations allowed by the waterfall module in the UniPose+ framework leverage the efficiency of progressive filtering in the cascade architecture, while maintaining multi-scale fields-of-view comparable to spatial pyramid configurations. Our results on multiple datasets demonstrate that UniPose+, with a HRNet, ResNet or SENet backbone and waterfall module, is a robust and efficient architecture for single person 2D and 3D pose estimation in single images and videos.

Validity of an artificial intelligence, human pose estimation model for measuring single-leg squat kinematics