Adaptive Joint Interdependency Learning for 2D Occluded Hand Pose Estimation

3D hand pose estimation has garnered great attention in recent years due to its critical applications in human-computer interaction, virtual reality, and related fields. Accurate estimation of hand joints is essential for high-quality hand pose estimation. However, existing methods neglect the importance of Distal Phalanx Tip (TIP) and Wrist in predicting hand joints overall and often fail to account for the phenomenon of error accumulation for distal joints in gesture estimation, which can cause certain joints to incur larger errors, resulting in misalignments and artifacts in pose estimation and degrading the overall reconstruction quality. To address this challenge, we propose a novel segmented architecture for enhanced hand pose estimation (EHPE). We perform a local extraction of the TIP and wrist, thus alleviating the effect of error accumulation on the prediction of the TIP and further reduce the predictive errors for all joints on this basis. EHPE consists of two key stages: In the TIP and Wrist Joints Extraction stage (TW-stage), the positions of the TIP and wrist joints are estimated to provide an initial accurate joint configuration; In the Prior Guided Joints Estimation stage (PG-stage), a dual-branch interaction network is employed to refine the positions of the remaining joints. Extensive experiments on two widely used benchmarks demonstrate that EHPE achieves state-of-the-art performance.

Camera distance helps 3D hand pose estimated from a single RGB image

3D hand pose estimation can provide basic information about gestures, which has an important significance in the fields of Human-Machine Interaction (HMI) and Virtual Reality (VR). In recent years, 3D hand pose estimation from a single depth image has made great research achievements due to the development of depth cameras. However, 3D hand pose estimation from a single RGB image is still a highly challenging problem. In this work, we propose a novel four-stage cascaded hierarchical CNN (4CHNet), which leverages hierarchical network to decompose hand pose estimation into finger pose estimation and palm pose estimation, extracts separately finger features and palm features, and finally fuses them to estimate 3D hand pose. Compared with direct estimation methods, the hand feature information extracted by the hierarchical network is more representative. Furthermore, concatenating various stages of the network for end-to-end training can make each stage mutually beneficial and progress. The experimental results on two public datasets demonstrate that our 4CHNet can significantly improve the accuracy of 3D hand pose estimation from a single RGB image.

Hand pose estimation from 3D data is a key challenge in computer vision as well as an essential step for human–computer interaction. A lot of deep learning-based hand pose estimation methods have made significant progress but give less consideration to the inner interactions of input data, especially when consuming hand point clouds. Therefore, this paper proposes an end-to-end capsule-based hand pose estimation network (Capsule-HandNet), which processes hand point clouds directly with the consideration of structural relationships among local parts, including symmetry, junction, relative location, etc. Firstly, an encoder is adopted in Capsule-HandNet to extract multi-level features into the latent capsule by dynamic routing. The latent capsule represents the structural relationship information of the hand point cloud explicitly. Then, a decoder recovers a point cloud to fit the input hand point cloud via a latent capsule. This auto-encoder procedure is designed to ensure the effectiveness of the latent capsule. Finally, the hand pose is regressed from the combined feature, which consists of the global feature and the latent capsule. The Capsule-HandNet is evaluated on public hand pose datasets under the metrics of the mean error and the fraction of frames. The mean joint errors of Capsule-HandNet on MSRA and ICVL datasets reach 8.85 mm and 7.49 mm, respectively, and Capsule-HandNet outperforms the state-of-the-art methods on most thresholds under the fraction of frames metric. The experimental results demonstrate the effectiveness of Capsule-HandNet for 3D hand pose estimation.

\n Hand pose estimation is a challenging problem in computer vision with a wide range of applications, especially in human-computer interface. With the development of inexpensive consumer-level depth cameras and the evolution on deep learning techniques, the current state-of-art in the problem is continuously developing and several new methods have been proposed in recent years. Those methods are mostly data-driven and reach good results in standard datasets such as NYU, ICVL and HANDS17. An application that would benefit from the use of computer vision techniques is hand occupational therapy. In chronic diseases like rheumatoid arthritis (RA), the evaluation of the hand functional state is fundamental for the treatment and prevention of finger deformities. One of the procedures for deformity diagnosis is the measurement of movement angles i.e. flexion/extension and abduction/addution, made using goniometers in a process that can be time-consuming and invasive for the patient. The main proposal of this PhD is to fill a gap in the literature by proposing and evaluating the viability of using a framework composed of a 3D low-cost sensor and a 3D hand pose estimation state-of-art method for automatic assessment of rheumatoid arthritis patients. Given depth maps as input, our framework estimates 3D hand joint positions using a deep convolutional neural network. The proposed pose estimation algorithm can be executed in real-time, allowing users to visualise 3D skeleton tracking results at the same time as the depth images are acquired. Once 3D joint poses are obtained, our framework estimates flexion/extension and abduction/adduction angles by applying computational geometry oper- ations. The absence of public datasets with RA patients in the literature makes the estimation of hand poses of patients a challenge for computer vision data-driven methods. We therefore proposed a protocol to acquire new data from groups of patients and control. We performed experiments of identification of RA patients and control sets and also performed comparison with goniometer data. Results show that a method based on Fourier descriptors is able to perform automatic discrimination of hands with Rheumatoid Arthritis (RA) and healthy pa- tients. The angle between joints can be used as an indicative of current movement capabilities and function. The acquisition is much easier, non-invasive and patient-friendly, significantly reducing the evaluation time and offering real-time data for the dynamic movement.\n

The latest advances in smart sensor technology, e.g., Leap Motion Sensor, has increased the precision in tracking fully articulated human hand and finger movements, without the need for placing electrical or optical markers. A remaining challenge is finger occlusion, which can affect tracking accuracy. In this paper, we introduce a spatio-temporal optimization technique for motion data generated from multiple sensors. We demonstrate that our algorithm can produce a fused stream of probabilistic optimal hand poses, by improving local spatial domain analysis and proposing a fast and effective flow analysis technique in the temporal domain, which computes how well the hand pose estimation in the current frame fits the movement flow within a time segment. By using an artificial hand to represent the hand pose ground truth at selected time steps, experimental results demonstrate that our spatio-temporal optimization algorithm increases the estimation accuracy by 6% compared to the reference method, achieving an overall accuracy of 91.29%. Our proposed method can be used offline or in real-time, and can benefit a wide range of applications, including surgical planning and training, where hand motion is the focus of performance efficiency and assessment.

Top-down model fitting for hand pose recovery in sequences of depth images

Accurate 3D hand pose estimation supports surgical applications such as skill assessment and robot-assisted interventions. However, surgical environments pose severe challenges, including intense and localized lighting, frequent occlusions by instruments or staff, and uniform hand appearance due to gloves, combined with a scarcity of annotated datasets for reliable model training. We propose a multi-view pipeline for 3D hand pose estimation in surgical contexts that requires no domain-specific fine-tuning and relies on off-the-shelf pretrained models. The pipeline combines person detection, whole-body pose estimation, and 2D hand keypoint prediction on tracked hand crops, followed by constrained 3D optimization. We also introduce a benchmark dataset with over 68,000 frames and 3,000 manually annotated 2D hand poses with triangulated 3D ground truth, recorded in a high-fidelity physical operating room replica. Quantitative evaluation on the proposed dataset shows that the pipeline consistently outperforms state-of-the-art baselines for both two- and three-dimensional hand pose estimation. The method substantially improves joint localization accuracy, achieving up to a 31% reduction in 2D joint error and a 76% reduction in 3D joint position error. The approach remains robust across increasing levels of scene complexity, including motion and partial occlusions. The proposed multi-view pipeline demonstrates the potential of combining robust detection, tracking, and geometric optimization for three-dimensional hand pose estimation in surgical environments without domain-specific retraining. Together with the introduced dataset, this work provides a baseline framework and benchmarking resource to support future research on surgical motion analysis and objective skill assessment.

Manipulation performed by humans contains a lot of information that helps robots to learn how to handle objects. Since hand poses and motions are related to manipulated objects, extracting this information is one of the important tasks for robotics community. This paper presents a framework to recognize human manipulations including hand motions, hand poses, and shapes of manipulated objects using egocentric RGB-D videos. Our framework is straightforward but powerful through the efficient use of depth information and egocentric vision. We estimate hand poses with an example-based method through the limited appearances of the hand in egocentric vision. First, from a sensed point cloud, our framework distinguishes hands, manipulated objects and an environment using skin color detection and limitation on the range of the moving hand. Next, we estimate a hand pose by aligning the extracted hand point cloud with a pre-recorded database of hand point clouds of different poses. The position and orientation of the head-mounted sensor are estimated to acquire the hand motion in the world coordinate system. Then, the type of hand motion is classified using Dynamic Programming matching between a series of velocity vectors of estimated and a database of wrist trajectories. Finally, we experiment the effectiveness of our framework for hand motion recognition to validate our work.

We present a method for robust and accurate hand pose recognition from egocentric depth cameras. Our method combines CNN based hand pose estimation and joint locations based hand gesture recognition. In pose estimation stage, we use a hand geometry prior network to estimate the hand pose. In gesture recognition stage, we defined a hand language which based on a set of pre-define basic propositions, obtained by applying four predicate types to the fingers and palm states. The hand language is used to convert the estimated joint location to hand gesture. Our experimental results indicate that the method enables robust and accurate gesture recognition in self-occlusion environment.

In this paper, a hand pose estimation method is introduced that combines MobileNetV3 and CrossInfoNet into a single pipeline. The proposed approach is tailored for mobile phone processors through optimizations, modifications, and enhancements made to both architectures, resulting in a lightweight solution. MobileNetV3 provides the bottleneck for feature extraction and refinements while CrossInfoNet benefits the proposed system through a multitask information sharing mechanism. In the feature extraction stage, we utilized an inverted residual block that achieves a balance between accuracy and efficiency in limited parameters. Additionally, in the feature refinement stage, we incorporated a new best-performing activation function called “activate or not” ACON, which demonstrated stability and superior performance in learning linearly and non-linearly gates of the whole activation area of the network by setting hyperparameters to switch between active and inactive states. As a result, our network operated with 65% reduced parameters, but improved speed by 39% which is suitable for running in a mobile device processor. During experiment, we conducted test evaluation on three hand pose datasets to assess the generalization capacity of our system. On all the tested datasets, the proposed approach demonstrates consistently higher performance while using significantly fewer parameters than existing methods. This indicates that the proposed system has the potential to enable new hand pose estimation applications such as virtual reality, augmented reality and sign language recognition on mobile devices.

Hand pose estimation in object-interaction based on deep learning for virtual reality applications

We tackle the practical problem of hand pose estimation from a single noisy depth image. A dedicated three-step pipeline is proposed: Initial estimation step provides an initial estimation of the hand in-plane orientation and 3D location, Candidate generation step produces a set of 3D pose candidate from the Hough voting space with the help of the rotational invariant depth features, Verification step delivers the final 3D hand pose as the solution to an optimization problem. We analyze the depth noises, and suggest tips to minimize their negative impacts on the overall performance. Our approach is able to work with Kinect-type noisy depth images, and reliably produces pose estimations of general motions efficiently (12 frames per second). Extensive experiments are conducted to qualitatively and quantitatively evaluate the performance with respect to the state-of-the-art methods that have access to additional RGB images. Our approach is shown to deliver on par or even better results.

Markerless vision-based teleoperation that leverages innovations in computer vision offers the advantages of allowing natural and noninvasive finger motions for multifingered robot hands. However, current pose estimation methods still face inaccuracy issues due to the self-occlusion of the fingers. Herein, we develop a novel vision-based hand-arm teleoperation system that captures the human hands from the best viewpoint and at a suitable distance. This teleoperation system consists of an end-to-end hand pose regression network and a controlled active vision system. The end-to-end pose regression network (Transteleop), combined with an auxiliary reconstruction loss function, captures the human hand through a low-cost depth camera and predicts joint commands of the robot based on the image-to-image translation method. To obtain the optimal observation of the human hand, an active vision system is implemented by a robot arm at the local site that ensures the high accuracy of the proposed neural network. Human arm motions are simultaneously mapped to the slave robot arm under relative control. Quantitative network evaluation and a variety of complex manipulation tasks, for example, tower building, pouring, and multitable cup stacking, demonstrate the practicality and stability of the proposed teleoperation system.

Hand pose estimation from monocular images has been widely applied in many fields. However, the depth ambiguity of the hand and occlusion or lighting change caused by hand-object interaction make the accurate pose estimation challenge. Currently, many methods increase the depth of models to improve the accuracy of hand pose estimation networks, significantly increasing the number of model parameters and computational complexity. To solve this issue, we propose an improved High-Resolution Network (HRDNet) for hand pose estimation. Our method mainly consists of four stages: image feature extraction, 2D information prediction, 3D joint prediction, and hand mesh reconstruction. For image feature extraction, to enhance the network’s multiscale representation capabilities, we propose a hand feature perception module and a lightweight basic module. For the 2D information prediction module, a hand pose estimation network is proposed to fully utilize the 2D information of the hand while predicting 2D joint heat maps. Additionally, we also propose a hand joint pose encoder to improve the prediction accuracy of key points in heat maps. Then, in the process of 3D joint prediction, a two-level cascaded pose estimation network is proposed to predict 3D joint coordinates. Finally, we take the Inverse Kinematics Network (IKNet) to regress pose and shape parameters for generating reconstructed hand mesh in the hand Model with Articulated and Nonrigid defOrmations (MANO) model. Our extensive experiments on two publicly available datasets have shown that our method achieves competitive results compared with state-of-the-art hand pose estimation methods.