- https://doi.org/10.1088/1361-6501/ae5328
Dynamic robotic tracking of human limbs using vision-based reinforcement learning
- Mar 20, 2026
- Measurement Science and Technology
- Jin Sui +4 more
Abstract Most vision-based robotic tracking methods focus on general objects instead of human beings. This limitation stems from the inherent morphological variations in humans, characterized by non-rigid deformations and diverse kinematic configurations, which constrains the reliable limb tracking through conventional methods. The 3D representation of the human body provides rich spatial information and the inherent semantics of the limbs offer stable target signatures, while these properties remain under-exploited in dynamic motion tracking tasks. To address these challenges, we propose a dual-view fusion of perception-to-action framework for limb tracking, incorporating cross-view feature aggregation. This dual-view visual fusion attention mechanism captures both body shape and posture variations, as well as maintaining egocentric-perception of the target limb. The reasoning and planning capabilities across perspectives also be enhanced through this approach. Additionally, we present a feature extractor based on human semantic parsing to improve the limb perception. To further reduce the complexity of human image features and increase sim-to-real transfer efficiency, we develop an image matching mechanism based on 3D human reconstruction. Through the examination of various simulated and real-robot examples, our methods exhibit superior properties compared with current vision-based methods in terms of the success rate.
- # Vision-based Reinforcement Learning
- # 3D Human Reconstruction
- # Rich Spatial Information
- # Limb Tracking
- # Vision-based Methods
- # Non-rigid Deformations
- # Posture Variations
- # Planning Capabilities
- # Robotic Tracking
- # Dynamic Tracking