• Home
  • Search
  • Dynamic robotic tracking of human limbs using vision-based reinforcement learning
  • https://doi.org/10.1088/1361-6501/ae5328Copy DOI Icon

Dynamic robotic tracking of human limbs using vision-based reinforcement learning

Show More
  • Abstract
  • Literature Map
  • References
  • Similar Papers
Abstract

Abstract Most vision-based robotic tracking methods focus on general objects instead of human beings. This limitation stems from the inherent morphological variations in humans, characterized by non-rigid deformations and diverse kinematic configurations, which constrains the reliable limb tracking through conventional methods. The 3D representation of the human body provides rich spatial information and the inherent semantics of the limbs offer stable target signatures, while these properties remain under-exploited in dynamic motion tracking tasks. To address these challenges, we propose a dual-view fusion of perception-to-action framework for limb tracking, incorporating cross-view feature aggregation. This dual-view visual fusion attention mechanism captures both body shape and posture variations, as well as maintaining egocentric-perception of the target limb. The reasoning and planning capabilities across perspectives also be enhanced through this approach. Additionally, we present a feature extractor based on human semantic parsing to improve the limb perception. To further reduce the complexity of human image features and increase sim-to-real transfer efficiency, we develop an image matching mechanism based on 3D human reconstruction. Through the examination of various simulated and real-robot examples, our methods exhibit superior properties compared with current vision-based methods in terms of the success rate.

Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.