How do Foundation Models Compare to Skeleton-Based Approaches for Gesture Recognition in Human-Robot Interaction?

MuViH: Multi-View Hand gesture dataset and recognition pipeline for human–robot interaction in a collaborative robotic finishing platform

We explore new aspects on assistive living via smart social human-robot interaction (HRI) involving automatic recognition of multimodal gestures and speech in a natural interface, providing social features in HRI. We discuss a whole framework of resources, including datasets and tools, briefly shown in two real-life use cases for elderly subjects: a multimodal interface of an assistive robotic rollator and an assistive bathing robot. We discuss these domain specific tasks, and open source tools, which can be used to build such HRI systems, as well as indicative results. Sharing such resources can open new perspectives in assistive HRI.

International retailing: B. Sternquist, Fairchild Publications, New York, five parts, fifteen chapters, 592 pages, hardback, 1998, ISBN 1-56367-103-4

This paper presents an adaptive visual gesture recognition method for human–robot interaction using a knowledge-based software platform. The system is capable of recognizing users, static gestures comprised of the face and hand poses, and dynamic gestures of face in motion. The system learns new users, poses using multi-cluster approach, and combines computer vision and knowledge-based approaches in order to adapt to new users, gestures and robot behaviors. In the proposed method, a frame-based knowledge model is defined for the person-centric gesture interpretation and human-robot interaction. It is implemented using the Frame-based Software Platform for Agent and Knowledge Management (SPAK). The effectiveness of this method has been demonstrated by an experimental human-robot interaction system using a humanoid robot, namely, ‘Robovie’.

Incorporating hand gesture recognition in human–robot interaction has the potential to provide a natural way of communication, thus contributing to a more fluid collaboration toward optimizing the efficiency of the application at hand and overcoming possible challenges. A very promising field of interest is agriculture, owing to its complex and dynamic environments. The aim of this study was twofold: (a) to develop a real-time skeleton-based recognition system for five hand gestures using a depth camera and machine learning, and (b) to enable a real-time human–robot interaction framework and test it in different scenarios. For this purpose, six machine learning classifiers were tested, while the Robot Operating System (ROS) software was utilized for “translating” the gestures into five commands to be executed by the robot. Furthermore, the developed system was successfully tested in outdoor experimental sessions that included either one or two persons. In the last case, the robot, based on the recognized gesture, could distinguish which of the two workers required help, follow the “locked” person, stop, return to a target location, or “unlock” them. For the sake of safety, the robot navigated with a preset socially accepted speed while keeping a safe distance in all interactions.

Human–Robot Interaction (HRI) using hand gesture recognition offers an effective and non-contact approach to enhancing operational intuitiveness and user convenience. However, most existing studies primarily focus on either static sign language recognition or the tracking of hand position and orientation in space. These approaches often prove inadequate for controlling complex robotic systems. This paper proposes an advanced HRI system leveraging dynamic hand gestures for controlling quadruped robots equipped with a robotic arm. The proposed system integrates both semantic and pose information from dynamic gestures to enable comprehensive control over the robot’s diverse functionalities. First, a Depth–MediaPipe framework is introduced to facilitate the precise three-dimensional (3D) coordinate extraction of 21 hand bone keypoints. Subsequently, a Semantic-Pose to Motion (SPM) model is developed to analyze and interpret both the pose and semantic aspects of hand gestures. This model translates the extracted 3D coordinate data into corresponding mechanical actions in real-time, encompassing quadruped robot locomotion, robotic arm end-effector tracking, and semantic-based command switching. Extensive real-world experiments demonstrate the proposed system’s effectiveness in achieving real-time interaction and precise control, underscoring its potential for enhancing the usability of complex robotic platforms.

Human Behavior and Hand Gesture Classification for Smart Human-robot Interaction

Gesture recognition technology is rapidly growing in the recent years due to the demands of many application such as computer game and sport, human robot interaction, assistant systems, sign language interpretation and e-commerce. One of the most important of gesture recognition is hand-gesture recognition. For example, it can be used to control all devices (television, radio, air-condition, and doors) by just hand gestures for smart home application. The HGM-4 dataset is built for hand gesture recognition (the full dataset is available from: https://data.mendeley.com/datasets/jzy8zngkbg/4) which contains total 4,160 color images (1280 × 700 pixels) of 26 hand gestures captured by four cameras at different position. The training and testing set are defined to create a benchmark framework for comparing the experimental results.

Body gesture Recognition has been one of the interested research field for Human-Robot Interaction(HRI). Most of the conventional body gesture recognition algorithms used Hidden Markov Model(HMM) for modeling gestures which have spatio-temporal variabilities. However, HMM-based algorithms have difficulties excluding meaningless gestures. Besides, it is necessary for conventional body gesture recognition algorithms to perform gesture segmentation first, then sends the extracted gesture to the HMM for gesture recognition. This separated system causes time delay between two continuing gestures to be recognized, and it makes the system inappropriate for continuous gesture recognition. To overcome these two limitations, this paper suggests primitive body model encoding, which performs spatio/temporal quantization of motions from human body model and encodes them into predefined primitive codes for each link of a body model, and Selective/Asynchronous Input-Parallel State machine(SAI-PSM) for multiple-simultaneous gesture recognition. The experimental results showed that the proposed gesture recognition system using primitive body model encoding and SAI-PSM can exclude meaningless gestures well from the continuous body model data, while performing multiple-simultaneous gesture recognition without losing recognition rates compared to the previous HMM-based work.

Dynamic gesture recognition is one of the most interesting and challenging areas of Human-Robot-Interaction (HRI). Problems like image segmentation, temporal and spatial feature extraction and real time recognition are the most promising issues to name in this context. This work proposes a deep neural model to recognize dynamic gestures with minimal image preprocessing and real time recognition in an experimental set up using a humanoid robot. We conducted two experiments with command gestures in an offline fashion and for demonstration in a Human-Robot-Interaction (HRI) scenario. Our results showed that the proposed model achieves high classification rates of the gestures executed by different subjects, who perform them with varying speed. With our additional audio feedback we demonstrate that our system performs in real time.

Real-time gesture recognition is mandatory for gesture based human-robot natural interaction. Several glove-based approaches are introduced for this purpose, but they are un-natural and force user to wear some cumbersome glove. This paper presents a vision-based gesture recognition method for human-robot interface. Hand poses and face are detected using multiple feature-based template matching techniques from the three largest skin-like regions that are segmented using skin color information from YIQ color space. Gesture is recognized using rule-based approach whenever the three skin-like regions at a particular image frame match with the predefined gesture. Gesture command is sent to robot through a knowledge based software platform for human-robot interface. In this paper, a method has also been proposed to detect left hand and right hand relative to face position as well as, to detect and locate its position. The effectiveness of the proposed method has been demonstrated over the interaction with a pet robot named AIBO

As a promising component for body sensor networks, the wearable sensors for hand gesture recognition have increasingly received great attention in recent years. By interpreting human intentions through hand gestures, the natural human–robot interaction can be realized in the smart home where the youth and the elderly can perform hand gestures to control the household robot or the robotic wheelchair. Here, a wearable wrist-worn camera sensor was shown to recognize hand trajectory gestures. The moving velocity of the user’s hand was deduced from the matched speeded up robust features keypoints of the moving background of the video sequence. Furthermore, the segmentation of continuous gestures was achieved by detecting the predefined gesture starting signal from the hand region of the image, which was segmented by the lazy snapping algorithm. In this paper, 10 types of gestures and 1350 gesture samples collected from 15 subjects at three different scenes were classified by the dynamic time warping algorithm and the results achieved an average recognition accuracy up to 97.6%. Moreover, the practicability of the proposed system was further demonstrated by controlling a cooperative robot to draw letters on paper.

Establishing a communication mechanism for service and assistive robots to help the disabled and old age population through Human-Robot Interaction (HRI) is still a challenging task for further study. This research explores a vision and gesture-based control strategy, taking into account static and dynamic human gestures and full-body pose estimation without the use of any wearable devices. The human action recognition algorithm is coupled with an object detector algorithm to automatically perform a robotic task. The Deep Learning (DL) techniques, MediaPipe with Long Short-Term Memory (LSTM) model and You Only Look Once (YOLO) v9 algorithms are implemented for human gesture recognition and object detection respectively. For the robotic manipulation task, there are eight object classes, each uniquely associated with one of eight corresponding human gesture classes. These gestures are categorized into two types: four static gestures and four dynamic gestures. The dataset of object and gesture classes is prepared from scratch in this study. Based on the particular human gesture or action, the Intel RealSense depth camera provides position and depth information of the detected object, and the robotic arm picks and places that object using its Inverse Kinematics (IK) solution. The prediction outcomes and experimental demonstrations suggest that the proposed study performs quite well as compared to the recently used related studies and exhibits remarkable performance in establishing a real-time HRI system.

This paper presents an efficient approach to solve the problem of real-time robust hand segmentation and fingertip extraction for finger gesture recognition in Human-robot interaction (HRI). Firstly, we propose an improved cascade filtering based hand region candidate estimation by the combination of YC <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">b</inf> C <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">r</inf> skin color probability image based segmentation with genetic algorithm and post-processing with morphological operation and blob analysis in blurred, low-resolution images. Using the segmented hand candidate regions, we estimate the hand region center and fingertip position from distance transform and geometrical feature of hand. From the hand orientation and hand/palm center, we find the optimal each fingertip position and its orientation. Experimental results show that the proposed algorithm not only rapidly detects the hand regions under various illumination conditions, but also it efficiently extracts the finger information with size and rotation invariance.

Vision based human robot interaction (HRI) in a crowded scene is a challenging research problem. The aim of this paper is to provide a reliable framework for simple gesture recognition for robotic navigation under partial occlusion and varying illumination conditions. The proposed method combines hand motion segmentation and skin colour detection for gesture recognition. Motion clustering based on least median square error (LMedS) followed by Kalman filtering and HMM gesture detection has been used. Experimental results have shown that the method can successfully restore the motion field that allows accurate, dominant affine motion detection for consistent gesture estimation.