Real-time Vision-based Gesture Recognition for Human Robot Interaction

This paper presents a vision-based face and gesture recognition system for human-robot interaction. By using subspace method, face and predefined hand poses are detected from the three largest skin-like regions that are segmented using YIQ color representation system. In this subspace method we consider separate eigenspaces for each class or pose. Gesture is recognized using the rule-based approach whenever the combination of three skin-like regions at a particular image frame matches with the predefined gesture. These gesture commands are sent to robot through TCP/IP network for human-robot interaction. Using subspace method pose invariant face recognition has also been addressed. The effectiveness of this method has been demonstrated by interacting with an entertainment robot named AIBO.

Dynamic gesture recognition is one of the most interesting and challenging areas of Human-Robot-Interaction (HRI). Problems like image segmentation, temporal and spatial feature extraction and real time recognition are the most promising issues to name in this context. This work proposes a deep neural model to recognize dynamic gestures with minimal image preprocessing and real time recognition in an experimental set up using a humanoid robot. We conducted two experiments with command gestures in an offline fashion and for demonstration in a Human-Robot-Interaction (HRI) scenario. Our results showed that the proposed model achieves high classification rates of the gestures executed by different subjects, who perform them with varying speed. With our additional audio feedback we demonstrate that our system performs in real time.

MuViH: Multi-View Hand gesture dataset and recognition pipeline for human–robot interaction in a collaborative robotic finishing platform

Gesture recognition, as an important means of human-computer interaction, can achieve more natural and flexible human-computer interaction, so it has been widely concerned by researchers in the field of computer vision. At present, most gesture recognition algorithms are based on monocular visual images and recognize the apparent features of hands. Most gesture image segmentation methods are carried out in color space according to skin color information. These methods are highly susceptible to interference from the external environment, such as lighting, background, etc. Convolutional neural network has the advantages of strong anti-interference and outstanding self-organization and self-learning ability. Therefore, based on the principle of convolutional neural network, a novel deep convolutional neural network dedicated to gesture recognition was designed in this paper. This network combines skin color information with finger position information for gesture recognition. Experimental results showed that the algorithm based on fingertip position information has better performance than the algorithm based solely on skin color information. Moreover, the network has simple structure and few parameters. Compared with VGG16 and other classical networks, the recognition accuracy is basically the same under the premise of fewer parameters and structural layers, and the recognition effect is better than other classical networks.

Towards establishing future-oriented relations between humans and robots, not only verbal, but also nonverbal robot functions have been studied. However, it is not clear how these different functions should be combined and implemented in robots in order to achieve ‘natural’ human-robot communication or interaction. In this paper, we focus on the timing for the coordination between utterance as a verbal function and body motion as a nonverbal function. To begin with, we investigated the temporal order and correlation between utterance and gesture in an interpersonal (human-human) interaction scenario as a preliminary experiment (Experiment 1). Then, we controlled the robot's response timing based on findings of the first experiment and evaluated the effectiveness of the robot's timing control in human-robot interaction (Experiment 2). It was shown that elderly participants significantly preferred the controlled robot's response. In addition, the timing of utterance and gestures affected the impression of the robot and interpretation of the verbal message. These results are important findings for designing socially interactive robots for the aging society.

This research focuses on real-time gesture learning and recognition. Events arrive in a continuous stream without explicitly given boundaries. To obtain temporal accuracy, weneed to consider the lag between the detection of an eventand any effects we wish to trigger with it. Two methodsfor real time gesture recognition using a Nintendo Wii controller are presented. The first detects gestures similar to agiven template using either a Euclidean distance or a cosinesimilarity measure. The second method uses novel information theoretic methods to detect and categorize gestures inan unsupervised way. The role of supervision, detection lagand the importance of haptic feedback are discussed.

One of the challenges of outdoor robots is developing effective portable Human-Robot-Interaction (HRI) frameworks. Hand-held devices offer a practical solution. By equipping these devices with robot software, they can be made to interact with the outdoor robots. Android devices are ideal as they are open source and can be integrated with robots powered by the Robot Operating System (ROS), also open source. However, due to the limits of rosjava, the mechanism that links ROS with android, and the conflicting modes of operation between ROS and android, current implementations of ROS-android offer limited robot applications that do not support advanced operations such as autonomous navigation and others. This paper implements selective compartmentalization to overcome these limitations, by combining ROS with android through a number of ROS and android bridges that would facilitate the development of advanced robot applications. Through the proposed method, authors were able to develop a portable HRI framework that allowed human operators to supervise an outdoor mobile robot while it performed an autonomous task. From their mobile devices, users were able to initialize the robot, configure its motion, and monitor its progress. Also, users were able to reprogram the robot to perform new tasks (not previously planned) through a creative use of features offered in the developed HRI framework. Also, user cognitive effort was reported to be low as evident by the positive score on the NASA-TLX scale test which was corroborated with robot performance data. This paper presents the detailed development and implementation steps.

Gesture is one of the fundamental ways of human machine natural interaction. To understand gesture, the system should be able to interpret 3D movements of human. This paper presents a computer vision-based real-time 3D gesture recognition system using depth image which tracks 3D joint position of head, neck, shoulder, arms, hands and legs. This tracking is done by Kinect motion sensor with OpenNI API and 3D motion gesture is recognized using the movement trajectory of those joints. User to Kinect sensor distance is adapted using proposed center of gravity (COG) correction method and 3D joint position is normalized using proposed joint position normalization method. For gesture learning and recognition, data mining classification algorithms such as Naive Bayes and neural network is used. The system is trained to recognize 12 gestures used by umpires in a cricket match. It is trained and tested using about 2000 training instances for 12 gesture of 15 persons. The system is tested using 5-fold cross validation method and achieved 98.11% accuracy with neural network and 88.84% accuracy with Naive Bayes classification method.

This paper develops a Kinect-based gesture command control method for driving the humanoid robot to learn human actions. The popular Kinect sensor is well-known for its high performances on gesture recognition. This work uses three different recognition mechanisms, dynamic time wrapping (DTW), hidden Markov model (HMM) and principal component analysis (PCA)-based eigenspace approaches for performing the recognition task of the specified human active gestures captured by the Kinect sensor. The recognized gesture command is then used to control the action of a humanoid robot where the humanoid robot will imitate the human active gesture according the content of the gesture command. By the presented method, the humanoid robot will effectively learn the human action. A series of experiments on gesture recognition and humanoid robot control are done to compare the gesture command recognition performance of three recognition methods and evaluate the similarity degree of humanoid robot imitations on the gesture command-indicated human active gesture.

Human–robot interaction (HRI), in all its different variants, has gained significant attention in industry. The combination of human skills and robotics significantly enhances manufacturing, improving operations quality and efficiency. This is particularly relevant in highly variable contexts like remanufacturing, which aims to preserve the value of products and components over time and involves complex, non-standardized tasks such as inspection, disassembly, reprocessing, and reassembly. Uncertainties related to the conditions of products to remanufacture cannot be managed by pure automation making HRI highly beneficial. Although researchers have focused on the use of HRI in several fields of application (from construction to logistics), no extensive mapping of the research trends has specifically addressed the intersection of HRI and remanufacturing contexts. To fill this gap, this paper presents a comprehensive study based on a bibliometric analysis to explore the current state of research on this topic. Based on 120 articles from the Scopus database, this study points out the main countries, authors, and affiliations involved in the field. In addition, the key topics were analyzed. A predominant focus, above all in the last three years, on Human-Robot Collaboration (HRC) among the different levels of HRI, and disassembly processes was highlighted, while other related activities, such as reassembly, have received comparatively less attention. An unexplored frontier is the integration of Artificial Intelligence and Machine Learning with HRC tasks in remanufacturing. These findings offer valuable insights for researchers, scholars, and industrial professionals aiming to advance HRI in remanufacturing.

The market for industrial robots grew rapidly during the 1970s and 1980s, with a peak demand in 1991. However, due to the subsequent recession in the world economy, the market for industrial robots has been slow or stagnant over the last decade. The value of industrial robots has decreased in that term, even though they have undergone considerable technical advances. On the other hand, human interactive robots for psychological enrichment are a type of service robots that provide a service by interacting with humans while stimulating their minds. As opposed to industrial robots, accuracy or speed is not always of prime importance. Their function or purpose is not simply entertainment, but also to render assistance, to guide, to provide therapy, to educate, to enable communication, and so on. The market for human interactive robots designed for psychological enrichment is expected to grow rapidly and to become more widespread. This overview explains human-robot interactions in terms of the relationship between humans and robots, in terms of the duration of these interactions and in terms of design issues affecting human interactive robots for psychological enrichment. Then examples of cross-cultural differences in the subjective evaluation of human interactive robots are described.

Actually, the research on human interactive robot (HIR) has been a topic of both science fiction and academic speculation for a long time. The origin of HIR as a discrete issue was stated by 20th century author Isaac Asimov in 1941, in his novel “I, Robot”. He stated the Three Laws of Robotics1 as, “ a. A robot may not injure a human being or, through inaction, allow a human being to come to harm. b. A robot must obey any orders given to it by human beings, except where such orders would conflict with the First Law. c. A robot must protect its own existence as long as such protection does not conflict with the First or Second Law. ” The three laws of robotics determine the idea of safe interaction which constitutes the basic rules of HIR. With the advances of artificial intelligence (AI), the HIR could eventually have more proactive behaviours, planning their motion in complex unknown environments. Nowadays, HIR are artificial agents with capacities of perception and action in the human’s environment. Their use has been tended to be found in the most technologically advanced societies in critical domains as search and rescue, military battle, law enforcement, entertainment, hospital care, etc. These domains of applications imply a closer interaction with human. The concept of closeness is to be taken in its full meaning, HIR and humans not only share the workspace but also share goals in terms of task achievement. The HIR has to adapt itself to human’s way of expressing desires and fulfill its task. Taking lifting up human in elder care for example, the human interactive robot RI-MAN, designed by the RIKEN Bio-Mimetic Control Research Center, communicates with human by listening and speaking, which makes it understand the human will (Onish, Luo et al. 2007). To fulfil the task, it also estimates the attitude of human body in real-time by tactile sense (Mukai, Onishi et al. 2008). This example contains two aspects of HIR, one is to understand the human mind and the other is to accomplish the manipulation. The former is based on AI techniques, like language comprehension, and the latter is relied on force control. On the other hand, human’s environments are much more complex. Thus, the HIR needs perceiving and understanding capacities to build dynamic models of its surroundings. It

As robots become increasingly prevalent and capable, the complexity of roles and responsibilities assigned to them as well as our expectations for them will increase in kind. For these autonomous systems to operate safely and efficiently in human-populated environments, they will need to cooperate and coordinate with human teammates. Mental models provide a formal mechanism for achieving fluent and effective teamwork during human–robot interaction by enabling awareness between teammates and allowing for coordinated action. Much recent research in human–robot interaction has made use of standardized and formalized mental modeling techniques to great effect, allowing for a wider breadth of scenarios in which a robotic agent can act as an effective and trustworthy teammate. This paper provides a structured overview of mental model theory and methodology as applied to human–robot teaming. Also discussed are evaluation methods and metrics for various aspects of mental modeling during human–robot interaction, as well as recent emerging applications and open challenges in the field.

Recognizing dynamic gestures from a continuous data stream has been treated as a difficult task. One challenge is the gesture spotting problem, or segmentation of a gesture pattern from a data stream containing consecutive gestures. Although various types of gesture spotting strategies have been introduced so far, all methods has its own limitations to be applied to real world situations such as sign language conversation. One of the conditions that must be satisfied for practical applications is to detect repetitive gestures for emphasizing a certain meaning.In the view of gesture spotting, repetitive gestures are hard to be detected because similar starting and ending moments are repeated without constraints of the number of repetitions. We introduce a data glove-based real-time dynamic gesture recognition system that covers both repetitive and nonrepetitive gestures. The proposed system consists of three steps: Gesture spotting, gesture sequence compression, and gesture recognition. For gesture spotting, we define a quantity termed as a gesture progress sequence, which quantifies a progress of a gesture using numbers from 0 to 1. Gesture sequence compression removes variations including the number of repetitions that seek to impart the same message. At the gesture recognition step, the compressed gesture patterns are classified. The proposed system was evaluated using 17 American sign language (ASL) gestures and some ASL sentences.

Recently, gesture recognition using visual sensors has been paid increased attention as an intelligent technology for human-computer interaction. To achieve good recognition performance in vision-based gesture recognition, it is important to find an efficient representation of complex visual signals and a robust classification method that can deal with diverse variations of gesture data. To treat these challenging topics, we propose a hierarchical structure for feature extraction and measuring similarity between two gestures obtained through RGB-D video sensors such as Kinect. The efficiency of the proposed method is confirmed through computational experiments on a public benchmark database.