Multimodal Behavior Analysis of Human-Robot Navigational Commands

Human communication is multimodal in nature. In a normal environment, people use to interact with other humans and with the environment using more than one modality or medium of communication. They speak, use gestures and look at things to interact with nature and other humans. By listening to the different voice tones, looking at face gazes, and arm movements people understand communication cues. A discussion with two people will be in vocal communication, hand gestures, head gestures, and facial cues, etc. [1]. If textbook definition is considered synergistic use of these interaction methods is known as multimodal interaction [2]. For example, , a wheelchair user might instruct the smart wheelchair or the assistant to go forward, as shown in Fig. 1(a). However, with a hand gesture shown in the figure, he or she might want to go slowly. In the same way as of Fig. 1(b), a person might give someone a direction with a vocal command ‘that way’ and gesture the direction with his or her hand. In most Human-Robot Interaction (HRI) developments, there is an assumption that human interactions are unimodal. This forces the researchers to ignore the information other modalities carry with them. Therefore, it would provide an additional dimension for interpretation of human robot interactions. This article provides a concise description of how to adapt the concept of multimodal interaction in human-robot applications.

With rapid developments in biometric recognition, a great deal of attention is being paid to robots which interact smartly with humans and communicate certain types of biometrical information. Such human–machine interaction (HMI), also well-known as human–robot interaction (HRI), will, in the future, prove an important development when it comes to automotive manufacturing applications. Currently, hand gesture recognition-based HRI designs are being practically used in various areas of automotive manufacturing, assembly lines, supply chains, and collaborative inspection. However, very few studies are focused on material-handling robot interactions combined with hand gesture communication of the operator. The current work develops a depth sensor-based dynamic hand gesture recognition scheme for continuous-time operations with material-handling robots. The proposed approach properly employs the Kinect depth sensor to extract features of Hu moment invariants from depth data, through which feature-based template match hand gesture recognition is developed. In order to construct continuous-time robot operations using dynamic hand gestures with concatenations of a series of hand gesture actions, the wake-up reminder scheme using fingertip detection calculations is established to accurately denote the starting, ending, and switching timestamps of a series of gesture actions. To be able to perform typical template match on continuous-time dynamic hand gesture recognition with the ability of real-time recognition, representative frame estimates using centroid, middle, and middle-region voting approaches are also presented and combined with template match computations. Experimental results show that, in certain continuous-time periods, the proposed complete hand gesture recognition framework can provide a smooth operation for the material-handling robot when compared with robots controlled using only extractions of full frames; presented representative frames estimated by middle-region voting will maintain fast computations and still reach the competitive recognition accuracy of 90.8%. The method proposed in this study can facilitate the smart assembly line and human–robot collaborations in automotive manufacturing.

To naturally interact with virtual environment by hand gesture, this paper presents a robust RGB-D data based recognition method of static and dynamic hand gesture. Firstly, for static hand gesture recognition, starting from the hand gesture contour extraction, the palm center is identified by Distance Transform (DT) algorithm. The fingertips are localized by employing the K-Curvature-Convex Defects Detection algorithm (K-CCD). On the basis, the distances of the pixels on hand gesture contour to palm center and the angle between fingertips are considered as the auxiliary features to construct a multimodal feature vector, and then recognition algorithm is presented to robustly recognize the static hand gestures. Secondly, combining Euclidean distance between hand joints and shoulder center joint with the modulus ratios of skeleton features, this paper generates a unifying feature descriptor for each dynamic hand gesture and proposes an improved dynamic time warping (IDTW) algorithm to obtain recognition results of dynamic hand gestures. Finally, we conduct extensive experiments to test and verify the static and dynamic hand gesture recognition algorithm and realize a low-cost real-time application of natural interaction with virtual environment by hand gestures.

MuViH: Multi-View Hand gesture dataset and recognition pipeline for human–robot interaction in a collaborative robotic finishing platform

Recently, human - machine interaction (HMI) becomes a hot research topic because of its wide applications, ranging from automatic device control to designing and development of assistant robot or even smart building at sparser scale. One of the most important questions in this research field is finding out an efficient and natural method of HMI. Among several channels of communication, hand gestures have been shown to be an intuitive and efficient mean to express an idea or to control something. For a successful hand gesture based interaction between human and robot, a vocabulary of hand gestures needs to be defined. To resolve this problem, in this paper, we propose a framework to study the behavior of Vietnamese in using of hand gesture in communication with robot. This study allows designing a hand gesture vocabulary for human - robot interaction (HRI) applications. In the literature, there are no works similar to ours. This makes our twofold contributions: (1) a proposed framework for hand gesture recognition: hand gesture vocabulary design, feature extraction for hand posture representation, hand posture classification, and hand gesture database construction; (2) some experimentations are realized to evaluate the defined hand gesture set that can be used in general situation of HRI. The experiment results show that the defined hand gesture set satisfies the both criteria: intuitiveness and recognisability.

In everyday live head gestures such as head shaking or nodding and hand gestures like pointing gestures form important aspects of human-human interaction. Therefore, recent research considers integrating these intuitive communication cues into technical systems for improving and easing human-computer interaction. In this paper we present a vision-based system to recognize head gestures (nodding, shaking, neutral) and dynamic hand gestures (hand moving right/left/up/down, fist moving right/left) in real-time. The gestural input delivers a communication modality for a human-robot interaction scenario situated in an assistive household environment. The use of fast low-level image-feature extraction methods contributes to the real-time capability of the system and advanced classification approaches relying on Graphical Models provide high robustness. Graphical Models offer the possibility to group the input features in several sub-nodes resulting in a better classification than obtained via a traditional Hidden Markov Model classification. The applied grouping can regard interdependencies owing to, either physical constraints (like for the head gestures), or interrelations between shape and motion (like for the hand gestures).

In this paper, we tackle advantages of cyclical movement patterns of hand gestures. The cyclical patterns are defined as closed-form which hand moves away from a rest position, follows one or more of a series of the movement of hand shapes and returns to its rest position. Due to the cyclical pattern characteristic, phase of gestures are supportive cues for deploying robust recognition schemes. We conduct a spatial-temporal representation of the hand gestures which takes into account both hand shapes and its movements during a gesture. The phase alignment then is deployed in the conducted space. The proposed scheme ensures inter-period phase continuity as well as normalizes length of the hand gestures. Three different datasets of dynamic hand gestures consisting of non-cyclical and cyclical patterns are examined. Evaluation results confirm that the best accuracy rate achieves at 96% for cyclical pattern that is significantly higher than results for typical gestures. The proposed method suggests a feasible and robust solution addressing technical issues in developing human-computer interaction applications such as using hand gestures to control home appliance devices.

This paper presents a hand gesture interaction system in virtual environment, in which mixed means static and dynamic hand gestures are combined for both navigation and object manipulation. Firstly, a simple average background model and skin color are used for hand area segmentation. Then a state-based spotting algorithm is employed to automatically identify two types of hand gestures. A voting-based method is used for quick classification of static gestures. And we use the hidden Markov model (HMM) to recognize dynamic gestures. Since the training of HMM requires the consistency of the training data, outputted by the feature extraction, a data aligning algorithm is raised. Through our hand gesture system, users can perform complicated operating commands in a natural way. The experimental results demonstrate that our methods are effective and accurate.

Multimodal Interfaces that enable natural means of interaction using multiple modalities such as touch, hand gestures, speech, and facial expressions represent a paradigm shift in human-computer interfaces. Their aim is to allow rich and intuitive multimodal interaction similar to human-to-human communication and interaction. From the multimodal system's perspective, apart from the various input modalities themselves, context information such as states of attention and activity, and identities of interacting users can help greatly in improving the interaction experience. For example, when sensors such as cameras (webcams, depth sensors etc.) and microphones are always on and continuously capturing signals in their environment, context information is very useful to distinguish genuine system-directed activity from ambient speech and gesture activity in the surroundings, and distinguish the active user from among a set of users. Information about identity may be used to personalize the system's interface and behavior -- e.g. the look of the GUI, modality recognition profiles, and information layout -- to suit the specific user. In this paper, we present a set of algorithms and an architecture that performs audiovisual analysis of context using sensors such as cameras and microphone arrays, and integrates components for lip activity and audio direction detection (speech activity), face detection and tracking (attention), and face recognition (identity). The proposed architecture allows the component data flows to be managed and fused with low latency, low memory footprint, and low CPU load, since such a system is typically required to run continuously in the background and report events of attention, activity, and identity, in real-time, to consuming applications.

Human–Robot Interaction (HRI) using hand gesture recognition offers an effective and non-contact approach to enhancing operational intuitiveness and user convenience. However, most existing studies primarily focus on either static sign language recognition or the tracking of hand position and orientation in space. These approaches often prove inadequate for controlling complex robotic systems. This paper proposes an advanced HRI system leveraging dynamic hand gestures for controlling quadruped robots equipped with a robotic arm. The proposed system integrates both semantic and pose information from dynamic gestures to enable comprehensive control over the robot’s diverse functionalities. First, a Depth–MediaPipe framework is introduced to facilitate the precise three-dimensional (3D) coordinate extraction of 21 hand bone keypoints. Subsequently, a Semantic-Pose to Motion (SPM) model is developed to analyze and interpret both the pose and semantic aspects of hand gestures. This model translates the extracted 3D coordinate data into corresponding mechanical actions in real-time, encompassing quadruped robot locomotion, robotic arm end-effector tracking, and semantic-based command switching. Extensive real-world experiments demonstrate the proposed system’s effectiveness in achieving real-time interaction and precise control, underscoring its potential for enhancing the usability of complex robotic platforms.

We have successfully designed, developed and deployed a pressure sensing floor system with a higher frame rate, less latency, high sensor resolution, large sensing area that can provide us with real time data about the location and amount of pressure exerted on the floor. The floor has been integrated and synchronized with the marker based motion capture system to create a smart environment for movement based human computer interaction. Future direction shall be towards extending the context aware capabilities of the floor system. An algorithm that can clearly distinguish between the left foot and the right foot shall find extensive usage of the floor to numerous applications. Shape descriptors such as Fourier and Hu moments to distinguish left foot and right foot on the basis of shape come in handy for such an analysis. Such intelligence to the floor to recognize and distinguish the left and right foot shall pave way for recognizing gestures with varying foot contact. The above work may be further extended to make a distinction of the heel and toe of a particular foot as well. This shall find extensive use in diagnosis of various gait pathologies as most disorders are reflected by abnormal pressure patterns localized to either the toe or heel. AME Floor-III as it stands now is not portable. Further work is also being done in the design of the interfacing hardware to make a portable system. The creation of a wireless pressure sensing system is a possible alternative towards a portable system. Integration of other sensing modalities such as audio-based sensing (microphone arrays), a wireless EMG system into the existing multimodal framework to create a very powerful tool for movement based human computer interaction is another major challenge in the future. The establishment of statistical models and machine learning techniques to model the

The research in the dynamic hand gestures recognition field has been promoted thanks to the recent success of the emerging Leap Motion controller (LMC) sensor, especially in the human-computer interaction field. During surgery, surgeons need to make more effective use of computers through a touchless human-computer interaction. In this paper, we present a set of gestures that are about commands to manipulate medical images. The public availability of hand gesture datasets for dynamic hand gestures, in particular with the LMC as an acquisition device, is currently quite poor. Therefore, we collect a more challenging dataset, called “LeapGestureDB”. The dataset is presented as text files, which contains 6,600 samples of 11 different dynamic hand gestures performed by over 120 participants. It is hosted online for the public interest, and especially to enhance research results. We suggest an approach of three-dimensional dynamic hand gesture recognition. We extract spatial, frequential and spatio-frequential features from the three-dimensional positions of the fingertips and the palm center delivered from the input sensor. These descriptors are fed into a support vector machine classifier to determine the performed gestures. We first evaluate our work on an existing dataset, namely “RIT dataset”, and then we use our collected dataset. We achieved a better accuracy rate of 93% comparing to results obtained from the “RIT dataset”.

The use of physical devices like mouse and keyboard to communicate to our computer has hindered the natural interface between the computer and its user. To eradicate this barrier, we have designed a virtual mouse which will act upon the various hand gestures the user provides and will help them interact with the computer in a better way. The virtual mouse control of our system will keep a track of the movements of the fingers and palm, thus providing us with a specific output based on the recognized action of our hand. The proposed model will recognize both static as well as dynamic hand gestures and the results will show that the model that has been proposed and built will intuitively interact with the computer and it can be achieved the minimum usage of any hardware requirements. Our model will also have a major advantage over the current existing models, which is, it also has the feature of voice command recognition, meaning that even if the user if not able to provide appropriate gestures using their hand due to some medical condition, they will still be able to interact with the computer via voice commands.

In this digital information explosion era, more and more products are proposed to enhance the quality and convenience of our life. However, vision-based hand gesture recognition is still a challenging problem to overcome. In this paper, an architecture system was proposed with dual-camera to construct the depth map and recognize dynamic hand gesture. It contains one static and two dynamic hand gestures. This design is implemented in SMIMS development board to verify and demonstrate the whole system.

People use gestures to communitcate with other people without difficutlty. As a method of computer/human interaction, gestures are more intuitive and more direct than existing input devices, such as keyboards, mice, etc., and futhermore, are more functional. Using hand gestures, which are important human gestures, is a key technique to build a friendly user interface.Some gestures are considered to be simbols, such as "OK" gesture, American sign language. Others have numerical meaning, i.e. how long, how large, how fast. etc. To deal with both kinds of hand gestures, we propose a hand shape representation method.The method represents a hand shape with 15 dimensional data of 5 finger tip positions in 3D. To implement both static and dynamic hand gestures in a graphic object manipulation system, at each measurement time the current hand shape is classified into 10 classes of hand shapes using subspace method and the hand movement is evaluated using recurrnet neural network.The system allows a user to manipulate graphic objects intuitively and directly.