Applications in HHI: Physical Cooperation

Internet of Things (IoT) based voice interaction system, as a new artificial intelligence application, provides a new human–computer interaction mode. The more intelligent and efficient communication approach poses greater challenges to the semantic understanding module in the system. Facing with the complex and diverse interactive scenarios in practical applications, the academia and the industry urgently need more powerful Natural Language Understanding (NLU) methods as support. Intent Detection and Slot Filling joint task, as one of the core sub-tasks in NLU, has been widely used in different human–computer interaction scenarios. In the current era of deep learning, the joint task of Intent Detection and Slot Filling has also changed from previous rule-based methods to deep learning-based methods. It is an important problem to explore how to realize the models of these tasks to be refined and targeted designed, and to make the Intent Detection task better serve the improvement of precision of Slot Filling task by connecting the before and after tasks. It has great significance for building a more humanized IoT voice interaction system. In this study, we designed two joint models to realize Intent Detection and Slot Filling joint task. For the Intent Detection type task, one is based on BiGRU-Att-CapsuleNet (hybrid-based model) and the other is based on the RCNN model. Both methods use the BiGRU-CRF model for the Slot Filling type task. The hybrid-based model can enhance the semantic capture capability of a single model. And by combining specialized models built independently for each task to achieve a complete joint task, it can be better to achieve optimal performance on each task. This study also carried out detailed comparative experiments of tasks and joint tasks on multiple datasets. Experiments show that the joint models have achieved competitive results in 7 typical datasets included in multiple scenarios in English and Chinese compared with other models.

We envision a telepresence system that enhances remote work by facilitating both physical and immersive visual interactions between individuals. However, during robot teleoperation, communication often lacks realism, as users see the robot’s body rather than the remote individual. To address this, we propose a method for overlaying a digital human model onto a humanoid robot using XR visualization, enabling an immersive 3D telepresence experience. Our approach employs a learning-based method to estimate the 2D poses of the humanoid robot from head-worn stereo views, leveraging a newly collected dataset of full-body poses for humanoid robots. The stereo 2D poses and sparse inertial measurements from the remote operator are optimized to compute 3D poses over time. The digital human is localized from the perspective of a continuously moving observer, utilizing the estimated 3D pose of the humanoid robot. Our moving camera-based pose estimation method does not rely on any markers or external knowledge of the robot’s status, effectively overcoming challenges such as marker occlusion, calibration issues, and dependencies on headset tracking errors. We demonstrate the system in a remote physical training scenario, achieving real-time performance at 40 fps, which enables simultaneous immersive and physical interactions. Experimental results show that our learning-based 3D pose estimation method, which operates without prior knowledge of the robot, significantly outperforms alternative approaches requiring the robot’s global pose, particularly during rapid headset movements, achieving markerless digital human augmentation from head-worn views.

A common approach when simulating face-to-face interpersonal scenarios with virtual humans is to afford users only verbal interaction while providing rich verbal and non-verbal interaction from the virtual human. This is due to the difficulty in providing robust recognition of user non-verbal behavior and interpretation of these behaviors within the context of the verbal interaction between user and virtual human. To afford robust hand and tool-based non-verbal interaction with life-sized virtual humans, we propose virtual multi-tools. A single hand-held, tracked interaction device acts as a surrogate for the virtual multi-tools: the user's hand, multiple tools, and other objects. By combining six degree-of-freedom, high update rate tracking with extra degrees of freedom provided by buttons and triggers, a commodity device, the Nintendo Wii Remote, provides the kinesthetic and haptic feedback necessary to provide a high-fidelity estimation of the natural, unencumbered interaction provided by one's hands and physical hand-held tools. These qualities allow virtual multi-tools to be a less error-prone interface to social and task-oriented non-verbal interaction with a life-sized virtual human. This paper discusses the implementation of virtual multi-tools for hand and tool-based interaction with life-sized virtual humans, and provides an initial evaluation of the usability of virtual multi-tools in the medical education scenario of conducting a neurological exam of a virtual human.

This paper presents an approach for robot leading in physical human-robot interaction during dance training and support with a mobile humanoid robot. In this highly coupled physical interaction, we convey the direction of motion by varying the center of mass (COM) elevation. This method increases the human partner's dependency on the robot for stability, allowing the robot to guide the interaction. To achieve a human-like leading, we based the robot control on an impedance behavior while following a desired trajectory. In experimental tests, the COM control improved the understanding of motion direction from 70% to 90%. Moreover, it reduced the interaction forces by 10N (on average) during the guidance. We conclude that the COM control enhances the synchronization of the moving human-robot couple, and encourages humanlike behavior of the humanoid robot. Consequently, the human partner can better understand the humanoid robot, especially when the robot takes the leading role.

It is known that physical coupling between two subjects may be advantageous in joint tasks. However, little is known about how two people mutually exchange information to exploit the coupling. Therefore, we adopted a reversed, novel perspective to the standard one that focuses on the ability of physically coupled subjects to adapt to cooperative contexts that require negotiating a common plan: we investigated how training in pairs on a novel task affects the development of motor skills of each of the interacting partners. The task involved reaching movements in an unstable dynamic environment using a bilateral non-linear elastic tool that could be used bimanually or dyadically. The main result is that training with an expert leads to the greatest performance in the joint task. However, the performance in the individual test is strongly affected by the initial skill level of the partner. Moreover, practicing with a peer rather than an expert appears to be more advantageous for a naive; and motor skills can be transferred to a bimanual context, after training with an expert, only if the non-expert subject had prior experience of the dynamics of the novel task.

Learning new motor tasks from physical interactions is an important goal for both robotics and machine learning. However, when moving beyond basic skills, most monolithic machine learning approaches fail to scale. For more complex skills, methods that are tailored for the domain of skill learning are needed. In this paper, we take the task of learning table tennis as an example and present a new framework that allows a robot to learn cooperative table tennis from physical interaction with a human. The robot first learns a set of elementary table tennis hitting movements from a human table tennis teacher by kinesthetic teach-in, which is compiled into a set of motor primitives represented by dynamical systems. The robot subsequently generalizes these movements to a wider range of situations using our mixture of motor primitives approach. The resulting policy enables the robot to select appropriate motor primitives as well as to generalize between them. Finally, the robot plays with a human table tennis partner and learns online to improve its behavior. We show that the resulting setup is capable of playing table tennis using an anthropomorphic robot arm.

The modalities of communication – speech, song, writing, images, etc. – constitute general registers of human expression and experience. As such, they lend themselves to different forms of technological mediation, both individually and in combination as multimodal communication.

The next generation of robot companions or robot working partners will need to satisfy social requirements somehow similar to the famous laws of robotics envisaged by Isaac Asimov time ago (Asimov, 1942). The necessary technology has almost reached the required level, including sensors and actuators, but the cognitive organization is still in its infancy and is only partially supported by the current understanding of brain cognitive processes. The brain of symbiotic robots will certainly not be a “positronic” replica of the human brain: probably, the greatest part of it will be a set of interacting computational processes running in the cloud. In this article, we review the challenges that must be met in the design of a set of interacting computational processes as building blocks of a cognitive architecture that may give symbiotic capabilities to collaborative robots of the next decades: (1) an animated body-schema; (2) an imitation machinery; (3) a motor intentions machinery; (4) a set of physical interaction mechanisms; and (5) a shared memory system for incremental symbiotic development. We would like to stress that our approach is totally un-hierarchical: the five building blocks of the shared cognitive architecture are fully bi-directionally connected. For example, imitation and intentional processes require the “services” of the animated body schema which, on the other hand, can run its simulations if appropriately prompted by imitation and/or intention, with or without physical interaction. Successful experiences can leave a trace in the shared memory system and chunks of memory fragment may compete to participate to novel cooperative actions. And so on and so forth. At the heart of the system is lifelong training and learning but, different from the conventional learning paradigms in neural networks, where learning is somehow passively imposed by an external agent, in symbiotic robots there is an element of free choice of what is worth learning, driven by the interaction between the robot and the human partner. The proposed set of building blocks is certainly a rough approximation of what is needed by symbiotic robots but we believe it is a useful starting point for building a computational framework.

Many human activities are performed in groups and require that the individuals in the group coordinate their actions. For example, carrying bulky objects, dancing, handshaking, musicians playing together in an orchestra are examples of joint actions. A crucial aspect of joint action is that it requires that the partners share information and communicate to update the information in order to be able to coordinate their actions. This Chapter focuses on emergent coordination, implicit communication in human-robot cooperative actions, which rely on subtle cues about the body and the movement of partners. In particular, this Chapter describes research that analyzes whether and under which conditions a robotic device can trigger this form of covert communication with a human partner. It also focuses on physical interaction with robots, a crucial form of interaction in many types of joint action. It presents work done to develop novel compliant actuation technologies that aim at facilitating physical interaction between a robot and a person and it illustrates the importance of being able to read the state of the partners in the context of robot-assisted rehabilitation of the upper-limb. A characteristic of the research on human-robot interaction presented in this chapter is to be closely inspired by our current understanding of the human sensory, motor and cognitive systems. As a matter of fact, a deep understanding of humans’ body and mind appears crucial to develop machines and robots, whether they have a humanoid appearance or not, that can interact closely and cooperate with humans.

Physical interaction between man and machines is increasing the interest of the research as well as the industrial community. It is known that physical coupling between active persons can be beneficial and increase the performance of the dyad compared to an individual. However, the factors that may result in performance benefits are still poorly understood. The aim of this work is to investigate how the different initial skill levels of the interacting partners influence the learning of a stabilization task. Twelve subjects, divided in two groups, trained in couples in a joint stabilization task. In the first group the couples were composed of two naive, while in the second a naive was trained together with an expert. Results show that training with an expert results in the greatest performance in the joint task. However, this benefit is not transferred to the individual when performing the same task bimanually.

BackgroundTheory of Mind (ToM) refers to the ability to represent one's own and others' mental states, and emotion understanding involves appropriately comprehending and responding to others' emotional cues in social interactions. Individual differences in mind and emotion understanding have been associated strongly with verbal ability and interaction and, as such, existing training for children's ToM and emotion understanding is mostly language-based. Building on the literature on embodied cognition, this study proposes that mind and emotion understanding could be facilitated by one's visuospatial experience in simulating other's frames of reference.MethodsThis protocol consists of two training studies. Study 1 will examine if visuospatial perspective-taking training promotes ToM and emotion understanding. Participants will consist of 96 4.5-year-olds and will be randomly assigned to one of two training groups: the altercentric block building group (trained to be visuospatial perspective-takers), or the egocentric block building group (no visuospatial perspective-taking is involved). Study 2 will compare the engagement of visuospatial perspective-taking and verbal interaction in the development of mind and emotion understanding. Participants will consist of 120 4.5-year-olds. They will be randomly assigned to one of three training groups: the socialized altercentric block building (both visuospatial perspective-taking and verbal interaction), the parallel altercentric block building (visuospatial perspective-taking only), or the paired dialogic reading (verbal interaction only).ConclusionsIn terms of theoretical implications, the potential causal relationship between visuospatial perspective-taking and ToM and emotion understanding may shed new insights on what underlies the development of mental state understanding. The findings of this study also have practical implications: researchers and educators may popularize visuospatial perspective-taking training in the form of block-building games if it is found to be effective in complementing conventional language-based theory-of-mind training.

In this overview paper we present current work on safety analysis for physical Human-Robot Interaction (pHRI) and motion control methods for robotic co-workers. In particular, we introduce the analysis tools for investigating the potential injury a human would suffer during robot-human impacts. Furthermore, we outline our concept for establishing a procedure towards standardized crash testing in robotics with automobile crash-test dummies. Since it is only possible to investigate blunt impacts with these devices, we developed a drop testing setup for analyzing soft-tissue injury in robotics from a biomechanics perspective. In the second part of the paper, some of our methods for task preserving and task relaxing motion schemes are described, which enable collision avoidance in real-time. The algorithms are well suited to work in an integrated fashion with the soft robotics control developed for the DLR Lightweight Robot III (LWR-III). In addition, it is shown how the torque sensing capabilities of the robot can be used to support reactive motion schemes. Finally, an overview of our human-friendly control architecture for the LWR-III is given, which unifies the rich bundle of developed methods for this manipulator.

A real-time human-robot interaction framework with robust background invariant hand gesture detection

The aim of this study is to contribute to a better understanding of the nuances of multimodal communication through which educational designers construct shared meanings and ideas. Educational design is a broad and multi-faceted area. Meaning-making in face-to-face educational design meetings is a complex process but is yet to gain attention as a topic of research. Most educational design projects involve various stakeholders, including professional designers and people who offer distinct expertise according to the needs of the project. Effective communication is imperative in such environments. We need a thorough understanding of how educational designers work together before we theorise their practice and attempt to create prescriptive tools and process models. The analysis of the meaning-making process reported in this paper shows that educational design is accomplished through intricate verbal and non-verbal interaction with various tools, resources, and representations. In particular, the results show how educational designers reconstructed pre-created visual design artefacts through drawing inscriptions over them – blending digital and non-digital artefacts. In addition, this study shows that gestures are indispensable to the meaning-making process in educational design team meetings. It is important to build a body of knowledge in this area that supports the practice and research in educational design. This paper would be of primary interest to researchers who study the way in which educational design is accomplished. Beyond this, those who work as educational designers or those who work with educational designers, might benefit from an awareness of the different interactional tools at their disposal.

It has been postulated that when people engage in joint actions they form internal representations not only of their part of the joint task but of their co-actors' parts of the task as well. However, empirical evidence for this claim is scarce. By means of high-density electroencephalography, this study investigated whether one represents and simulates the action of an interaction partner when planning to perform a joint action. The results showed that joint action planning compared with individual action planning resulted in amplitude modulations of the frontal P3a and parietal P3b event-related potentials, which are associated with stimulus classification, updating of representations, and decision-making. Moreover, there was evidence for anticipatory motor simulation of the partner's action in the amplitude and peak latency of the late, motor part of the Contingent Negative Variation, which was correlated with joint action performance. Our results provide evidence that when people engage in joint tasks, they represent in advance each other's actions in order to facilitate coordination.