Architecture design for performing grasp‐and‐lift tasks in brain–machine‐interface‐based human‐in‐the‐loop robotic system

Chapter Six - Human–robot interaction in teleoperation systems

Fully autonomous robot systems, though highly desired, face substantial theoretical and practical challenges when being deployed into a dynamic environment where humans co-exist. To tackle this challenge, this thesis investigates the concept of human-in-the-loop (HITL) systems, which incorporate human input to enhance robot functionality. HITL systems offer a pragmatic alternative, combining human versatility with robotic precision. This research aims to address critical questions in one particular HITL architecture which prioritizes the dominant role of the human within the system, positioning the robot primarily in an assistive capacity that adheres to human commands to facilitate the achievement of a shared goal. It explores two primary paradigms of shared assistance methods-Shared Control (SC) and Shared Autonomy (SA)-and discusses the system designs as well as specific algorithms to implement the three critical components in a HITL system: human intention estimation, modulation of human inputs and robot autonomy, and the human-robot communication channel. Due to the variety of use cases and their unique challenges, four distinct HITL systems are developed and analyzed to illustrate how shared assistance methods could be employed to assist human operators: an assistive wheelchair for indoor navigation, a human-centered robot system design for industrial tasks, a mobile bi-manual robot for tele-manipulation, and a virtual reality (VR)-based customizable shared control system for fine teleoperation. Although each system represents a comprehensive robotic solution, the research contributions for each work vary. In the assistive wheelchair navigation system, the focus was on human intent estimation via a low-throughput interface utilizing a recursive Bayesian filter, with significant efforts dedicated to developing a real-time user interface serving as the communication channel. In the human-robot collaboration system for industrial setting, the emphasis was on human state estimation through camera-based posture tracking and exploring the interplay between robot behavior and human ergonomics. For the two teleoperation systems, the primary focus was on the real-time modulation of human inputs and robot autonomy to aid in achieving dexterous manipulation tasks. A novel VR-based user interface was developed to enable users to customize the robot autonomy level. Each system was validated through a pilot study involving 10-20 human subjects, accompanied by extensive data analysis to provide insights into designing HITL systems for various applications. In conclusion, this thesis contributes to a deeper understanding of HITL systems, highlighting their potential to enhance human productivity, ergonomics, and quality of life in varied applications through concrete examples. The integration of human intent estimation and real-time shared control methods into robotic systems demonstrates the feasibility and benefits of HITL approaches. Our extensive experimental analysis underscores the critical role of human feedback in designing practical HITL systems that can be deployed in real-world scenarios. --Author's abstract

Towards establishing future-oriented relations between humans and robots, not only verbal, but also nonverbal robot functions have been studied. However, it is not clear how these different functions should be combined and implemented in robots in order to achieve ‘natural’ human-robot communication or interaction. In this paper, we focus on the timing for the coordination between utterance as a verbal function and body motion as a nonverbal function. To begin with, we investigated the temporal order and correlation between utterance and gesture in an interpersonal (human-human) interaction scenario as a preliminary experiment (Experiment 1). Then, we controlled the robot's response timing based on findings of the first experiment and evaluated the effectiveness of the robot's timing control in human-robot interaction (Experiment 2). It was shown that elderly participants significantly preferred the controlled robot's response. In addition, the timing of utterance and gestures affected the impression of the robot and interpretation of the verbal message. These results are important findings for designing socially interactive robots for the aging society.

Human performance plays an important role in system performance and is influenced by a wide set of internal, organizational, and environmental factors. Human–robot systems are being developed for a large number of domains, including in-home care, military deployments, emergency response, industrial robots, and therapeutic applications. Human performance modeling provides the opportunity to investigate multiple system prototypes and theories regarding human performance that can be further evaluated in human-in-the-loop experiments, to quickly adapt to rapidly changing robotic technology, and to represent human behavior in extreme conditions. Interacting with robots also influences human performance, and models of the human–robot system provide an avenue to investigate the impact on human performance that the human–robot teaming elicits. This chapter provides an overview of examples representing common methods for assessing human performance in human–robot systems. The topics are summarized in Table 3.2 and include a description of human performance modeling, a summary of commonly used human performance modeling tools, examples of human performance modeling with a focus on its application in human–robot systems, information regarding the validation of models, and guidelines for implementing human performance modeling techniques for robotics.

Rescue robotics has been suggested by a recent DARPA/NSF study as an application domain for the research in human-robot integration (HRI). This paper provides a short tutorial on how robots are currently used in urban search and rescue (USAR) and discusses the HRI issues encountered over the past eight years. A domain theory of the search activity is formulated. The domain theory consists of two parts: 1) a workflow model identifying the major tasks, actions, and roles in robot-assisted search (e.g., a workflow model) and 2) a general information flow model of how data from the robot is fused by various team members into information and knowledge. The information flow model also captures the types of situation awareness needed by each agent in the rescue robot system. The article presents a synopsis of the major HRI issues in reducing the number of humans it takes to control a robot, maintaining performance with geographically distributed teams with intermittent communications, and encouraging acceptance within the existing social structure.

Human–robot interaction (HRI), in all its different variants, has gained significant attention in industry. The combination of human skills and robotics significantly enhances manufacturing, improving operations quality and efficiency. This is particularly relevant in highly variable contexts like remanufacturing, which aims to preserve the value of products and components over time and involves complex, non-standardized tasks such as inspection, disassembly, reprocessing, and reassembly. Uncertainties related to the conditions of products to remanufacture cannot be managed by pure automation making HRI highly beneficial. Although researchers have focused on the use of HRI in several fields of application (from construction to logistics), no extensive mapping of the research trends has specifically addressed the intersection of HRI and remanufacturing contexts. To fill this gap, this paper presents a comprehensive study based on a bibliometric analysis to explore the current state of research on this topic. Based on 120 articles from the Scopus database, this study points out the main countries, authors, and affiliations involved in the field. In addition, the key topics were analyzed. A predominant focus, above all in the last three years, on Human-Robot Collaboration (HRC) among the different levels of HRI, and disassembly processes was highlighted, while other related activities, such as reassembly, have received comparatively less attention. An unexplored frontier is the integration of Artificial Intelligence and Machine Learning with HRC tasks in remanufacturing. These findings offer valuable insights for researchers, scholars, and industrial professionals aiming to advance HRI in remanufacturing.

The market for industrial robots grew rapidly during the 1970s and 1980s, with a peak demand in 1991. However, due to the subsequent recession in the world economy, the market for industrial robots has been slow or stagnant over the last decade. The value of industrial robots has decreased in that term, even though they have undergone considerable technical advances. On the other hand, human interactive robots for psychological enrichment are a type of service robots that provide a service by interacting with humans while stimulating their minds. As opposed to industrial robots, accuracy or speed is not always of prime importance. Their function or purpose is not simply entertainment, but also to render assistance, to guide, to provide therapy, to educate, to enable communication, and so on. The market for human interactive robots designed for psychological enrichment is expected to grow rapidly and to become more widespread. This overview explains human-robot interactions in terms of the relationship between humans and robots, in terms of the duration of these interactions and in terms of design issues affecting human interactive robots for psychological enrichment. Then examples of cross-cultural differences in the subjective evaluation of human interactive robots are described.

One of the challenges of outdoor robots is developing effective portable Human-Robot-Interaction (HRI) frameworks. Hand-held devices offer a practical solution. By equipping these devices with robot software, they can be made to interact with the outdoor robots. Android devices are ideal as they are open source and can be integrated with robots powered by the Robot Operating System (ROS), also open source. However, due to the limits of rosjava, the mechanism that links ROS with android, and the conflicting modes of operation between ROS and android, current implementations of ROS-android offer limited robot applications that do not support advanced operations such as autonomous navigation and others. This paper implements selective compartmentalization to overcome these limitations, by combining ROS with android through a number of ROS and android bridges that would facilitate the development of advanced robot applications. Through the proposed method, authors were able to develop a portable HRI framework that allowed human operators to supervise an outdoor mobile robot while it performed an autonomous task. From their mobile devices, users were able to initialize the robot, configure its motion, and monitor its progress. Also, users were able to reprogram the robot to perform new tasks (not previously planned) through a creative use of features offered in the developed HRI framework. Also, user cognitive effort was reported to be low as evident by the positive score on the NASA-TLX scale test which was corroborated with robot performance data. This paper presents the detailed development and implementation steps.

The electroencephalography (EEG) signal is an essential source of Brain-Computer Interface (BCI) technology implementation. The BCI is nothing but a non-muscle communication medium among the external devices and the brain. The basic concept of BCI is to enable the interaction among the neurological ill patients to others with the help of brain signals. EEG signal classification is an essential requirement for various applications such as motor imagery classification, drug effects diagnosis, emotion classification, seizure prediction/detection, eye state prediction/detection, and so on. Thus, there is a need for an efficient classification model that can deal with the EEG datasets more adequately with better classification accuracy, which will further help in developing the automatic solution for the medical domain. In this paper, we have introduced a hybrid classification model for eye state detection using electroencephalogram (EEG) signals. This hybrid classification model has been evaluated with the other traditional machine learning models, eight classification models (Prepossessed + Hypertuned) and six state-of-the-art methods to assess its appropriateness and correctness. This proposed classification model establishes a machine learning-based hybrid model for the classification of eye state using EEG signals with greater exactness. It is also capable of solving the issue of outlier detection and removal to address the class imbalance problem, which will offer the solution toward building the robotic or smart machine-based solution for social well-being.

To better prepare for the future society in which artificial intelligences (AI) will have much more pervasive influence on our lives, a better understanding of the difference between AI and human intelligence is necessary. Human and biological intelligence cannot be separated from the process of self-replication. Therefore, a fundamental gap exists between human intelligence and AI until AI acquires artificial life. Humans’ social and metacognitive intelligence most clearly distinguish human intelligence from nonhuman intelligence. Although advances are likely to improve the functioning of AI, AI will remain a function of human activity. However, if AI can learn to self-replicate and thus become a life form, albeit a man-made one, outcomes become uncertain.

As robots become increasingly prevalent and capable, the complexity of roles and responsibilities assigned to them as well as our expectations for them will increase in kind. For these autonomous systems to operate safely and efficiently in human-populated environments, they will need to cooperate and coordinate with human teammates. Mental models provide a formal mechanism for achieving fluent and effective teamwork during human–robot interaction by enabling awareness between teammates and allowing for coordinated action. Much recent research in human–robot interaction has made use of standardized and formalized mental modeling techniques to great effect, allowing for a wider breadth of scenarios in which a robotic agent can act as an effective and trustworthy teammate. This paper provides a structured overview of mental model theory and methodology as applied to human–robot teaming. Also discussed are evaluation methods and metrics for various aspects of mental modeling during human–robot interaction, as well as recent emerging applications and open challenges in the field.

Robots instead of humans work in unstructured environments, expanding the scope of human work. The interactions between humans and robots are indirect through operating terminals. The mental workloads of human increase with the lack of direct perception to the real scenes. Thus, mental workload assessment is important, which could effectively avoid serious accidents caused by mental overloading. In this paper, the operating object is a dual-arm robot. The classification of operator’s mental workload is studied by using the heart rate variability (HRV) signal. First, two kinds of electrocardiogram (ECG) signals are collected from six subjects who performed tasks or maintained a relaxed state. Then, HRV data is obtained from ECG signals and 20 kinds of HRV features are extracted. Last, six different classifications are used for mental workload classification. Using each subject’s HRV signal to train the model, the subject’s mental workload is classified. Average classification accuracy of 98.77% is obtained using the K-Nearest Neighbor (KNN) method. By using the HRV signal of five subjects for training and that of one subject for testing with the Gentle Boost (GB) method, the highest average classification accuracy (80.56%) is obtained. This study has implications for the analysis of HRV signals characteristic of mental workload in different subjects, which could improve operators’ well-being and safety in the human-robot interaction process.

In this paper, a network distributed monitoring system for human assistance robot system was developed to improve the interaction among the users and local service robotic system and enable a remote user to get a better understanding of what is going on in the local environment. Home integration robot system and network monitoring system using QuickCam Orbit cameras were developed and demonstrated from June 9 to June 19 at the 2005 World Exposition, Aichi, Japan. Improvements of network distributed monitoring system using IEEE1394 cameras with high performance and high resolution have been done in order to extend the application of system. Robot Technology Middleware (RTM) was used in the developed system. By using RTM, we can develop cameras functional elements as “RT software components” that can be implemented by different programming languages, run in different operating system, or connected in different networks to inter-operate. It is also easy to create comprehensive robot system application by re-using existing modules thus facilitating network-distributed software sharing and improving the cost of writing and maintaining software.

Actually, the research on human interactive robot (HIR) has been a topic of both science fiction and academic speculation for a long time. The origin of HIR as a discrete issue was stated by 20th century author Isaac Asimov in 1941, in his novel “I, Robot”. He stated the Three Laws of Robotics1 as, “ a. A robot may not injure a human being or, through inaction, allow a human being to come to harm. b. A robot must obey any orders given to it by human beings, except where such orders would conflict with the First Law. c. A robot must protect its own existence as long as such protection does not conflict with the First or Second Law. ” The three laws of robotics determine the idea of safe interaction which constitutes the basic rules of HIR. With the advances of artificial intelligence (AI), the HIR could eventually have more proactive behaviours, planning their motion in complex unknown environments. Nowadays, HIR are artificial agents with capacities of perception and action in the human’s environment. Their use has been tended to be found in the most technologically advanced societies in critical domains as search and rescue, military battle, law enforcement, entertainment, hospital care, etc. These domains of applications imply a closer interaction with human. The concept of closeness is to be taken in its full meaning, HIR and humans not only share the workspace but also share goals in terms of task achievement. The HIR has to adapt itself to human’s way of expressing desires and fulfill its task. Taking lifting up human in elder care for example, the human interactive robot RI-MAN, designed by the RIKEN Bio-Mimetic Control Research Center, communicates with human by listening and speaking, which makes it understand the human will (Onish, Luo et al. 2007). To fulfil the task, it also estimates the attitude of human body in real-time by tactile sense (Mukai, Onishi et al. 2008). This example contains two aspects of HIR, one is to understand the human mind and the other is to accomplish the manipulation. The former is based on AI techniques, like language comprehension, and the latter is relied on force control. On the other hand, human’s environments are much more complex. Thus, the HIR needs perceiving and understanding capacities to build dynamic models of its surroundings. It

Based on social exchange theory, this study investigates the effects of robots' fairness and social status on humans' reward-punishment behaviors and trust in human-robot interactions. In human-robot teamwork, robots show fair behaviors, dedication (altruistic unfair behaviors), and selfishness (self-interested unfair behaviors), but few studies have discussed the effects of these robots' behaviors on teamwork. This study adopts a 3 (the independent variable is the robot's fairness: self-interested unfair behaviors, fair behaviors, and altruistic unfair behaviors) × 3 (the moderator variable is the robot's social status: superior, peer, and subordinate) experimental design. Each participant and a robot completed the experimental task together through a computer. When robots have different social statuses, the more altruistic the fairness of the robot, the more reward behaviors, the fewer punishment behaviors, and the higher human-robot trust of humans. Robots' higher social status weakens the influence of their fairness on humans' punishment behaviors. Human-robot trust will increase humans' reward behaviors and decrease humans' punishment behaviors. Humans' reward-punishment behaviors will increase repaired human-robot trust. Robots' fairness has a significant impact on humans' reward-punishment behaviors and trust. Robots' social status moderates the effect of their fair behavior on humans' punishment behavior. There is an interaction between humans' reward-punishment behaviors and trust. The study can help to better understand the interaction mechanism of the human-robot team and can better serve the management and cooperation of the human-robot team by appropriately adjusting the robots' fairness and social status.