Implementation of Engagement Detection for Human-Robot Interaction in Complex Environments.

The lightweight human-robot interaction model with high real-time, high accuracy, and strong anti-interference capability can be better applied to future lunar surface exploration and construction work. Based on the feature information inputted from the monocular camera, the signal acquisition and processing fusion of the astronaut gesture and eye-movement modal interaction can be performed. Compared with the single mode, the human-robot interaction model of bimodal collaboration can achieve the issuance of complex interactive commands more efficiently. The optimization of the target detection model is executed by inserting attention into YOLOv4 and filtering image motion blur. The central coordinates of pupils are identified by the neural network to realize the human-robot interaction in the eye movement mode. The fusion between the astronaut gesture signal and eye movement signal is performed at the end of the collaborative model to achieve complex command interactions based on a lightweight model. The dataset used in the network training is enhanced and extended to simulate the realistic lunar space interaction environment. The human-robot interaction effects of complex commands in the single mode are compared with those of complex commands in the bimodal collaboration. The experimental results show that the concatenated interaction model of the astronaut gesture and eye movement signals can excavate the bimodal interaction signal better, discriminate the complex interaction commands more quickly, and has stronger signal anti-interference capability based on its stronger feature information mining ability. Compared with the command interaction realized by using the single gesture modal signal and the single eye movement modal signal, the interaction model of bimodal collaboration is shorter about 79% to 91% of the time under the single mode interaction. Regardless of the influence of any image interference item, the overall judgment accuracy of the proposed model can be maintained at about 83% to 97%. The effectiveness of the proposed method is verified.

In this paper, we deal with the human-robot interaction control problem. Levels of actuation of the user are considered in the human-robot interaction model from a stochastic point of view. It is given in terms of a Markovian approach. Electromyographic signals are used to compute jump parameters between different levels of interaction. In this way, human neuromuscular system defines the behavior of the Markov chain. A unified approach composed by robust Kalman filter and robust regulator for discrete-time Markovian jump linear systems is proposed. Also, a serious game is used to generate visual feedback and promote the active participation of the user. Experimental results show high accuracy in the Markovian compliance control for a robotic platform applied in ankle rehabilitation.

The integration of Natural Language Processing (NLP) in Human-Robot Interaction (HRI) represents a significant advancement towards achieving more natural and effective communication between humans and robots. This research explores the application of state-of-the-art NLP techniques to enhance HRI, focusing on improving robots' abilities to understand and generate human language. Key components of our approach include advanced speech recognition, natural language understanding (NLU), dialogue management, and natural language generation (NLG). We designed and implemented an HRI system that leverages models such as BERT for language understanding and GPT-3 for generating contextually appropriate responses. Our methodology involves integrating these NLP models with a robotics platform, ensuring real-time interaction capabilities while maintaining a high level of accuracy and context awareness. The system was evaluated through a series of user studies, measuring performance metrics such as accuracy, latency, and user satisfaction. Results indicate that our NLP-enhanced HRI system significantly improves the quality of interactions, demonstrating superior understanding and responsiveness compared to traditional systems. This paper discusses the implementation challenges, including computational constraints and ambiguity resolution, and provides insights into user feedback and system performance. Future work will focus on enhancing context management, exploring multimodal interaction, and addressing ethical considerations in deploying advanced HRI systems. Our findings underscore the potential of NLP to transform human-robot communication, paving the way for more intuitive and effective robotic assistants in various domains. Keywords: Human-Robot Interaction (HRI), Natural Language Processing (NLP), Conversational AI, Speech Recognition, Natural Language Understanding (NLU), Natural Language Generation (NLG), Multimodal Interaction, Dialogue Systems, Context Awareness, Emotion Recognition, Machine Learning in HRI, Personalized Interaction, User Experience (UX) in HRI, Human-Centered Design, Collaborative Robots (Cobots)

In this paper a novel concept of human-robot interaction (HRI) modeling is proposed. Including factors like trust in automation, situational awareness, expertise and expectations a new user experience framework is formed for industrial robots. Service Oriented Robot Operation, proposed in a previous paper, creates an abstract level in HRI and it is also included in the framework. This concept is evaluated with exhaustive tests. Results prove that significant improvement in task execution may be achieved and the new system is more usable for operators with less experience with robotics; personnel specific for small and medium enterprises (SMEs).

Millions have adopted tools like ChatGPT in recent years, yet indifference and resistance among employees remain. This qualitative study employs monodramatic projective techniques to explore employees' hidden assumptions and unconscious beliefs in a division attempting to integrate Generative Artificial Intelligence (AI, GAI). Through pretensive work, soliloquy, symbolic representation, modeling with intermediate objects, concretization, and role reversal techniques, the interviewees' internal representations of GAI and trust were materialized in physical artifacts, such as a ball of straw or a potted plant. The study identified three principal themes: GAI's appearance as a Janus-faced presence, unmet performance promises, and avoided proximity. Findings highlight ambiguities in acceptance and show that adoption was driven more by industry hype and normative pressures than genuine organizational needs, leading to disorganized implementation dependent on individual employee characteristics, mistrust, and disenchantment. The study's main contribution lies in refining human-robot interaction (HRI) models and psychodrama methods for GAI, emphasizing the significance of physicality and embodiment in technology-mediated relationships, identifying trust as a complex phenomenon with potential reciprocal causation, and emphasizing the importance of affective attitudes, illustrating how adoption projects can falter despite cognitive openness – all insights crucial for understanding self-driven, bottom-up GAI adaptation in an organizational context.

Enhancing Humans Trust in Robots through Explanations

Robots are being used in the operating room to aid in surgery, prompting changes to workflow and adaptive behavior by the users. This case study presents a methodology for examining human-robot team interaction in a complex environment, along with the results of its application in a study of the effects of experience and workplace culture, for human-robot team interaction in the operating room. The analysis of verbal and non-verbal events in robotic surgery in two different surgical teams (one in the US and one in France) revealed differences in workflow, timeline, roles, and communication patterns as a function of experience and workplace culture. Longer preparation times and more verbal exchanges related to uncertainty in use of the robotic equipment were found for the French team, who also happened to be less experienced. This study offers an effective method for studying human-robot team interaction and has implications for the future design and training of teamwork with robotic systems in other complex work environments.

One of the challenges of outdoor robots is developing effective portable Human-Robot-Interaction (HRI) frameworks. Hand-held devices offer a practical solution. By equipping these devices with robot software, they can be made to interact with the outdoor robots. Android devices are ideal as they are open source and can be integrated with robots powered by the Robot Operating System (ROS), also open source. However, due to the limits of rosjava, the mechanism that links ROS with android, and the conflicting modes of operation between ROS and android, current implementations of ROS-android offer limited robot applications that do not support advanced operations such as autonomous navigation and others. This paper implements selective compartmentalization to overcome these limitations, by combining ROS with android through a number of ROS and android bridges that would facilitate the development of advanced robot applications. Through the proposed method, authors were able to develop a portable HRI framework that allowed human operators to supervise an outdoor mobile robot while it performed an autonomous task. From their mobile devices, users were able to initialize the robot, configure its motion, and monitor its progress. Also, users were able to reprogram the robot to perform new tasks (not previously planned) through a creative use of features offered in the developed HRI framework. Also, user cognitive effort was reported to be low as evident by the positive score on the NASA-TLX scale test which was corroborated with robot performance data. This paper presents the detailed development and implementation steps.

Social navigation framework for assistive robots in human inhabited unknown environments

Mobile robots are already applied in factories and hospitals, merely to do a distinct task. It is envisioned that robots assist in households soon. Those service robots will have to cope with several situations and tasks and of course with sophisticated human-robot interactions (HRI). Therefore, a robot has not only to consider social rules with respect to proxemics, it must detect in which (interaction) situation it is in and act accordingly. With respect to spatial HRI, we concentrate on the use of non-verbal communication. This chapter stresses the meaning of both, machine movements as signals towards a human and human body language. Considering these aspects will make interaction simpler and smoother. An observational study is presented to acquire a concept of spatial prompting by a robot and by a human. When a person and robot meet in a narrow hallway in order to pass by, they have to make room for each other. But how can a robot make sure that both really want to pass by instead of starting interaction? This especially concerns narrow, non-artificial surroundings. Which social signals are expected by the user and on the other side, can be generated or processed by a robot? The results will show what an appropriate passing behaviour is and how to distinguish between passage situations and others. The results shed light upon the readability of signals in spatial HRI.

Natural human–robot interaction (HRI) in complex and unpredictable environments is important with many potential applications. While vision-based HRI has been thoroughly investigated, robot hearing and audio-based HRI are emerging research topics in robotics. In typical real-world scenarios, humans are at some distance from the robot and, hence, the sensory (microphone) data are strongly impaired by background noise, reverberations and competing auditory sources. In this context, the detection and localization of speakers plays a key role that enables several tasks, such as improving the signal-to-noise ratio for speech recognition, speaker recognition, speaker tracking, etc. In this paper we address the problem of how to detect and localize people that are both seen and heard. We introduce a hybrid deterministic/probabilistic model. The deterministic component allows us to map 3D visual data onto a 1D auditory space. The probabilistic component of the model enables the visual features to guide the grouping of the auditory features in order to form audiovisual (AV) objects. The proposed model and the associated algorithms are implemented in real-time (17 FPS) using a stereoscopic camera pair and two microphones embedded into the head of the humanoid robot NAO. We perform experiments with (i) synthetic data, (ii) publicly available data gathered with an audiovisual robotic head, and (iii) data acquired using the NAO robot. The results validate the approach and are an encouragement to investigate how vision and hearing could be further combined for robust HRI.

Recently, face recognition became a key element in social cognition which is used in various applications including human–robot interaction (HRI), pedestrian identification, and surveillance systems. Deep convolutional neural networks (CNNs) have achieved notable progress in recognizing faces. However, achieving accurate and real-time face recognition is still a challenging problem, especially in unconstrained environments due to occlusion, lighting conditions, and the diversity in head poses. In this paper, we present a robust face recognition and tracking framework in unconstrained settings. We developed our framework based on lightweight CNNs for all face recognition stages, including face detection, alignment and feature extraction, to achieve higher accuracies in these challenging circumstances while maintaining the real-time capabilities required for HRI systems. To maintain the accuracy, a single-shot multi-level face localization in the wild (RetinaFace) is utilized for face detection, and additive angular margin loss (ArcFace) is employed for recognition. For further enhancement, we introduce a face tracking algorithm that combines the information from tracked faces with the recognized identity to use in the further frames. This tracking algorithm improves the overall processing time and accuracy. The proposed system performance is tested in real-time experiments applied in an HRI study. Our proposed framework achieves real-time capabilities with an average of 99%, 95%, and 97% precision, recall, and F-score respectively. In addition, we implemented our system as a modular ROS package that makes it straightforward for integration in different real-world HRI systems.

Development of robotics and growing use of robots in many areas of the economy and social life require research which will identify problems related to the use of modern technologies and the development of human-robot interaction (HRI, Human-Robot Interaction). Robotics, autonomous systems and artificial intelligence also create a number of new legal and ethical challenges. Research in this area has been expanding in recent years, but there are still issues that require in-depth scientific exploration. The objective of this paper is to systematize knowledge about interactions between people and robots, with particular emphasis on interactions in the work environment, and to identify research gaps in the analysed problem area. The following research questions were formulated: 1) What determines people's attitudes towards robots? 2) What are the key challenges in human-robot communication? 3) What ethical challenges determine human-robot interaction? In the studies the the method of narrative literature review has been used. The latest scientific publications on the researched issue were analysed. The results of the study indicate that the main factors which determine people's attitudes towards robots are apart from demographic variables and experiences gained by people from contacts with robots, the perceived usefulness of using robots, the image of robots created in the mass media and religiousness. The key challenges in human-robot interaction apart from effective communication is the reflection of emotions. The research results enable the identification of further directions of scientific research on human-robot interaction. They can also enrich practitioner’s perspective in the studied area.

Human-robot interaction is inevitable due to the increase of the autonomous intelligent machines in the human vicinity. The autonomous machines should synchronize with the interacting human. The key factor for synchronized Human-Robot Interaction (HRI) is a human intention. The interacting machine must have the clue about the intention of the interacting human for a useful interaction. In this review paper, recently proposed intention-based approaches for human-robot interaction are discussed. The approaches are categorized concerning different aspects, e.g., application area, specialized and generalized estimation techniques, etc. The review categorized the recently proposed approaches into five categories. The categorization is mainly based on the application areas of the intention recognition approaches. The type of approaches includes general and application-specific approaches. The application areas include synchronized physical human-assistance, human-synchronized vehicles, etc. The vehicles are synchronized with the driver and the pedestrian. The study highlighted the currently active as well as the dormant areas of the intention-based human-robot interaction and indicated new directions.

Intention recognition in Human–Robot Interaction (HRI) is critical for enabling robots to anticipate and respond to human actions effectively. This study explores the application of deep learning techniques for the classification of human intentions in HRI, utilizing data collected from Virtual Reality (VR) environments. By leveraging VR, a controlled and immersive space is created, where human behaviors can be closely monitored and recorded. Ensemble deep learning models, particularly Convolutional Neural Networks (CNNs), Long Short-Term Memory (LSTM), and Transformers, are trained on this rich dataset to recognize and predict human intentions with high accuracy. While CNN and CNN-LSTM models yielded high accuracy rates, they encountered difficulties in accurately identifying certain intentions (e.g., standing and walking). In contrast, the CNN-Transformer model outshone its counterparts, achieving near-perfect precision, recall, and F1-scores. The proposed approach demonstrates the potential for enhancing HRI by providing robots with the ability to anticipate and act on human intentions in real time, leading to more intuitive and effective collaboration between humans and robots. Experimental results highlight the effectiveness of VR as a data collection tool and the promise of deep learning in advancing intention recognition in HRI.