Human–robot shared control for humanoid manipulator trajectory planning

Fully autonomous robot systems, though highly desired, face substantial theoretical and practical challenges when being deployed into a dynamic environment where humans co-exist. To tackle this challenge, this thesis investigates the concept of human-in-the-loop (HITL) systems, which incorporate human input to enhance robot functionality. HITL systems offer a pragmatic alternative, combining human versatility with robotic precision. This research aims to address critical questions in one particular HITL architecture which prioritizes the dominant role of the human within the system, positioning the robot primarily in an assistive capacity that adheres to human commands to facilitate the achievement of a shared goal. It explores two primary paradigms of shared assistance methods-Shared Control (SC) and Shared Autonomy (SA)-and discusses the system designs as well as specific algorithms to implement the three critical components in a HITL system: human intention estimation, modulation of human inputs and robot autonomy, and the human-robot communication channel. Due to the variety of use cases and their unique challenges, four distinct HITL systems are developed and analyzed to illustrate how shared assistance methods could be employed to assist human operators: an assistive wheelchair for indoor navigation, a human-centered robot system design for industrial tasks, a mobile bi-manual robot for tele-manipulation, and a virtual reality (VR)-based customizable shared control system for fine teleoperation. Although each system represents a comprehensive robotic solution, the research contributions for each work vary. In the assistive wheelchair navigation system, the focus was on human intent estimation via a low-throughput interface utilizing a recursive Bayesian filter, with significant efforts dedicated to developing a real-time user interface serving as the communication channel. In the human-robot collaboration system for industrial setting, the emphasis was on human state estimation through camera-based posture tracking and exploring the interplay between robot behavior and human ergonomics. For the two teleoperation systems, the primary focus was on the real-time modulation of human inputs and robot autonomy to aid in achieving dexterous manipulation tasks. A novel VR-based user interface was developed to enable users to customize the robot autonomy level. Each system was validated through a pilot study involving 10-20 human subjects, accompanied by extensive data analysis to provide insights into designing HITL systems for various applications. In conclusion, this thesis contributes to a deeper understanding of HITL systems, highlighting their potential to enhance human productivity, ergonomics, and quality of life in varied applications through concrete examples. The integration of human intent estimation and real-time shared control methods into robotic systems demonstrates the feasibility and benefits of HITL approaches. Our extensive experimental analysis underscores the critical role of human feedback in designing practical HITL systems that can be deployed in real-world scenarios. --Author's abstract

Shared control of avatars in virtual reality is emerging as a promising method to enhance collaborative interactions. While the literature investigates shared control as an input modality to understand the design of functional experiences, the user experience associated with shared control remains underexplored. In this paper, we investigate the user experience of shared control using a networked VR environment, "CoplayingVR". Through a user study comprised of three gaming tasks in VR with 48 participants, we collect both quantitative and qualitative data on the user experience and performance of shared control. Our findings reveal that the shared control mode significantly improved user experience factors and brought novelty to the play. Furthermore, shared control significantly enhanced the task efficiency for novice users. Finally, we discuss how our findings inform the implications for research and practice of shared control for the growing body of research in VR.

Human–robot collaboration can enhance productivity of production lines and reduce human ergonomic risk. The numbers and types of robots and stations in which robots are allocated need to be determined. Operations should be scheduled carefully when a human and robot work on a part in a station to obtain a feasible operation allocation with the highest efficiency and lowest ergonomic risk. A mixed-integer linear programming model, constraint programming model, and Benders decomposition algorithm were developed to analyse advantages of collaborative robots in assembly lines. An energy expenditure method was used to evaluate ergonomic risk. By scheduling and balancing collaborative human–robot assembly lines, operational advantages and scheduling constraints from human–robot collaboration were studied when immobile and mobile robots are used. Regression lines were developed that can help managers determine how many and what types of robots are best for a line and what the impact of robot mobility on robot and line performance can be. The best configuration for equipping a line with collaborative robots is when (number of robots)/(number of stations) is near .7 and about 37% of robots are mobile. Robots can be efficiently used in lines with both a small and large number of passive resources and in simple and mixed-model lines.

Robots present an innovative solution to the construction industry’s challenges, including safety concerns, skilled worker shortages, and productivity issues. Successfully collaborating with robots requires new competencies to ensure safety, smooth interaction, and accelerated adoption of robotic technologies. However, limited research exists on the specific competencies needed for human—robot collaboration in construction. Moreover, the perspectives of construction industry professionals on these competencies remain underexplored. This study examines the perceptions of construction industry professionals regarding the knowledge, skills, and abilities necessary for the effective implementation of human—robot collaboration in construction. A two-round Delphi survey was conducted with expert panel members from the construction industry to assess their views on the competencies for human—robot collaboration. The results reveal that the most critical competencies include knowledge areas such as human—robot interface, construction robot applications, human—robot collaboration safety and standards, task planning and robot control system; skills such as task planning, safety management, technical expertise, human—robot interface, and communication; and abilities such as safety awareness, continuous learning, problemsolving, critical thinking, and spatial awareness. This study contributes to knowledge by identifying the most significant competencies for human—robot collaboration in construction and highlighting their relative importance. These competencies could inform the design of educational and training programs and facilitate the integration of robotic technologies in construction. The findings also provide a foundation for future research to further explore and enhance these competencies, ultimately supporting safer, more efficient, and more productive construction practices.

Human–robot interaction (HRI), in all its different variants, has gained significant attention in industry. The combination of human skills and robotics significantly enhances manufacturing, improving operations quality and efficiency. This is particularly relevant in highly variable contexts like remanufacturing, which aims to preserve the value of products and components over time and involves complex, non-standardized tasks such as inspection, disassembly, reprocessing, and reassembly. Uncertainties related to the conditions of products to remanufacture cannot be managed by pure automation making HRI highly beneficial. Although researchers have focused on the use of HRI in several fields of application (from construction to logistics), no extensive mapping of the research trends has specifically addressed the intersection of HRI and remanufacturing contexts. To fill this gap, this paper presents a comprehensive study based on a bibliometric analysis to explore the current state of research on this topic. Based on 120 articles from the Scopus database, this study points out the main countries, authors, and affiliations involved in the field. In addition, the key topics were analyzed. A predominant focus, above all in the last three years, on Human-Robot Collaboration (HRC) among the different levels of HRI, and disassembly processes was highlighted, while other related activities, such as reassembly, have received comparatively less attention. An unexplored frontier is the integration of Artificial Intelligence and Machine Learning with HRC tasks in remanufacturing. These findings offer valuable insights for researchers, scholars, and industrial professionals aiming to advance HRI in remanufacturing.

Driven by the evolving role of modern robots in collaboration with human operators, this paper proposes a virtual potential field-based motion planning and control strategy to realize effective human-robot collaboration through kinesthetic guidance via robot teleoperation. The motion of the teleoperated robot is determined by the human operator using a haptic device programmed to replicate movement under the influence of the virtual potential field constructed based on the obstacles and objectives in the environment. This approach facilitates seamless cooperation between the human and collaborative robot. It can also be readily generalized to include moving obstacles and goals in a dynamic environment. The results suggest that the virtual potential field can effectively enhance the safety and accuracy of human-robot cooperation. The developed method has promising applications in human-centric robotics fields such as medical robotics and robot-assisted surgeries.

The presented paper suggests a design method which seeks to identify the best scheduling of human robot collaborative (HRC) operations with respect to a required safety level. The human behavior along manufacturing scenarios is effectively forecasted through dedicated computer-aided tools. Consequently, this method stresses the usage of virtual environment to replicate both human postures and robot encumbrances over the manufacturing operations. Moreover, it proposes a safety index formulation for HRC systems based on the minimum distance between human and robot (H-R). As results, the approach returns the safety index for every possible combination of H-R operations. Subsequently, a scheduling algorithm suggests the operations sequence depending on the expected value of the safety index, providing an evaluation of the time needed to complete the process. The method is validated on surface control phase involved in post-processing of parts produced by laser powder bed fusion (L-PBF) Additive Manufacturing.

The human-robot collaboration (HRC) solutions presented so far have the disadvantage that the interaction between humans and robots is based on the human's state or on specific gestures purposely performed by the human, thus increasing the time required to perform a task and slowing down the pace of human labor, making such solutions uninteresting. In this study, a different concept of the HRC system is introduced, consisting of an HRC framework for managing assembly processes that are executed simultaneously or individually by humans and robots. This HRC framework based on deep learning models uses only one type of data, RGB camera data, to make predictions about the collaborative workspace and human action, and consequently manage the assembly process. To validate the HRC framework, an industrial HRC demonstrator was built to assemble a mechanical component. Four different HRC frameworks were created based on the convolutional neural network (CNN) model structures: Faster R-CNN ResNet-50 and ResNet-101, YOLOv2 and YOLOv3. The HRC framework with YOLOv3 structure showed the best performance, showing a mean average performance of 72.26% and allowed the HRC industrial demonstrator to successfully complete all assembly tasks within a desired time window. The HRC framework has proven effective for industrial assembly applications.

Adaptation to the operator's intents and the external physical uncertainties is critical to the performances of the human–robot collaboration. In this study, we proposed a control interface that combined the electrical impedance tomography (EIT) based sensing approach with the robotic controllers to produce proper assistance with external uncertainties in the collaborative task. The interface first estimates the continuous forearm muscle contractions (represented by the grasp forces) by the optimized EIT features with an easily worn fabric band. The recognition decisions then serve as the control inputs to adjust the state transitions and the desired interaction forces in real time. We evaluated the interface in the tasks of grasp force estimation and human–robot sawing by recruiting more than 20 subjects in total. For grasp force estimation, the interface produced an average of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$R^{2}\geq 0.9$</tex-math></inline-formula> in both offline and online validations with the feature optimization procedure and a sigmoid regression function. The interface was robust to external disturbances without retraining or manual calibrations. The average <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$R^{2}=0.86$</tex-math></inline-formula> with the untrained dynamic postures, and the average <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$R^{2}$</tex-math></inline-formula> values ranged from 0.85 to 0.88 in the tests with redonning the front-end in interday and intraday uses. For human–robot sawing, the interface accomplished the tasks with a high success rate in controlling the states ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$&gt;$</tex-math></inline-formula> 96%) and intuitive adjusting of the sawing forces, being combined with the designed hybrid admittance/position controller. It was also adaptive to the sawing frequency changes and the sawing directions according to the operator's intents. The interface's performances are comparable, if not better, to the state of the art on both biological signal based grasp force estimation and human–robot sawing. Future efforts are worth being paid in this new direction to get more promising outcomes.

Energy aware semi-automatic assembly line balancing problem considering ergonomic risk and uncertain processing time

In this paper, we present a control system for the continuous teleoperation of a robotic manipulator via brain-machine interface (BMI). The proposed solution is based on shared control approach that allows the user to only focus on the operational tasks, while the low-level control details are automatically handled by the robotic intelligence. The user drives the manipulator through the imagination of limb movements (both hands vs. both feet) and a parameterized mapping function is implemented to convert the continuous BMI outputs into robot velocity commands which are sent to the shared control framework. The latter consists in: (i) a target predictor module, to infer the most probable target objects from the sequence of BMI commands; (ii) a control module based on an improved version of artificial potential fields (APF) to assist the user in reaching the target while avoiding collisions with obstacles in the environment. The system has been tested with a sample subject in a tabletop reach-to-grasp experiment with multiple target objects and obstacles achieving a success rate of 80%. The proposed system could be used in the future to help people with severe motor disabilities in performing daily life operations, such as drinking, feeding or manipulating objects.

The evolution of industrial robotics has advanced from isolated, caged systems through basic human–robot interaction (HRI) to sophisticated human–robot collaboration (HRC). However, conventional vision systems based on red, green, blue (RGB) cameras remain a significant limiting factor in realizing the full potential of collaborative automation. This comprehensive review examines the transformative role of event cameras in advancing HRC capabilities and addressing current limitations in industrial settings. Event cameras, with their microsecond-level temporal resolution and robust performance under challenging lighting conditions, offer substantial advantages over traditional RGB cameras that are constrained by fixed frame rates and ambient lighting dependencies. We present a systematic framework for leveraging event cameras to enhance the human state understanding in collaborative robotics, encompassing real-time detection of poses, gestures, facial expressions, and emotional states. This framework addresses fundamental challenges in workplace safety and collaborative efficiency while enabling more sophisticated and responsive HRC systems. Our review synthesizes recent research developments in event camera applications specific to HRC, providing a detailed comparative analysis of their advantages over conventional vision systems. We identify emerging opportunities and potential research directions for advancing event-based vision in industrial robotics. In addition, we examine integration challenges and propose strategies for implementing event camera technology in existing industrial infrastructure. This work contributes valuable insights into the future trajectory of adaptive and intuitive HRC systems, offering a roadmap for researchers and practitioners in the field of industrial automation.

Human-Robot Collaboration (HRC) is key to achieving the flexible automation required by the mass personalization trend, especially towards human-centric intelligent manufacturing. Nevertheless, existing HRC systems suffer from poor task understanding and poor ergonomic satisfaction, which impede empathetic teamwork skills in task execution. To overcome the bottleneck, a Mixed Reality (MR) and visual reasoning-based method is proposed in this research, providing mutual-cognitive task assignment for human and robotic agents’ operations. Firstly, an MR-enabled mutual-cognitive HRC architecture is proposed, with the characteristic of monitoring Digital Twins states, reasoning co-working strategies, and providing cognitive services. Secondly, a visual reasoning approach is introduced, which learns scene interpretation from the visual perception of each agent’s actions and environmental changes to make task planning strategies satisfying human–robot operation needs. Lastly, a safe, ergonomic, and proactive robot motion planning algorithm is proposed to let a robot execute generated co-working strategies, while a human operator is supported with intuitive task operation guidance in the MR environment, achieving empathetic collaboration. Through a demonstration of a disassembly task of aging Electric Vehicle Batteries, the experimental result facilitates cognitive intelligence in Proactive HRC for flexible automation.

In many current assembly scenarios, efficient collaboration between humans and robots can improve collaborative efficiency and quality. However, the efficient arrangement of human–robot collaborative (HRC) tasks constitutes a significant challenge. In a collaborative workspace where humans and robots collaborate on assembling a shared product, the determination of task allocation between them is of crucial importance. To address this issue, offline feasible HRC paths are established based on assembly task constraint information. Subsequently, the HRC process is simulated within a virtual environment leveraging these feasible paths. Human assembly intentions are explicitly expressed through human assembly trajectories, and implicitly expressed through simulation results such as assembly time and human–robot resource allocation. Furthermore, a two-stage MLP-LSTM network is employed to train and optimize the assembly simulation database. In the first stage, a sequence generation model is trained using high-quality HRC processes. Then, the network learns human evaluation patterns to score the generated sequences. Ultimately, task allocation for HRC is performed based on the high-scoring generated sequences. The effectiveness of the proposed method is demonstrated through assembly scenarios of two products. Compared with traditional optimization methods like DFS and Greedy, the human collaboration ratio has been optimized by 10%, while the collaborative quality evaluation has been improved by 3%.

The human-robot collaboration (HRC) is an integral part of numerous research activities. The combination of the positive capabilities of human workers and robots ensures a high resource efficiency and productivity. However, especially the planning and implementation of HRC applications requires the consideration of several issues like the economic feasibility, acceptance of employees and in particular technical practicability. Based on the study of existing approaches which consider different technical or economic issues, we present a new method to identify suitable areas of HRC applications within assembly systems and to estimate the optimal allocation of human-related and automated subtasks. Finally, our approach is verified in an assembly line for terminal strips which in turn arises from an industrial use case.