Cognitive Interaction Analysis in Human–Robot Collaboration Using an Assembly Task

In recent years, collaborative robots have been introduced into production systems as a new generation of industrial robots, sharing work with workers. Unlike industrial robots and workers in the past to work in isolation, collaborative robots and workers can often share work tasks and space. The introduction of these collaborative robots makes human-robot collaboration (HRC) more and more important, which is an emerging technology in the field of production systems. According to the characteristics of workers and collaborative robots, a measurement model of assembly complexity is first proposed. The assembly tasks are analyzed from the three aspects of process depth, operation process, and cognitive process, and the complexity of each assembly task is evaluated. The indicator of the machine-machine collaboration attribute measures each task to determine whether the task can be completed by a collaborative robot; finally, an assembly task allocation evaluation model is established, to ensure the assembly complexity of each worker’s task is balanced, so that the cognitive load assigned to the robot task is lower than the worker’s task, to maximize production efficiency and reduce the assembly system cost.

Driven by the evolving role of modern robots in collaboration with human operators, this paper proposes a virtual potential field-based motion planning and control strategy to realize effective human-robot collaboration through kinesthetic guidance via robot teleoperation. The motion of the teleoperated robot is determined by the human operator using a haptic device programmed to replicate movement under the influence of the virtual potential field constructed based on the obstacles and objectives in the environment. This approach facilitates seamless cooperation between the human and collaborative robot. It can also be readily generalized to include moving obstacles and goals in a dynamic environment. The results suggest that the virtual potential field can effectively enhance the safety and accuracy of human-robot cooperation. The developed method has promising applications in human-centric robotics fields such as medical robotics and robot-assisted surgeries.

In many current assembly scenarios, efficient collaboration between humans and robots can improve collaborative efficiency and quality. However, the efficient arrangement of human–robot collaborative (HRC) tasks constitutes a significant challenge. In a collaborative workspace where humans and robots collaborate on assembling a shared product, the determination of task allocation between them is of crucial importance. To address this issue, offline feasible HRC paths are established based on assembly task constraint information. Subsequently, the HRC process is simulated within a virtual environment leveraging these feasible paths. Human assembly intentions are explicitly expressed through human assembly trajectories, and implicitly expressed through simulation results such as assembly time and human–robot resource allocation. Furthermore, a two-stage MLP-LSTM network is employed to train and optimize the assembly simulation database. In the first stage, a sequence generation model is trained using high-quality HRC processes. Then, the network learns human evaluation patterns to score the generated sequences. Ultimately, task allocation for HRC is performed based on the high-scoring generated sequences. The effectiveness of the proposed method is demonstrated through assembly scenarios of two products. Compared with traditional optimization methods like DFS and Greedy, the human collaboration ratio has been optimized by 10%, while the collaborative quality evaluation has been improved by 3%.

Human Pose Estimation (HPE) is an essential computer vision task that plays a significant role in ergonomics, human-centered design of workplaces, and collaborative robotics, particularly in providing a safe, adaptive, and valuable human-robot collaboration (HRC). Nevertheless, the need for a concise and practical approach to setting up a multicamera and multimodal system for HPE dataset generation remains an open issue in the literature.The main goal of this work is to describe in concise steps a protocol that serves as a guide in the exploratory stage of the approach to construct an extensive multicamera and multimodal dataset used to enhance HRC. In this paper, the proposed protocol specifically addresses the challenge of designing a biomechanical model that can consistently reproduce complex and variable human motion analyses in an assembly task while considering ergonomic factors. Furthermore, the resulting work led to the definition of a marker set for one single-person future pipeline involving the placement of thirty-two reflective markers for 3D motion analysis, specifically emphasizing the upper segments of the human body, including the hands. The future generation of this dataset will hold significant promise for advancing the study of HRC. It will introduce reliable and precise multimodal data collection, such as human kinematics and video data, including depth data, which will then be used for posture metrics analysis. Finally, the dataset will be a valuable resource for the research community, enabling the training of machine learning models. These models will empower collaborative robots (cobots) to learn from human demonstrations, enhancing their efficiency and ergonomic performance in assembly tasks.

Human-robot collaboration (HRC), as a part of Industry 4.0 strategy, requires a completely new type of robots able to co-work with humans, called collaborative robots or cobots. This kind of collaboration is especially needed in assembly systems, which are known for having a low level of automation. For some assembly tasks human is still an irreplaceable factor. On the other hand, some assembly tasks are monotonous and tiring for humans. Therefore, the different approaches to cope with the challenge of identification and selection of proper task allocation between human worker and cobots are reported by many researchers. It is not an easy task since multiple and often conflicting criteria need to be taken into account. Some kind of artificial decision support is needed, to successfully solve this problem. In this research, task allocation procedure is presented for identification of different improvement options that utilize cobots into the assembly line for different tasks to be performed. The decision support system based on the HUMANT algorithm has been used for selection of the option which represents the best compromise solution. The procedure is experimentally tested on the assembly line with car gearboxes as a real product.

Human–robot collaboration can enhance productivity of production lines and reduce human ergonomic risk. The numbers and types of robots and stations in which robots are allocated need to be determined. Operations should be scheduled carefully when a human and robot work on a part in a station to obtain a feasible operation allocation with the highest efficiency and lowest ergonomic risk. A mixed-integer linear programming model, constraint programming model, and Benders decomposition algorithm were developed to analyse advantages of collaborative robots in assembly lines. An energy expenditure method was used to evaluate ergonomic risk. By scheduling and balancing collaborative human–robot assembly lines, operational advantages and scheduling constraints from human–robot collaboration were studied when immobile and mobile robots are used. Regression lines were developed that can help managers determine how many and what types of robots are best for a line and what the impact of robot mobility on robot and line performance can be. The best configuration for equipping a line with collaborative robots is when (number of robots)/(number of stations) is near .7 and about 37% of robots are mobile. Robots can be efficiently used in lines with both a small and large number of passive resources and in simple and mixed-model lines.

Industry 5.0 aims to prioritize human operators, focusing on their well-being and capabilities, while promoting collaboration between humans and robots to enhance efficiency and productivity. The integration of collaborative robots must ensure the health and well-being of human operators. Indeed, this paper addresses the need for a human-centered framework proposing a preference-based optimization algorithm in a human–robot collaboration (HRC) scenario with an ergonomics assessment to improve working conditions. The HRC application consists of optimizing a collaborative robot end-effector pose during an object-handling task. The following approach (AmPL-RULA) utilizes an Active multi-Preference Learning (AmPL) algorithm, a preference-based optimization method, where the user is requested to iteratively provide qualitative feedback by expressing pairwise preferences between a couple of candidates. To address physical well-being, an ergonomic performance index, Rapid Upper Limb Assessment (RULA), is combined with the user’s pairwise preferences, so that the optimal setting can be computed. Experimental tests have been conducted to validate the method, involving collaborative assembly during the object handling performed by the robot. Results illustrate that the proposed method can improve the physical workload of the operator while easing the collaborative task.

Human-Robot Collaboration (HRC) is key to achieving the flexible automation required by the mass personalization trend, especially towards human-centric intelligent manufacturing. Nevertheless, existing HRC systems suffer from poor task understanding and poor ergonomic satisfaction, which impede empathetic teamwork skills in task execution. To overcome the bottleneck, a Mixed Reality (MR) and visual reasoning-based method is proposed in this research, providing mutual-cognitive task assignment for human and robotic agents’ operations. Firstly, an MR-enabled mutual-cognitive HRC architecture is proposed, with the characteristic of monitoring Digital Twins states, reasoning co-working strategies, and providing cognitive services. Secondly, a visual reasoning approach is introduced, which learns scene interpretation from the visual perception of each agent’s actions and environmental changes to make task planning strategies satisfying human–robot operation needs. Lastly, a safe, ergonomic, and proactive robot motion planning algorithm is proposed to let a robot execute generated co-working strategies, while a human operator is supported with intuitive task operation guidance in the MR environment, achieving empathetic collaboration. Through a demonstration of a disassembly task of aging Electric Vehicle Batteries, the experimental result facilitates cognitive intelligence in Proactive HRC for flexible automation.

This study proposes to improve the reliability, robustness and human-like nature of Human–Robot Collaboration (HRC). For that, the classical Perception–Action cycle is extended to a Perception–Intention–Action (PIA) cycle, which includes an Intention stage at the same level as the Perception one, being in charge of obtaining both the implicit and the explicit intention of the human, opposing to classical approaches based on inferring everything from perception. This complete cycle is exposed theoretically including its use of the concept of Situation Awareness, which is shown as a key element for the correct understanding of the current situation and future action prediction. This enables the assignment of roles to the agents involved in a collaborative task and the building of collaborative plans. To visualize the cycle, a collaborative transportation task is used as a use-case. A force-based model is designed to combine the robot’s perception of its environment with the force exerted by the human and other factors in an illustrative way. Finally, a total of 58 volunteers participate in two rounds of experiments. In these, it is shown that the human agrees to explicitly state their intention without undue extra effort and that the human understands that this helps to minimize robot errors or misunderstandings. It is also shown that a system that correctly combines inference with explicit elicitation of the human’s intention is the best rated by the human on multiple parameters related to effective Human–Robot Interaction (HRI), such as perceived safety or trust in the robot.

The evolution of industrial robotics has advanced from isolated, caged systems through basic human–robot interaction (HRI) to sophisticated human–robot collaboration (HRC). However, conventional vision systems based on red, green, blue (RGB) cameras remain a significant limiting factor in realizing the full potential of collaborative automation. This comprehensive review examines the transformative role of event cameras in advancing HRC capabilities and addressing current limitations in industrial settings. Event cameras, with their microsecond-level temporal resolution and robust performance under challenging lighting conditions, offer substantial advantages over traditional RGB cameras that are constrained by fixed frame rates and ambient lighting dependencies. We present a systematic framework for leveraging event cameras to enhance the human state understanding in collaborative robotics, encompassing real-time detection of poses, gestures, facial expressions, and emotional states. This framework addresses fundamental challenges in workplace safety and collaborative efficiency while enabling more sophisticated and responsive HRC systems. Our review synthesizes recent research developments in event camera applications specific to HRC, providing a detailed comparative analysis of their advantages over conventional vision systems. We identify emerging opportunities and potential research directions for advancing event-based vision in industrial robotics. In addition, we examine integration challenges and propose strategies for implementing event camera technology in existing industrial infrastructure. This work contributes valuable insights into the future trajectory of adaptive and intuitive HRC systems, offering a roadmap for researchers and practitioners in the field of industrial automation.

Human–robot collaboration in sensorless assembly task learning enhanced by uncertainties adaptation via Bayesian Optimization

Adaptation to the operator's intents and the external physical uncertainties is critical to the performances of the human–robot collaboration. In this study, we proposed a control interface that combined the electrical impedance tomography (EIT) based sensing approach with the robotic controllers to produce proper assistance with external uncertainties in the collaborative task. The interface first estimates the continuous forearm muscle contractions (represented by the grasp forces) by the optimized EIT features with an easily worn fabric band. The recognition decisions then serve as the control inputs to adjust the state transitions and the desired interaction forces in real time. We evaluated the interface in the tasks of grasp force estimation and human–robot sawing by recruiting more than 20 subjects in total. For grasp force estimation, the interface produced an average of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$R^{2}\geq 0.9$</tex-math></inline-formula> in both offline and online validations with the feature optimization procedure and a sigmoid regression function. The interface was robust to external disturbances without retraining or manual calibrations. The average <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$R^{2}=0.86$</tex-math></inline-formula> with the untrained dynamic postures, and the average <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$R^{2}$</tex-math></inline-formula> values ranged from 0.85 to 0.88 in the tests with redonning the front-end in interday and intraday uses. For human–robot sawing, the interface accomplished the tasks with a high success rate in controlling the states ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$&gt;$</tex-math></inline-formula> 96%) and intuitive adjusting of the sawing forces, being combined with the designed hybrid admittance/position controller. It was also adaptive to the sawing frequency changes and the sawing directions according to the operator's intents. The interface's performances are comparable, if not better, to the state of the art on both biological signal based grasp force estimation and human–robot sawing. Future efforts are worth being paid in this new direction to get more promising outcomes.

In human–robot collaboration (HRC) scenarios with direct physical contact, accurately estimating human intentions and adjusting robot behaviors based on multimodal information is the core factors that restrict the efficiency and precision of current HRC tasks. To enhance the performance of human–robot collaboration under physical contact conditions, we propose a joint network model named ACT_force_cooperative (AFC). This model leverages force sensing information as a representation of human intent to achieve human intent prediction during physical interaction, while simultaneously capturing visual information and robot state data, thereby enabling more efficient execution of human–robot collaborative tasks with multimodal information processing. Existing HRC methods often ignore humans’ collaborative experience in the environment and fail to recognize the uniqueness of interactive force in expressing human intentions. Focusing on the special role of interactive force among various types of data in physical interaction environments, the proposed model predicts humans’ future behavioral intentions and adopts a Transformer model to realize the fusion and representation of multimodal information, thus accomplishing more accurate and effective HRC tasks. Experimental results demonstrate that, through the processing of force sensing information and fusion of multimodal data, the proposed model reduces the motion error by 44.9% and increases the effective collaborative time ratio by 20.2% compared with the baseline Action Chunk Transformer (ACT) model. This not only improves the motion accuracy of the robot in collaborative tasks but also enhances the collaborative experience of human operators.

Using humans and robots working together to disassemble end-of-life (EoL) products so that they can be remanufactured enables catering for the effects of uncertainties in the condition of those products and unpredictability in remanufacturing operations. Collaborative robots are one of the key elements of Industry 4.0. This paper presents a strategy for human-robot collaborative disassembly that exploits the active compliance control capability of collaborative robots. The paper outlines the proposed human-robot collaboration strategy and active compliance control. It then presents a case study of robotic disassembly designed to show a practical implementation of the proposed strategy to take products apart. The study involved the dismantling of an automotive water pump by a collaborative industrial robot working with a human operator and by two collaborative robots and a human operator. Active compliance control was implemented to separate press-fitted components from the water pump. The results demonstrate the feasibility of the strategy and its effectiveness at increasing flexibility and adaptability by reducing setup efforts.

As the development of robotics technologies for collaborative robots (COBOTs), the applications of human–robot collaboration (HRC) have been growing in the past decade. Despite the tremendous efforts from both academia and industry, the overall usage and acceptance of COBOTs are still not so high as expected. One of the major affecting factors is the comfort of humans in HRC, which is usually less emphasized in COBOT development; however, it is critical to the user acceptance during HRC. Therefore, this paper gives a review of human comfort in HRC including the influential factors of human comfort, measurement of human comfort in terms of subjective and objective manners, and human comfort improvement approaches in the context of HRC. Discussions on each topic are also conducted based on the review and analysis.