Tᴏᴏʟ: accessible automated reasoning for human robot collaboration

Human–robot collaboration can enhance productivity of production lines and reduce human ergonomic risk. The numbers and types of robots and stations in which robots are allocated need to be determined. Operations should be scheduled carefully when a human and robot work on a part in a station to obtain a feasible operation allocation with the highest efficiency and lowest ergonomic risk. A mixed-integer linear programming model, constraint programming model, and Benders decomposition algorithm were developed to analyse advantages of collaborative robots in assembly lines. An energy expenditure method was used to evaluate ergonomic risk. By scheduling and balancing collaborative human–robot assembly lines, operational advantages and scheduling constraints from human–robot collaboration were studied when immobile and mobile robots are used. Regression lines were developed that can help managers determine how many and what types of robots are best for a line and what the impact of robot mobility on robot and line performance can be. The best configuration for equipping a line with collaborative robots is when (number of robots)/(number of stations) is near .7 and about 37% of robots are mobile. Robots can be efficiently used in lines with both a small and large number of passive resources and in simple and mixed-model lines.

Robots present an innovative solution to the construction industry’s challenges, including safety concerns, skilled worker shortages, and productivity issues. Successfully collaborating with robots requires new competencies to ensure safety, smooth interaction, and accelerated adoption of robotic technologies. However, limited research exists on the specific competencies needed for human—robot collaboration in construction. Moreover, the perspectives of construction industry professionals on these competencies remain underexplored. This study examines the perceptions of construction industry professionals regarding the knowledge, skills, and abilities necessary for the effective implementation of human—robot collaboration in construction. A two-round Delphi survey was conducted with expert panel members from the construction industry to assess their views on the competencies for human—robot collaboration. The results reveal that the most critical competencies include knowledge areas such as human—robot interface, construction robot applications, human—robot collaboration safety and standards, task planning and robot control system; skills such as task planning, safety management, technical expertise, human—robot interface, and communication; and abilities such as safety awareness, continuous learning, problemsolving, critical thinking, and spatial awareness. This study contributes to knowledge by identifying the most significant competencies for human—robot collaboration in construction and highlighting their relative importance. These competencies could inform the design of educational and training programs and facilitate the integration of robotic technologies in construction. The findings also provide a foundation for future research to further explore and enhance these competencies, ultimately supporting safer, more efficient, and more productive construction practices.

The presented paper suggests a design method which seeks to identify the best scheduling of human robot collaborative (HRC) operations with respect to a required safety level. The human behavior along manufacturing scenarios is effectively forecasted through dedicated computer-aided tools. Consequently, this method stresses the usage of virtual environment to replicate both human postures and robot encumbrances over the manufacturing operations. Moreover, it proposes a safety index formulation for HRC systems based on the minimum distance between human and robot (H-R). As results, the approach returns the safety index for every possible combination of H-R operations. Subsequently, a scheduling algorithm suggests the operations sequence depending on the expected value of the safety index, providing an evaluation of the time needed to complete the process. The method is validated on surface control phase involved in post-processing of parts produced by laser powder bed fusion (L-PBF) Additive Manufacturing.

The human-robot collaboration (HRC) solutions presented so far have the disadvantage that the interaction between humans and robots is based on the human's state or on specific gestures purposely performed by the human, thus increasing the time required to perform a task and slowing down the pace of human labor, making such solutions uninteresting. In this study, a different concept of the HRC system is introduced, consisting of an HRC framework for managing assembly processes that are executed simultaneously or individually by humans and robots. This HRC framework based on deep learning models uses only one type of data, RGB camera data, to make predictions about the collaborative workspace and human action, and consequently manage the assembly process. To validate the HRC framework, an industrial HRC demonstrator was built to assemble a mechanical component. Four different HRC frameworks were created based on the convolutional neural network (CNN) model structures: Faster R-CNN ResNet-50 and ResNet-101, YOLOv2 and YOLOv3. The HRC framework with YOLOv3 structure showed the best performance, showing a mean average performance of 72.26% and allowed the HRC industrial demonstrator to successfully complete all assembly tasks within a desired time window. The HRC framework has proven effective for industrial assembly applications.

Human–robot interaction (HRI), in all its different variants, has gained significant attention in industry. The combination of human skills and robotics significantly enhances manufacturing, improving operations quality and efficiency. This is particularly relevant in highly variable contexts like remanufacturing, which aims to preserve the value of products and components over time and involves complex, non-standardized tasks such as inspection, disassembly, reprocessing, and reassembly. Uncertainties related to the conditions of products to remanufacture cannot be managed by pure automation making HRI highly beneficial. Although researchers have focused on the use of HRI in several fields of application (from construction to logistics), no extensive mapping of the research trends has specifically addressed the intersection of HRI and remanufacturing contexts. To fill this gap, this paper presents a comprehensive study based on a bibliometric analysis to explore the current state of research on this topic. Based on 120 articles from the Scopus database, this study points out the main countries, authors, and affiliations involved in the field. In addition, the key topics were analyzed. A predominant focus, above all in the last three years, on Human-Robot Collaboration (HRC) among the different levels of HRI, and disassembly processes was highlighted, while other related activities, such as reassembly, have received comparatively less attention. An unexplored frontier is the integration of Artificial Intelligence and Machine Learning with HRC tasks in remanufacturing. These findings offer valuable insights for researchers, scholars, and industrial professionals aiming to advance HRI in remanufacturing.

Driven by the evolving role of modern robots in collaboration with human operators, this paper proposes a virtual potential field-based motion planning and control strategy to realize effective human-robot collaboration through kinesthetic guidance via robot teleoperation. The motion of the teleoperated robot is determined by the human operator using a haptic device programmed to replicate movement under the influence of the virtual potential field constructed based on the obstacles and objectives in the environment. This approach facilitates seamless cooperation between the human and collaborative robot. It can also be readily generalized to include moving obstacles and goals in a dynamic environment. The results suggest that the virtual potential field can effectively enhance the safety and accuracy of human-robot cooperation. The developed method has promising applications in human-centric robotics fields such as medical robotics and robot-assisted surgeries.

Adaptation to the operator's intents and the external physical uncertainties is critical to the performances of the human–robot collaboration. In this study, we proposed a control interface that combined the electrical impedance tomography (EIT) based sensing approach with the robotic controllers to produce proper assistance with external uncertainties in the collaborative task. The interface first estimates the continuous forearm muscle contractions (represented by the grasp forces) by the optimized EIT features with an easily worn fabric band. The recognition decisions then serve as the control inputs to adjust the state transitions and the desired interaction forces in real time. We evaluated the interface in the tasks of grasp force estimation and human–robot sawing by recruiting more than 20 subjects in total. For grasp force estimation, the interface produced an average of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$R^{2}\geq 0.9$</tex-math></inline-formula> in both offline and online validations with the feature optimization procedure and a sigmoid regression function. The interface was robust to external disturbances without retraining or manual calibrations. The average <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$R^{2}=0.86$</tex-math></inline-formula> with the untrained dynamic postures, and the average <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$R^{2}$</tex-math></inline-formula> values ranged from 0.85 to 0.88 in the tests with redonning the front-end in interday and intraday uses. For human–robot sawing, the interface accomplished the tasks with a high success rate in controlling the states ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$&gt;$</tex-math></inline-formula> 96%) and intuitive adjusting of the sawing forces, being combined with the designed hybrid admittance/position controller. It was also adaptive to the sawing frequency changes and the sawing directions according to the operator's intents. The interface's performances are comparable, if not better, to the state of the art on both biological signal based grasp force estimation and human–robot sawing. Future efforts are worth being paid in this new direction to get more promising outcomes.

The evolution of industrial robotics has advanced from isolated, caged systems through basic human–robot interaction (HRI) to sophisticated human–robot collaboration (HRC). However, conventional vision systems based on red, green, blue (RGB) cameras remain a significant limiting factor in realizing the full potential of collaborative automation. This comprehensive review examines the transformative role of event cameras in advancing HRC capabilities and addressing current limitations in industrial settings. Event cameras, with their microsecond-level temporal resolution and robust performance under challenging lighting conditions, offer substantial advantages over traditional RGB cameras that are constrained by fixed frame rates and ambient lighting dependencies. We present a systematic framework for leveraging event cameras to enhance the human state understanding in collaborative robotics, encompassing real-time detection of poses, gestures, facial expressions, and emotional states. This framework addresses fundamental challenges in workplace safety and collaborative efficiency while enabling more sophisticated and responsive HRC systems. Our review synthesizes recent research developments in event camera applications specific to HRC, providing a detailed comparative analysis of their advantages over conventional vision systems. We identify emerging opportunities and potential research directions for advancing event-based vision in industrial robotics. In addition, we examine integration challenges and propose strategies for implementing event camera technology in existing industrial infrastructure. This work contributes valuable insights into the future trajectory of adaptive and intuitive HRC systems, offering a roadmap for researchers and practitioners in the field of industrial automation.

Human-Robot Collaboration (HRC) is key to achieving the flexible automation required by the mass personalization trend, especially towards human-centric intelligent manufacturing. Nevertheless, existing HRC systems suffer from poor task understanding and poor ergonomic satisfaction, which impede empathetic teamwork skills in task execution. To overcome the bottleneck, a Mixed Reality (MR) and visual reasoning-based method is proposed in this research, providing mutual-cognitive task assignment for human and robotic agents’ operations. Firstly, an MR-enabled mutual-cognitive HRC architecture is proposed, with the characteristic of monitoring Digital Twins states, reasoning co-working strategies, and providing cognitive services. Secondly, a visual reasoning approach is introduced, which learns scene interpretation from the visual perception of each agent’s actions and environmental changes to make task planning strategies satisfying human–robot operation needs. Lastly, a safe, ergonomic, and proactive robot motion planning algorithm is proposed to let a robot execute generated co-working strategies, while a human operator is supported with intuitive task operation guidance in the MR environment, achieving empathetic collaboration. Through a demonstration of a disassembly task of aging Electric Vehicle Batteries, the experimental result facilitates cognitive intelligence in Proactive HRC for flexible automation.

In many current assembly scenarios, efficient collaboration between humans and robots can improve collaborative efficiency and quality. However, the efficient arrangement of human–robot collaborative (HRC) tasks constitutes a significant challenge. In a collaborative workspace where humans and robots collaborate on assembling a shared product, the determination of task allocation between them is of crucial importance. To address this issue, offline feasible HRC paths are established based on assembly task constraint information. Subsequently, the HRC process is simulated within a virtual environment leveraging these feasible paths. Human assembly intentions are explicitly expressed through human assembly trajectories, and implicitly expressed through simulation results such as assembly time and human–robot resource allocation. Furthermore, a two-stage MLP-LSTM network is employed to train and optimize the assembly simulation database. In the first stage, a sequence generation model is trained using high-quality HRC processes. Then, the network learns human evaluation patterns to score the generated sequences. Ultimately, task allocation for HRC is performed based on the high-scoring generated sequences. The effectiveness of the proposed method is demonstrated through assembly scenarios of two products. Compared with traditional optimization methods like DFS and Greedy, the human collaboration ratio has been optimized by 10%, while the collaborative quality evaluation has been improved by 3%.

The human-robot collaboration (HRC) is an integral part of numerous research activities. The combination of the positive capabilities of human workers and robots ensures a high resource efficiency and productivity. However, especially the planning and implementation of HRC applications requires the consideration of several issues like the economic feasibility, acceptance of employees and in particular technical practicability. Based on the study of existing approaches which consider different technical or economic issues, we present a new method to identify suitable areas of HRC applications within assembly systems and to estimate the optimal allocation of human-related and automated subtasks. Finally, our approach is verified in an assembly line for terminal strips which in turn arises from an industrial use case.

Vision-based holistic scene understanding towards proactive human–robot collaboration

Purpose This paper aims to propose an intuitive shared control strategy to control a humanoid manipulator that can fully combine the advantages of humans and machines to produce a stronger intelligent form. Design/methodology/approach The working space of an operator’s arm and that of a manipulator are matched, and a genetic algorithm that limits the position of the manipulator’s elbow joint is used to find the optimal solution. Then, the mapping of the operator’s action to that of manipulators is realized. The controls of the human and robot are integrated. First, the current action of the operator is input. Second, the target object is predicted according to the maximum entropy hypothesis. Third, the joint angle of the manipulator is interpolated based on time. Finally, the confidence and weight of the current moment are calculated. Findings The modified weight adjustment method is the optimal way to adjust the weight during the task. In terms of time and accuracy, the experimental results of single target obstacle avoidance grabbing and multi-target predictive grabbing show that the shared control mode can provide full play to the advantages of humans and robots to accomplish the target task faster and more accurately than the control merely by a human or robot on its own. Originality/value A flexible and highly anthropomorphic human–robot action mapping method is proposed, which provides operator decisions in the shared control process. The shared control between human and the robot is realized, and it enhances the rapidity and intelligence, paving a new way for a novel human–robot collaboration.

Effective interactions between humans and robots are vital to achieving shared tasks in collaborative processes. Robots can utilize diverse communication channels to interact with humans, such as hearing, speech, sight, touch, and learning. Our focus, amidst the various means of interactions between humans and robots, is on three emerging frontiers that significantly impact the future directions of human–robot interaction (HRI): (i) human–robot collaboration inspired by human–human collaboration, (ii) brain-computer interfaces, and (iii) emotional intelligent perception. First, we explore advanced techniques for human–robot collaboration, covering a range of methods from compliance and performance-based approaches to synergistic and learning-based strategies, including learning from demonstration, active learning, and learning from complex tasks. Then, we examine innovative uses of brain-computer interfaces for enhancing HRI, with a focus on applications in rehabilitation, communication, brain state and emotion recognition. Finally, we investigate the emotional intelligence in robotics, focusing on translating human emotions to robots via facial expressions, body gestures, and eye-tracking for fluid, natural interactions. Recent developments in these emerging frontiers and their impact on HRI were detailed and discussed. We highlight contemporary trends and emerging advancements in the field. Ultimately, this paper underscores the necessity of a multimodal approach in developing systems capable of adaptive behavior and effective interaction between humans and robots, thus offering a thorough understanding of the diverse modalities essential for maximizing the potential of HRI.

The recent automation technology advances have introduced the shared human robot (HR) workspaces concept for the future production lines. This chapter presents the design and application of safety systems that enable the dynamic monitoring of zones inside a human robot collaborative cell. The safety concepts suggested have as a main enabler a 3D safety certified camera that can detect in real time any moving human or other object in the defined area. The discussion starts from a concept of replacing physical fences with virtual safety zones while monitoring human robot distance. As a next step, a more dynamic scheme is proposed by enhancing 3D safety zones with robot side safety functions while enabling the dynamic switching of zones to minimize the allowed HR distance. The control architecture design and the implementation of the systems in two industrial examples from the automotive industry are presented in this chapter.