Vision-based holistic scene understanding towards proactive human–robot collaboration

In a goal-oriented Human Robot Collaborative (HRC) scenario, where the goal is to complete an assembly process, a robust object tracker might not necessarily fulfill its functional role due to the dynamic nature of HRC. Moreover, for an efficient HRC, the functional role of the object tacker should not only be limited to localizing and tracking objects for robotic manipulation. It should also help to determine the current state of the assembly process and verify if the chosen action has been successfully performed and thus to enable an uninterrupted completion of an HRC assembly process. We present a Context Enhanced Framework for Multi Object Tracking, that i) allows uninterrupted completion of an assembly process, ii) improves the overall functional accuracy of the object tracker from 49 percent to 96 percent, and iii) enables the object tracker to handle multiple instance of multiple objects in a HRC setting.

The evolution of industrial robotics has advanced from isolated, caged systems through basic human–robot interaction (HRI) to sophisticated human–robot collaboration (HRC). However, conventional vision systems based on red, green, blue (RGB) cameras remain a significant limiting factor in realizing the full potential of collaborative automation. This comprehensive review examines the transformative role of event cameras in advancing HRC capabilities and addressing current limitations in industrial settings. Event cameras, with their microsecond-level temporal resolution and robust performance under challenging lighting conditions, offer substantial advantages over traditional RGB cameras that are constrained by fixed frame rates and ambient lighting dependencies. We present a systematic framework for leveraging event cameras to enhance the human state understanding in collaborative robotics, encompassing real-time detection of poses, gestures, facial expressions, and emotional states. This framework addresses fundamental challenges in workplace safety and collaborative efficiency while enabling more sophisticated and responsive HRC systems. Our review synthesizes recent research developments in event camera applications specific to HRC, providing a detailed comparative analysis of their advantages over conventional vision systems. We identify emerging opportunities and potential research directions for advancing event-based vision in industrial robotics. In addition, we examine integration challenges and propose strategies for implementing event camera technology in existing industrial infrastructure. This work contributes valuable insights into the future trajectory of adaptive and intuitive HRC systems, offering a roadmap for researchers and practitioners in the field of industrial automation.

Human–robot collaboration can enhance productivity of production lines and reduce human ergonomic risk. The numbers and types of robots and stations in which robots are allocated need to be determined. Operations should be scheduled carefully when a human and robot work on a part in a station to obtain a feasible operation allocation with the highest efficiency and lowest ergonomic risk. A mixed-integer linear programming model, constraint programming model, and Benders decomposition algorithm were developed to analyse advantages of collaborative robots in assembly lines. An energy expenditure method was used to evaluate ergonomic risk. By scheduling and balancing collaborative human–robot assembly lines, operational advantages and scheduling constraints from human–robot collaboration were studied when immobile and mobile robots are used. Regression lines were developed that can help managers determine how many and what types of robots are best for a line and what the impact of robot mobility on robot and line performance can be. The best configuration for equipping a line with collaborative robots is when (number of robots)/(number of stations) is near .7 and about 37% of robots are mobile. Robots can be efficiently used in lines with both a small and large number of passive resources and in simple and mixed-model lines.

Robots present an innovative solution to the construction industry’s challenges, including safety concerns, skilled worker shortages, and productivity issues. Successfully collaborating with robots requires new competencies to ensure safety, smooth interaction, and accelerated adoption of robotic technologies. However, limited research exists on the specific competencies needed for human—robot collaboration in construction. Moreover, the perspectives of construction industry professionals on these competencies remain underexplored. This study examines the perceptions of construction industry professionals regarding the knowledge, skills, and abilities necessary for the effective implementation of human—robot collaboration in construction. A two-round Delphi survey was conducted with expert panel members from the construction industry to assess their views on the competencies for human—robot collaboration. The results reveal that the most critical competencies include knowledge areas such as human—robot interface, construction robot applications, human—robot collaboration safety and standards, task planning and robot control system; skills such as task planning, safety management, technical expertise, human—robot interface, and communication; and abilities such as safety awareness, continuous learning, problemsolving, critical thinking, and spatial awareness. This study contributes to knowledge by identifying the most significant competencies for human—robot collaboration in construction and highlighting their relative importance. These competencies could inform the design of educational and training programs and facilitate the integration of robotic technologies in construction. The findings also provide a foundation for future research to further explore and enhance these competencies, ultimately supporting safer, more efficient, and more productive construction practices.

The human-robot collaboration (HRC) solutions presented so far have the disadvantage that the interaction between humans and robots is based on the human's state or on specific gestures purposely performed by the human, thus increasing the time required to perform a task and slowing down the pace of human labor, making such solutions uninteresting. In this study, a different concept of the HRC system is introduced, consisting of an HRC framework for managing assembly processes that are executed simultaneously or individually by humans and robots. This HRC framework based on deep learning models uses only one type of data, RGB camera data, to make predictions about the collaborative workspace and human action, and consequently manage the assembly process. To validate the HRC framework, an industrial HRC demonstrator was built to assemble a mechanical component. Four different HRC frameworks were created based on the convolutional neural network (CNN) model structures: Faster R-CNN ResNet-50 and ResNet-101, YOLOv2 and YOLOv3. The HRC framework with YOLOv3 structure showed the best performance, showing a mean average performance of 72.26% and allowed the HRC industrial demonstrator to successfully complete all assembly tasks within a desired time window. The HRC framework has proven effective for industrial assembly applications.

In many current assembly scenarios, efficient collaboration between humans and robots can improve collaborative efficiency and quality. However, the efficient arrangement of human–robot collaborative (HRC) tasks constitutes a significant challenge. In a collaborative workspace where humans and robots collaborate on assembling a shared product, the determination of task allocation between them is of crucial importance. To address this issue, offline feasible HRC paths are established based on assembly task constraint information. Subsequently, the HRC process is simulated within a virtual environment leveraging these feasible paths. Human assembly intentions are explicitly expressed through human assembly trajectories, and implicitly expressed through simulation results such as assembly time and human–robot resource allocation. Furthermore, a two-stage MLP-LSTM network is employed to train and optimize the assembly simulation database. In the first stage, a sequence generation model is trained using high-quality HRC processes. Then, the network learns human evaluation patterns to score the generated sequences. Ultimately, task allocation for HRC is performed based on the high-scoring generated sequences. The effectiveness of the proposed method is demonstrated through assembly scenarios of two products. Compared with traditional optimization methods like DFS and Greedy, the human collaboration ratio has been optimized by 10%, while the collaborative quality evaluation has been improved by 3%.

The presented paper suggests a design method which seeks to identify the best scheduling of human robot collaborative (HRC) operations with respect to a required safety level. The human behavior along manufacturing scenarios is effectively forecasted through dedicated computer-aided tools. Consequently, this method stresses the usage of virtual environment to replicate both human postures and robot encumbrances over the manufacturing operations. Moreover, it proposes a safety index formulation for HRC systems based on the minimum distance between human and robot (H-R). As results, the approach returns the safety index for every possible combination of H-R operations. Subsequently, a scheduling algorithm suggests the operations sequence depending on the expected value of the safety index, providing an evaluation of the time needed to complete the process. The method is validated on surface control phase involved in post-processing of parts produced by laser powder bed fusion (L-PBF) Additive Manufacturing.

Driven by the evolving role of modern robots in collaboration with human operators, this paper proposes a virtual potential field-based motion planning and control strategy to realize effective human-robot collaboration through kinesthetic guidance via robot teleoperation. The motion of the teleoperated robot is determined by the human operator using a haptic device programmed to replicate movement under the influence of the virtual potential field constructed based on the obstacles and objectives in the environment. This approach facilitates seamless cooperation between the human and collaborative robot. It can also be readily generalized to include moving obstacles and goals in a dynamic environment. The results suggest that the virtual potential field can effectively enhance the safety and accuracy of human-robot cooperation. The developed method has promising applications in human-centric robotics fields such as medical robotics and robot-assisted surgeries.

Human-Robot Collaboration (HRC) is key to achieving the flexible automation required by the mass personalization trend, especially towards human-centric intelligent manufacturing. Nevertheless, existing HRC systems suffer from poor task understanding and poor ergonomic satisfaction, which impede empathetic teamwork skills in task execution. To overcome the bottleneck, a Mixed Reality (MR) and visual reasoning-based method is proposed in this research, providing mutual-cognitive task assignment for human and robotic agents’ operations. Firstly, an MR-enabled mutual-cognitive HRC architecture is proposed, with the characteristic of monitoring Digital Twins states, reasoning co-working strategies, and providing cognitive services. Secondly, a visual reasoning approach is introduced, which learns scene interpretation from the visual perception of each agent’s actions and environmental changes to make task planning strategies satisfying human–robot operation needs. Lastly, a safe, ergonomic, and proactive robot motion planning algorithm is proposed to let a robot execute generated co-working strategies, while a human operator is supported with intuitive task operation guidance in the MR environment, achieving empathetic collaboration. Through a demonstration of a disassembly task of aging Electric Vehicle Batteries, the experimental result facilitates cognitive intelligence in Proactive HRC for flexible automation.

In this paper, we present a software-based approach for collision avoidance that can be applied in human–robot collaboration scenarios. One of the contributions is a method for converting clustered 3D sensor data into computationally efficient convex hull representations used for robot-obstacle distance computation. Based on the computed distance vectors, we generate collision avoidance motions using a potential field approach and integrate them with other simultaneously running robot tasks in a constraint-based control framework. In order to improve control performance, we apply evolutionary techniques for parameter optimization within this framework based on selected quality criteria. Experiments are performed on a dual-arm robotic system equipped with several depth cameras. The approach is able to generate task-compliant avoidance motions in dynamic environments with high performance.

Novel human-robot collaboration (HRC) methods need careful validation. This is often achieved with user studies in laboratory environments, which mostly rely on highly individual, complex prototype setups in early design stages. Lately, the lack of replicability imposed by such experiments has vitally been discussed. In this paper, we contribute a benchmark toolkit to compose scalable, synthetic tasks as a unified basis for future efforts towards more replicable research. To this end, we pro-pose modular task boards which cover different domains and HRC scenarios. The design of benchmark tasks with these task boards is supported by a software application which generates structural task models and materials for benchmark problem reproduction by 3D printing. Our experiments show that these tasks can be carried out robustly by robots, hence preventing unintended robot failure and providing a controllable, reliably reproducible setting for user studies.

The human-robot collaboration (HRC) is an integral part of numerous research activities. The combination of the positive capabilities of human workers and robots ensures a high resource efficiency and productivity. However, especially the planning and implementation of HRC applications requires the consideration of several issues like the economic feasibility, acceptance of employees and in particular technical practicability. Based on the study of existing approaches which consider different technical or economic issues, we present a new method to identify suitable areas of HRC applications within assembly systems and to estimate the optimal allocation of human-related and automated subtasks. Finally, our approach is verified in an assembly line for terminal strips which in turn arises from an industrial use case.

Adaptation to the operator's intents and the external physical uncertainties is critical to the performances of the human–robot collaboration. In this study, we proposed a control interface that combined the electrical impedance tomography (EIT) based sensing approach with the robotic controllers to produce proper assistance with external uncertainties in the collaborative task. The interface first estimates the continuous forearm muscle contractions (represented by the grasp forces) by the optimized EIT features with an easily worn fabric band. The recognition decisions then serve as the control inputs to adjust the state transitions and the desired interaction forces in real time. We evaluated the interface in the tasks of grasp force estimation and human–robot sawing by recruiting more than 20 subjects in total. For grasp force estimation, the interface produced an average of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$R^{2}\geq 0.9$</tex-math></inline-formula> in both offline and online validations with the feature optimization procedure and a sigmoid regression function. The interface was robust to external disturbances without retraining or manual calibrations. The average <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$R^{2}=0.86$</tex-math></inline-formula> with the untrained dynamic postures, and the average <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$R^{2}$</tex-math></inline-formula> values ranged from 0.85 to 0.88 in the tests with redonning the front-end in interday and intraday uses. For human–robot sawing, the interface accomplished the tasks with a high success rate in controlling the states ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$&gt;$</tex-math></inline-formula> 96%) and intuitive adjusting of the sawing forces, being combined with the designed hybrid admittance/position controller. It was also adaptive to the sawing frequency changes and the sawing directions according to the operator's intents. The interface's performances are comparable, if not better, to the state of the art on both biological signal based grasp force estimation and human–robot sawing. Future efforts are worth being paid in this new direction to get more promising outcomes.

Human–robot interaction (HRI), in all its different variants, has gained significant attention in industry. The combination of human skills and robotics significantly enhances manufacturing, improving operations quality and efficiency. This is particularly relevant in highly variable contexts like remanufacturing, which aims to preserve the value of products and components over time and involves complex, non-standardized tasks such as inspection, disassembly, reprocessing, and reassembly. Uncertainties related to the conditions of products to remanufacture cannot be managed by pure automation making HRI highly beneficial. Although researchers have focused on the use of HRI in several fields of application (from construction to logistics), no extensive mapping of the research trends has specifically addressed the intersection of HRI and remanufacturing contexts. To fill this gap, this paper presents a comprehensive study based on a bibliometric analysis to explore the current state of research on this topic. Based on 120 articles from the Scopus database, this study points out the main countries, authors, and affiliations involved in the field. In addition, the key topics were analyzed. A predominant focus, above all in the last three years, on Human-Robot Collaboration (HRC) among the different levels of HRI, and disassembly processes was highlighted, while other related activities, such as reassembly, have received comparatively less attention. An unexplored frontier is the integration of Artificial Intelligence and Machine Learning with HRC tasks in remanufacturing. These findings offer valuable insights for researchers, scholars, and industrial professionals aiming to advance HRI in remanufacturing.

As the development of robotics technologies for collaborative robots (COBOTs), the applications of human–robot collaboration (HRC) have been growing in the past decade. Despite the tremendous efforts from both academia and industry, the overall usage and acceptance of COBOTs are still not so high as expected. One of the major affecting factors is the comfort of humans in HRC, which is usually less emphasized in COBOT development; however, it is critical to the user acceptance during HRC. Therefore, this paper gives a review of human comfort in HRC including the influential factors of human comfort, measurement of human comfort in terms of subjective and objective manners, and human comfort improvement approaches in the context of HRC. Discussions on each topic are also conducted based on the review and analysis.