Hand-eye calibration method and machine vision research based on sensor network

Augmented reality (AR) allows the surgeon to represent holographic patient-specific anatomical information and surgical instruments in the physical world. To correctly superimpose virtual and physical objects, a hand-eye (HE) calibration method for mapping the virtual and physical spaces was proposed. Mathematical relationships between the virtual camera and the physical space were derived. Finally, the accuracy and robustness of the proposed HE calibration method were qualitatively and quantitatively evaluated. The proposed calibration method allows us to determine an optimal invariant spatiotemporal mapping between the virtual camera and the physical space. Qualitatively and quantitatively reliable and accurate estimates for the physical-virtual mapping transformation were verified. Consequently, imaging data and surgical instruments holograms can be precisely represented in the physical space.

Hand-eye calibration, a fundamental task in vision-based robotic systems, is commonly equipped with collaborative robots, especially for robotic applications in small and medium-sized enterprises (SMEs). Most approaches to hand-eye calibration rely on external markers or human assistance. We proposed a novel methodology that addresses the hand-eye calibration problem using the robot base as a reference, eliminating the need for external calibration objects or human intervention. Using point clouds of the robot base, a transformation matrix from the coordinate frame of the camera to the robot base is established as “I=AXB.” To this end, we exploit learning-based 3D detection and registration algorithms to estimate the location and orientation of the robot base. The robustness and accuracy of the method are quantified by ground-truth-based evaluation, and the accuracy result is compared with other 3D vision-based calibration methods. To assess the feasibility of our methodology, we carried out experiments utilizing a low-cost structured light scanner across varying joint configurations and groups of experiments. The proposed hand-eye calibration method achieved a translation deviation of 0.930 mm and a rotation deviation of 0.265 degrees according to the experimental results. Additionally, the 3D reconstruction experiments demonstrated a rotation error of 0.994 degrees and a position error of 1.697 mm. Moreover, our method offers the potential to be completed in 1 second, which is the fastest compared to other 3D hand-eye calibration methods. We conduct indoor 3D reconstruction and robotic grasping experiments based on our hand-eye calibration method. Related code is released at https://github.com/leihui6/LRBO.

Aiming at the problem of low accuracy of industrial robot hand-eye calibration, a robot hand-eye calibration method based on an intermediate measurement system is proposed. The checkerboard calibration board with the marker points pasted is photographed by the camera, and the coordinates of the marker points in the checkerboard calibration board coordinate system are calculated according to the principle of cross-ratio invariance, and the coordinate system matrix between the C-Track optical dynamic tracking system and the industrial robot is established. The transformation relationship finally obtains the closed-loop coordinate system matrix transformation equation with the marker point, the checkerboard calibration board and the C-Track optical dynamic tracking system as the intermediate measurement system, thereby avoiding the introduction of the mobile industrial robot manipulator in the traditional hand-eye calibration process Kinematic error. The experimental results show that the translation error in the hand-eye relationship matrix is 0.6 mm, and the rotation error is 0.3°. Compared with the traditional hand-eye calibration method, this method improves the accuracy of hand-eye calibration, simplifies the hand-eye calibration process, and meets the needs of actual processing and production.

To solve the problem of inflexibility of offline hand–eye calibration in “eye-in-hand” modes, an online hand–eye calibration method based on the ChArUco board is proposed in this paper. Firstly, a hand–eye calibration model based on the ChArUco board is established, by analyzing the mathematical model of hand–eye calibration, and the image features of the ChArUco board. According to the advantages of the ChArUco board, with both the checkerboard and the ArUco marker, an online hand–eye calibration algorithm based on the ChArUco board is designed. Then, the online hand–eye calibration algorithm, based on the ChArUco board, is used to realize the dynamic adjustment of the hand–eye position relationship. Finally, the hand–eye calibration experiment is carried out to verify the accuracy of the hand–eye calibration based on the ChArUco board. The robustness and accuracy of the proposed method are verified by online hand–eye calibration experiments. The experimental results show that the accuracy of the online hand–eye calibration method proposed in this paper is between 0.4 mm and 0.6 mm, which is almost the same as the offline hand–eye calibration accuracy. The method in this paper utilizes the advantages of the ChArUco board to realize online hand–eye calibration, which improves the flexibility and robustness of hand–eye calibration.

This letter describes a novel hand–eye calibration technique based on reprojection error minimization. In contrast to traditional hand–eye calibration methods, the proposed method directly takes images of the calibration pattern and does not require to explicitly estimate the camera pose for each input image. The proposed method is implemented as a pose graph optimization problem, so that it can solve the estimation problem efficiently and robustly, and it can be easily extended for different projection models. It can deal with different camera models (e.g, X-ray cameras with a source-detector projection model) by changing the projection model. Through simulations, we validated that the proposed method shows a good estimation accuracy, and it can be applied to hand–eye calibration with a source-detector camera model. The experimental results with real robots show that the proposed method is applicable to real environments, and it improves the quality of a task that requires accurate hand–eye estimation, such as three-dimensional reconstruction.

Received 16 March 2015; received in revised form 14 April 2015; accepted 14 April 2015ABSTRACTIn this paper, we propose calibration methods that can be applied to the markerless surgicalrobotic system for Intracerebral Hematoma (ICH) Surgery. This surgical robotic system does notrequire additional process of patient imaging but only uses CT images that are initially taken fora diagnosis purpose. Furthermore, the system applies markerless registration method other thanusing stereotactic frames. Thus, in overall, our system has many advantages when compared toother conventional ICH surgeries in that they are non-invasive, much less exposed to radiationexposure, and most importantly reduces a total operation time. In the paper, we specificallyfocus on the application of calibration methods and their verification which is one of the mostcritical factors that determine the accuracy of the system. We implemented three applications ofcalibration methods between the coordinates of robot’s end-effector and the coordinates of 3Dfacial surface scanner, based on the hand-eye calibration method. Phantom tests were conductedto validate the feasibility and accuracy of our proposed calibration methods and the surgicalrobotic system.Key Words:Hand-eye calibration, ICH surgery, Markerless registration, Stereotactic device,Surgical robot

Balance reactions and eye–hand coordination in idiopathic scoliosis

Robotic hand-eye coordination plays an important role in dealing with real time environment; and the learning procedure of this skill affects the fundamental framework of robotic cognition. Theories from developmental psychology have motivated the appearance of cognitive developmental robotics; however, several important features within infant developmental procedure have not been introduced into researches on developmental robotics. This paper proposes a new approach to robotic hand-eye coordination by imitating the developmental progress of human infant. The work employs a brain-like neural network system to learn hand-eye coordination. The learning procedure is driven by developmental constraints, the robot starts to learn under fully constrained conditions, when the system becomes saturated, a new constraint is assigned to the robot; the robot needs to learn this new conditions again, until the all contained conditions have been overcome. The work is supported by experimental evaluation, which shows that the new approach is able to drive the robot to own the hand-eye coordination ability autonomously, and the robot also behaves similar developmental progress with human infants'.

An accurate and reliable estimation of the transformation matrix between an optical sensor and a robot is a key aspect of the hand–eye system calibration process in vision-guided robotic applications. This paper presents a novel approach to markerless hand–eye calibration that achieves streamlined, flexible, and highly accurate results, even without error compensation. The calibration procedure is mainly based on using the robot’s tool center point (TCP) as the reference point. The TCP coordinate estimation is based on the robot’s flange point cloud, considering its geometrical features. A mathematical model streamlining the conventional marker-based hand–eye calibration is derived. Furthermore, a novel algorithm for the automatic estimation of the flange’s geometric features from its point cloud, based on a 3D circle fitting, the least square method, and a nearest neighbor (NN) approach, is proposed. The accuracy of the proposed algorithm is validated using a calibration setting ring as the ground truth. Furthermore, to establish the minimal required number and configuration of calibration points, the impact of the number and the selection of the unique robot’s flange positions on the calibration accuracy is investigated and validated by real-world experiments. Our experimental findings strongly indicate that our hand–eye system, employing the proposed algorithm, enables the estimation of the transformation between the robot and the 3D scanner with submillimeter accuracy, even when using the minimum of four non-coplanar points for calibration. Our approach improves the calibration accuracy by approximately four times compared to the state of the art, while eliminating the need for error compensation. Moreover, our calibration approach reduces the required number of the robot’s flange positions by approximately 40%, and even more if the calibration procedure utilizes just four properly selected flange positions. The presented findings introduce a more efficient hand–eye calibration procedure, offering a superior simplicity of implementation and increased precision in various robotic applications.

This paper presents a novel developmental learning approach for hand-eye coordination in an autonomous robotic system. Robotic hand-eye coordination plays an important role in dealing with real-time environments. Under the approach, infant developmental patterns are introduced to build our robot's learning system. The method works by first constructing a brain-like computational structure to control the robot, and then by using infant behavioural patterns to build a hand-eye coordination learning algorithm. This work is supported by an experimental evaluation, which shows that the control system is implemented simply, and that the learning approach provides fast and incremental learning of behavioural competence.

This paper presents a novel method for so-called hand-eye calibration. Using a calibration target is not possible for many applications of hand-eye calibration. In such situations Structure-from-Motion approach of hand-eye calibration is commonly used to recover the camera poses up to scaling. The presented method takes advantage of recent results in the L <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">∞</sub> -norm optimization using Second-Order Cone Programming (SOCP) to recover the correct scale. Further, the correctly scaled displacement of the hand-eye transformation is recovered solely from the image correspondences and robot measurements, and is guaranteed to be globally optimal with respect to the L <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">∞</sub> -norm. The method is experimentally validated using both synthetic and real world datasets.

The explosive ordnance disposal (EOD) robots work in a special environment, which requires dual robotic arms to work together for removing the bomb. Therefore, machine vision is very important to locating bombs, especially the accuracy of the hand-eye calibration technology. A basic problem encountered with the collaborative work of dual robotic arms is to solve the unknown homogeneous transformation matrix, including: hand-eye of robotic arm 1, base-base, and camera-end effector’s robotic arm 2. In this article, the hand-eye calibration problem of the dual robotic arm system is expressed as two matrix equations. A new method of simultaneously solving the unknowns in the matrix equation is proposed. This method consists of a closed-form method based on the Kronecker product and an iterative method that transforms a nonlinear problem with a convex function optimization problem. The closed-form method is used to quickly obtain the initial value of the iteration method to improve the efficiency and accuracy of the iteration. In addition, we propose a hand-eye calibration method based on the re-projection error of the RGB-D camera. In order to prove the feasibility and superiority in the proposed iterative method, we conducted simulation and actual experiments and compared them with the other two calibration methods. The comparison results verify the superiority in the proposed method in terms of accuracy.

Background Stroke is a leading cause of disability, often accompanied by unilateral spatial neglect (USN), which severely impairs recovery. Traditional assessments like paper-pencil tests provide limited insights into behaviors and eye–hand coordination during real-world tasks. Advances in hand pose estimation and eye tracking in combination with augmented reality (AR) offer potential for data-driven assessments of naturalistic interactions. Objective This proof-of-concept study presents and evaluates a multimodal behavioral tracking system that captures gaze, body, and hand movements during interactions within an AR environment. Our primary goals are to (1) validate that this system can achieve robust and accurate interaction data capture in clinical settings, (2) show that the system can reliably detect known USN behavioral patterns, and (3) explore how comprehensive data can provide new understanding of eye–hand coordination deficits in USN. Methods We developed an AR-based assessment system using Microsoft HoloLens 2 and an external body-tracking camera to capture real-time gaze, hand, and body movements in an interactive environment. Multimodal data streams were temporally synchronized, fused, and filtered to enhance spatial accuracy and availability. Tracking performance was benchmarked against a traditional optical motion-capture system to validate reliability. In a study, 7 patients with right-brain lesions with mild to moderate USN and 8 healthy controls participated. Each performed a designed reaching task, stamping virtual sheets of paper that appeared randomly on a table. We analyzed participants’ search behavior patterns to assess attentional biases and examined gaze anchoring timing during targeted reaching motions to explore potential eye–hand coordination deficits. Results The fusion of hand-tracking data from the HoloLens 2 and external system reduced tracking loss from 25.7% to 2.4%, with an absolute trajectory error of 3.27 cm. The system demonstrated high usability and was well accepted by patients. Data from the control group confirmed the absence of intrinsic lateral biases in the system and task design. The USN group displayed typical search behavior through ipsilesional biases in gaze direction during visual exploration (median deviation 7.46 [1.61-9.48] deg, P &lt;.05) and longer times to find contralesional targets (median difference 1.08 [0.20-1.80] s, P =.02). Additionally, the eye–hand coordination analysis revealed lateral differences in gaze anchoring during targeted reaching motions in the USN group, with earlier fixation on contralesional targets (median difference 112 [71-146] ms, P =.02). Conclusions The proposed AR framework provides a novel, comprehensive data-driven method for capturing interaction behavior in a controlled, yet naturalistic environment. Our results demonstrated the system’s effectiveness in measuring hallmark USN symptoms, such as gaze and head orientation biases, and highlighted its potential to complement traditional assessments by offering deep insights into torso rotation and eye–hand coordination with a high resolution and accuracy. This data-driven approach shows promise for enhancing current USN assessment practices and gaining new insights into patients’ behaviors.

Eye–hand co-ordination skills in very preterm infants < 29 weeks gestation at 3 years: effects of preterm birth and retinopathy of prematurity

Robot hand-eye coordination has recently enjoyed much attention. Previous research at MIT has examined combining vision and manipulation applied to the task of tracking and catching tossed balls in controlled environments. Building upon the foundations of this past research, this paper presents work which incorporates a new active vision system which requires a minimally controlled environment, and implements methods for object tracking, robot/camera calibration, and new catching algorithms. Experimental results for real time catching of free-flying spherical balls are presented. The system was tested on under-hand tosses from random locations approximately 1.5–2.5 meters distant from the base of the arm. The best performance results were found to be 70–80% success for similar tosses.