Robust visual servoing: bounding the task function tracking errors

Visual servoing (VS) is an automatic control technique which uses the feedback of visual information to guide the robot motion to converge to the desired location. This paper proposes new moment-based features for robust VS with spheres, especially in cameras obeying the unified model. Specifically, we introduce a virtual plane to reload contours of image projections of spheres, in the process of which image noises will however be greatly magnified and induce large converge error. To solve this, new moment-based features are proposed and extracted on this virtual plane to enhance the robustness of VS to calibration errors and image noises. Simulation results verify the effectiveness of the proposed scheme and demonstrate the high robustness of these set of new features to image noises. The altering trends of converge errors versus increasing noise levels are randomly tested to present the obvious advantage of new moment-based features over the usual ones. At last, a path-planning based VS application is performed with fisheye views of some spheres, benefiting from the robustness and an interpolation-based tracking scheme.

This study presents the design and validation of a novel sampled-time state feedback controller for a multi-robotic system composed of a Cartesian robot and a Delta robot, providing five degrees of freedom – two from the Cartesian and three from the Delta robot. The novelty of this work lies in the integration of a machine learning-based visual servoing strategy with the discrete-time control design, where the controller's sampling period is carefully selected based on the computational demands of the vision-based feedback. The end-effector's pose is estimated in real time using a modified eye-to-hand YOLOv8 algorithm embedded within a neural network visual servoing system. This approach enables the robotic system to continuously update its control actions based on visual feedback, enhancing trajectory tracking accuracy during task execution. A key contribution of this work is the investigation of the impact of sampling time on the convergence quality of the end-effector's tracking error. The proposed control architecture was rigorously evaluated in a virtual environment, where the end-effector pose estimated by the visual system was compared against ground truth data. The experimental results demonstrate that adjusting the sampling time significantly improves control accuracy, thereby justifying the proposed discrete-time controller design. This integrated machine learning and sampled-time control approach shows promising potential for real-time, vision-guided robotic manipulation. Highlights A synthetic database for the proposed controller using the Segment Anything Model (SAM) for image processing. A CNN algorithm to detect the position and orientation of the end-effector of a composite robotic manipulator based on a sequential configuration of an open-closed kinematic chain. A sampled controller that drives the position/orientation of the composite robotic manipulator to time-varying specific target configurations, including its convergence analysis.

In this article, a robust nonlinear model predictive control (NMPC) scheme is proposed for the visual servoing of quadrotors subject to external disturbances. By using the virtual camera approach, the image moments are defined in the virtual camera plane and adopted as visual features to derive the decoupled image kinematics. As a result, the image-based visual servoing (IBVS) system model is established by integrating the image kinematics and quadrotor dynamics. To handle the visibility constraint, a robust NMPC scheme is developed for the IBVS of the quadrotor such that the visual target can stay within the field of view of the camera. In addition, based on the Lipschitz condition, the tightened state constraints are constructed to tackle external disturbances. The sufficient conditions on guaranteeing recursive feasibility of the proposed NMPC algorithm are derived. Furthermore, we theoretically show that the tracking error will converge to a small set around the origin in finite time under some derived conditions. Finally, simulation studies and experimental tests are conducted to verify the efficacy of the proposed method.

This thesis deals with Visual Servoing and its strictly connected disciplines like projective geometry, image processing, robotics and non-linear control. More specifically the work addresses the problem to control a robotic manipulator through one of the largely used Visual Servoing techniques: the Image Based Visual Servoing (IBVS). In Image Based Visual Servoing the robot is driven by on-line performing a feedback control loop that is closed directly in the 2D space of the camera sensor. The work considers the case of a monocular system with the only camera mounted on the robot end effector (eye in hand configuration). Through IBVS the system can be positioned with respect to a 3D fixed target by minimizing the differences between its initial view and its goal view, corresponding respectively to the initial and the goal system configurations: the robot Cartesian Motion is thus generated only by means of visual informations. However, the execution of a positioning control task by IBVS is not straightforward because singularity problems may occur and local minima may be reached where the reached image is very close to the target one but the 3D positioning task is far from being fulfilled: this happens in particular for large camera displacements, when the the initial and the goal target views are noticeably different. To overcame singularity and local minima drawbacks, maintaining the good properties of IBVS robustness with respect to modeling and camera calibration errors, an opportune image path planning can be exploited. This work deals with the problem of generating opportune image plane trajectories for tracked points of the servoing control scheme (a trajectory is made of a path plus a time law). The generated image plane paths must be feasible i.e. they must be compliant with rigid body motion of the camera with respect to the object so as to avoid image jacobian singularities and local minima problems. In addition, the image planned trajectories must generate camera velocity screws which are smooth and within the allowed bounds of the robot. We will show that a scaled 3D motion planning algorithm can be devised in order to generate feasible image plane trajectories. Since the paths in the image are off-line generated it is also possible to tune the planning parameters so as to maintain the target inside the camera field of view even if, in some unfortunate cases, the feature target points would leave the camera images due to 3D robot motions. To test the validity of the proposed approach some both experiments and simulations results have been reported taking also into account the influence of noise in the path planning strategy. The experiments have been realized with a 6DOF anthropomorphic manipulator with a fire-wire camera installed on its end effector: the results demonstrate the good performances and the feasibility of the proposed approach.

This paper considers image-guided assembly for large composite panels. By using fiducial markers on the panels and robot gripper mounted cameras, we are able to use an industrial robot to align the panels to sub-millimeter accuracy. We considered two commonly used visual servoing schemes: position-based visual servoing (PBVS) and image-based visual servoing (IBVS). It has been noted that IBVS possesses superior robustness with respect to the camera calibration accuracy. However, we have found that in our case, PBVS is both faster and slightly more accurate than IBVS. This result is due to the fact that the visual servoing target in the image plane is derived from a reference target, which depends on the accuracy of the camera model. This additional dependency essentially nullifies the robustness advantage of IBVS. We also implemented a simple scheme to combine inputs from multiple cameras to improve the visual servoing accuracy. Both simulation and experimental results are included to show the effectiveness of visual servoing in an industrial setting.

In this paper, we present a robot positioning task with respect to a static target using visual servoing and optimization techniques. The vision system is uncalibrated and the kinematic model of the robot may be totally unknown. The displacements of the robot are generated in real time in order to minimize an objective function. The objective function includes the quadratic error between the current target image and the desired target image. A simplex method is used to minimize the objective function. Our method allows the system to include constraints in the image as well as in joint-space. We successfully validate this method, with simulations under the graphic library QpenGL and with experiments on a 6-DOF industrial manipulator.

In image-based visual servoing control, image change is directly interpreted to camera motion, there isnpsilat direct control over the Cartesian velocities of the robot end-effector. As a result, the robot executes trajectories can be indirect or seemingly roundabout in Cartesian space. This paper proposes a new control scheme that overcomes the problem of the conventional visual servoing. The proposed approach separates the rotation in z-axis from general motion, and control robot to rotate firstly, till current image feature is coincident in direction with goal image feature, then control robot to execute the rest motion by common method. As there is not interferences of translation and rotation of image, the performance of this approach is more advantage in the setting times and the generating optimal motion trajectories of Cartesian space than classical IBVS. We illustrate new control scheme with a variety of simulation results.

This paper presents a review of characteristics of the two main visual servoing methods: image-based visual servoing and position-based visual servoing. First, a generic structure is defined with an emphasis on the same system inputs and outputs for the two methods. The stability, robustness, sensitivity, and other general characteristics of the methods are presented. To overcome the dependency of both methods on a known target model, a midpoint triangulation method is proposed to estimate the unknown target model for robot visual servoing.

In this paper, an image-based impedance control strategy for force tracking of an unmanned aerial manipulator (UAM) is presented. Firstly, image features with nice decoupling characteristics are designed and the relationship between the camera motion and the image features is derived. Then, a two-stage strategy is proposed to achieve force tracking of the UAM on a planar object in an arbitrary pose. The first stage drives the end-effector perpendicular to the object’s planer surface by pure visual servoing. To achieve force tracking under the visual guidance, an adaptive visual impedance control method which adjusts the target stiffness according to the force tracking error and the visual feature error is proposed in the second stage. The closed-loop system is proved asymptotically stable by means of Lyapunov analysis. Further, the stability in free flight phase of the stage two is also analyzed and ensured. Finally, experiments were carried out including a whiteboard cleaning task in different poses. The experimental results illustrate the validity and effectiveness of the proposed approach. Note to Practitioners—This work is motivated by the contact force tracking problem of an unmanned aerial manipulator (UAM) without the position measurement. In recent years, impedance control is widely used in force tracking problems. However, position measurement is needed both for the robot and the object in most studies. To generate desired force under visual guidance, image-based visual servoing is combined with a novel impedance controller with variable stiffness. The idea is intuitive. For a human in contact with a wall, the contact force is controlled by adjusting his arm stiffness, which means the stiffness is adapted to the difference between the desired and actual contact force. Furthermore, the stiffness is also adapted to the visual tracking error, removing the assumption that no tracking error is in the inner loop of the impedance controller in the previous works. The proposed strategy is a promising solution for real applications and is validated by a board cleaning experiment in this paper.

For soft continuum arms, visual servoing is a popular control strategy that relies on visual feedback to close the control loop. However, robust visual servoing is challenging as it requires reliable feature extraction from the image, accurate control models and sensors to perceive the shape of the arm, both of which can be hard to implement in a soft robot. This letter circumvents these challenges by presenting a deep neural network-based method to perform smooth and robust 3D positioning tasks on a soft arm by visual servoing using a camera mounted at the distal end of the arm. A convolutional neural network is trained to predict the actuations required to achieve the desired pose in a structured environment. Integrated and modular approaches for estimating the actuations from the image are proposed and are experimentally compared. A proportional control law is implemented to reduce the error between the desired and current image as seen by the camera. The model together with the proportional feedback control makes the described approach robust to several variations such as new targets, lighting, loads, and diminution of the soft arm. Furthermore, the model lends itself to be transferred to a new environment with minimal effort.

Classical visual servoing approaches tend to constrain all degrees of freedom (DOF) of the robot during a task's execution. In this article a new approach is proposed. The key idea is to control the robot with a very under-constrained task when it is far of the desired position, and to incrementally constrain the global task by adding further tasks as the robot moves closer to the goal. A method is first proposed that stacks elementary tasks until the robot is fully constrained. To insure the continuity of the articular velocities when adding constraints, a new control law is then proposed. Experiments that prove the interest of the approach are also provided.

Image-based visual servoing for nonholonomic mobile robots using epipolar geometry is an efficient technology for visual servoing problem. An improved visual servoing strategy, namely, three-step epipolar-based visual servoing, is developed for the nonholonomic robot in this paper. The proposed strategy can keep the robot meeting FOV constraint without any 3D reconstruction. Moreover, the trajectory planned by this strategy is shorter than the existing strategies. The mobile robot can reach the desired configuration with exponential converge. The control scheme in this paper is divided into three steps. Firstly, by using the difference of epipoles as feedback, the robot rotates to make the current configuration and desired configuration in the same orientation. Then, by using a linear input-output feedback, the epipoles are zeroed so as to align the robot with the goal. Finally, by using the difference of feature points, the robot reaches the desired configuration. Simulation results and experimental results are given to illustrate the effectiveness of the proposed control scheme.

We present an approach to dynamic sensor planning problems in visual servoing. Specifically, one of the main problems an image-based visual servoing is to plan the camera trajectory in order to avoid undesired configurations (e.g., features out of view, collision with obstacles, etc.). Our approach uses the robot redundancy and employs a control scheme based on the task function approach. It combines the regulation of the selected vision-based task with the minimization of a secondary cost function, which reflects given constraints on the manipulator trajectory. We describe how this methodology is applied to common problems in robotic vision: occlusion avoidance, field of view constraint and obstacle avoidance. We demonstrate the validity of this approach with various experiments.

Background: Previous research has suggested separable short-term memory (STM) buffers for the maintenance of phonological and lexical-semantic information, as some patients with aphasia show better ability to retain semantic than phonological information and others show the reverse. Recently researchers have proposed that deficits to the maintenance of semantic information in STM are related to executive control abilities. Aims: The present study investigated the relationship of executive function abilities with semantic and phonological short-term memory (STM) and semantic processing in such patients, as some previous research has suggested that semantic STM deficits and semantic processing abilities are critically related to specific or general executive function deficits. Method & Procedures: A total of 20 patients with aphasia and STM deficits were tested on measures of short-term retention, semantic processing, and both complex and simple executive function tasks. Outcome & Results: In correlational analyses we found no relation between semantic STM and performance on simple or complex executive function tasks. In contrast, phonological STM was related to executive function performance in tasks that had a verbal component, suggesting that performance in some executive function tasks depends on maintaining or rehearsing phonological codes. Although semantic STM was not related to executive function ability, performance on semantic processing tasks was related to executive function, perhaps due to similar executive task requirements in both semantic processing and executive function tasks. Conclusions: Implications for treatment and interpretations of executive deficits are discussed.

Neither of the classical visual servoing approaches, position-based and image-based, are completely satisfactory. In position-based visual servoing the trajectory of the robot is well stated, but the approach suffers mainly from the image features going out of the visual field of the cameras. On the other hand, image-based visual servoing has been found generally satisfactory and robust in the presence of camera and hand-eye calibration errors. However, in some cases, singularities and local minima may arise, and the robot can go further from its joint limits. This paper is a step towards the synthesis of both approaches with their particular advantages, i.e., the trajectory of the camera motion is predictable and the image features remain in the field of view of the camera. The basis is the introduction of three-dimensional information in the feature vector. Point depth and object pose produce useful behavior in the control of the camera. Using the task-function approach, we demonstrate the relationship between the velocity screw of the camera and the current and desired poses of the object in the camera frame. Camera calibration is assumed, at least coarsely. Experimental results on real robotic platforms illustrate the presented approach.