Improving Image-Based Visual Servoing with Three-Dimensional Features

This thesis deals with Visual Servoing and its strictly connected disciplines like projective geometry, image processing, robotics and non-linear control. More specifically the work addresses the problem to control a robotic manipulator through one of the largely used Visual Servoing techniques: the Image Based Visual Servoing (IBVS). In Image Based Visual Servoing the robot is driven by on-line performing a feedback control loop that is closed directly in the 2D space of the camera sensor. The work considers the case of a monocular system with the only camera mounted on the robot end effector (eye in hand configuration). Through IBVS the system can be positioned with respect to a 3D fixed target by minimizing the differences between its initial view and its goal view, corresponding respectively to the initial and the goal system configurations: the robot Cartesian Motion is thus generated only by means of visual informations. However, the execution of a positioning control task by IBVS is not straightforward because singularity problems may occur and local minima may be reached where the reached image is very close to the target one but the 3D positioning task is far from being fulfilled: this happens in particular for large camera displacements, when the the initial and the goal target views are noticeably different. To overcame singularity and local minima drawbacks, maintaining the good properties of IBVS robustness with respect to modeling and camera calibration errors, an opportune image path planning can be exploited. This work deals with the problem of generating opportune image plane trajectories for tracked points of the servoing control scheme (a trajectory is made of a path plus a time law). The generated image plane paths must be feasible i.e. they must be compliant with rigid body motion of the camera with respect to the object so as to avoid image jacobian singularities and local minima problems. In addition, the image planned trajectories must generate camera velocity screws which are smooth and within the allowed bounds of the robot. We will show that a scaled 3D motion planning algorithm can be devised in order to generate feasible image plane trajectories. Since the paths in the image are off-line generated it is also possible to tune the planning parameters so as to maintain the target inside the camera field of view even if, in some unfortunate cases, the feature target points would leave the camera images due to 3D robot motions. To test the validity of the proposed approach some both experiments and simulations results have been reported taking also into account the influence of noise in the path planning strategy. The experiments have been realized with a 6DOF anthropomorphic manipulator with a fire-wire camera installed on its end effector: the results demonstrate the good performances and the feasibility of the proposed approach.

A review on visual servoing for underwater vehicle manipulation systems automatic control and case study

This paper considers image-guided assembly for large composite panels. By using fiducial markers on the panels and robot gripper mounted cameras, we are able to use an industrial robot to align the panels to sub-millimeter accuracy. We considered two commonly used visual servoing schemes: position-based visual servoing (PBVS) and image-based visual servoing (IBVS). It has been noted that IBVS possesses superior robustness with respect to the camera calibration accuracy. However, we have found that in our case, PBVS is both faster and slightly more accurate than IBVS. This result is due to the fact that the visual servoing target in the image plane is derived from a reference target, which depends on the accuracy of the camera model. This additional dependency essentially nullifies the robustness advantage of IBVS. We also implemented a simple scheme to combine inputs from multiple cameras to improve the visual servoing accuracy. Both simulation and experimental results are included to show the effectiveness of visual servoing in an industrial setting.

This paper addresses the challenges of choosing proper image features for planar symmetric shape objects and designing visual servoing controller to enhance the tracking performance in image-based visual servoing (IBVS). Six image moments are chosen as the image features and the analytical image interaction matrix related to the image features are derived. A controller is designed to efficiently increase the robustness of the visual servoing system. Experimental results on a 6-DOF robot visual servoing system are provided to illustrate the effectiveness of the proposed method.

In this paper, Neural Network (NN)-based image moments are proposed to address the challenges of choosing proper image features for Image-based Visual Servoing (IBVS). The proposed two NN-based image features can estimate the rotational angles around x and y axes of camera frame for planar objects. Based on the proposed image features and other 4 commonly used image moments, the interaction matrix relating the chosen image features to camera motion is derived to have maximal decoupled structure. In addition, the analytical form of depth computation is given by using classical geometrical primitives and image moment invariant. A proportional IBVS controller is designed based on the derived interaction matrix and the tracking performance is thus enhanced for the 6 degree-of-freedom robotic system. Simulation results on a 6-DOF robot system are provided to illustrate the effectiveness of the proposed method.

This paper presents a distributed fuzzy proportional control system for a vision guided redundant robot manipulator. Firstly, the Takagi Sugeno (TS) fuzzy algorithm is used to model analytical Image Based Visual Servoing (IBVS) using shape moments by offline learning. This control method is applied to an uncalibrated robotic system with eye-in-hand visual feedback. The system is able to track a moving object through a variety of motions and maintain the object's image features in a desired position in the image plane without a priori knowledge of the robot kinematic, camera calibration and inverse Jacobian. The experimental results of both static and moving targets using the 7 DOF PowerCube manipulator from Amtec Robotic show and verify its performance in a realtime application.

U ovom preglednom radu su prikazane metode vizuelnog upravljanja robotskih sistema, sa primarnim fokusom na mobilne robote sa diferencijalnim pogonom. Analizirane su standardne metode vizuelnog upravljanja bazirane na (i) greškama u parametrima slike (engl. Image-Based Visual Servoing - IBVS) i (ii) izdvojenim karakteristikama sa slike neophodnim za estimaciju položaja izabranog objekta (engl. Position-Based Visual Servoing - PBVS) i poređene sa novom metodom direktnog vizuelnog upravljanja (engl. Direct Visual Servoing - DVS). U poređenju sa IBVS i PBVS metodama, DVS metod se odlikuje višom tačnošću, ali i manjim domenom konvergencije. Zbog ovog razloga je DVS metod upravljanja pogodan za integraciju u hibridne sisteme vizuelnog upravljanja. Takođe, predstavljeni su radovi koji unapređuju sistem vizuelnog upravljanja korišćenjem stereo sistema (sistem sa dve kamere). Stereo sistem, u poređenju sa alternativnim metodama, omogućava tačniju ocenu dubine karakterističnih objekata sa slike, koja je neophodna za zadatke vizuelnog upravljanja. Predmet analize su i radovi koji integrišu tehnike veštačke inteligencije u sistem vizuelnog upravljanja. Ovim tehnikama sistemi vizuelnog upravljanja dobijaju mogućnost da uče, čime se njihov domen primene znatno proširuje. Na kraju, napominje se i mogućnost integracije vizuelne odometrije u sisteme vizuelnog upravljanja, što prouzrokuje povećanje robusnosti čitavog robotskog sistema.

The visual servoing technique has been studied extensively as a robust method for navigating robots toward goal positions and orientations reliably. Unfortunately, the existing visual servoing methods require the calibration of cameras and robots, which is time-consuming. Thus, in this paper, we propose a visual servoing method, which does not require the calibration of cameras and robots. In particular, we show that we can navigate uncalibrated robots to goal positions properly by using the projective reconstruction based on the abstract projection of control values.

This study proposes an optimized hybrid visual servoing approach to overcome the imperfections of classical two-dimensional, three-dimensional and hybrid visual servoing methods. These imperfections are mostly convergence issues, non-optimized trajectories, expensive calculations and singularities. The proposed method provides more efficient optimized trajectories with shorter camera path for the robot than image-based and classical hybrid visual servoing methods. Moreover, it is less likely to lose the object from the camera field of view, and it is more robust to camera calibration than the classical position-based and hybrid visual servoing methods. The drawbacks in two-dimensional visual servoing are mostly related to the camera retreat and rotational motions. To tackle these drawbacks, rotations and translations in Z-axis have been separately controlled from three-dimensional estimation of the visual features. The pseudo-inverse of the proposed interaction matrix is approximated by a neuro-fuzzy neural network called local linear model tree. Using local linear model tree, the controller avoids the singularities and ill-conditioning of the proposed interaction matrix and makes it robust to image noises and camera parameters. The proposed method has been compared with classical image-based, position-based and hybrid visual servoing methods, both in simulation and in the real world using a 7-degree-of-freedom arm robot.

Visual servoing systems are designed to solve pose alignment problems by providing the necessary linear and angular velocities using data extracted from images. Among the difficulties encountered by the traditional visual servoing approaches, there are feature detection and tracking, camera calibration, scene complexity, and robotic system constraints. Part of these problems can be solved if Convolutional Neural Networks (CNNs) are added to a visual servoing architecture. The main advantage of CNNs is the capability of understanding both the overall structure and specific details of the images corresponding to the current and desired layouts. To take a step further the state-of-the-art architectures, in this paper, we show how extra input data can improve the visual servoing behaviour. The extra data result from maps of regions induced by the feature points' positions, without the necessity of employing tracking. The results obtained on relevant data sets show that the proposed input fusion-based CNN provides an improved approximation of the linear and angular visual servoing velocities.

This paper presents a self-memory prediction model to mitigate the effects of image based visual servoing (IBVS) system under uncertainty. The performance of IBVS system is easily influenced by different tasks, diverse environments and uncertain disturbances. Through building a self-memory prediction model to keep previous movement tendency in the every current movement, the framework of a self-memory is proposed for incremental self-learning capability. The main challenge in IBVS system is avoid abrupt change in the operation of the system. The proposed approach is processed from state pattern, and introduces the memory function which contain multiple historical observed values. According to past events, predict the current state and avoid abrupt changes in the every current movement. Simulations on IBVS system illustrate the effectiveness of the proposed method.

Prosthesis is a device that replaces a missing body part. These devices are used to reinstate the lost functionality due to an amputation. Above elbow amputees are equipped with trans-humeral prostheses. The existing prostheses use biological signals such as electromyography (EMG) in combine with external sensors for the controlling purposes. Vision is the latest addition to external sensor domain to eliminate the drawbacks of existing hybrid prostheses. This paper proposes a vision based task planning method in combination with available EMG signals to plan reach-to-grasp tasks of a trans-humeral prosthesis. The visual servoing system consists of an Image Based Visual Servoing (IBVS) system in conjunction with an Artificial Neural Network (ANN) to center the object in the image frame. Moreover, a gradient descent algorithm is proposed for the elbow motion to reach the palm towards the object of interest. EMG based system is capable of controlling the prosthetic elbow motion with available muscles' signals. EMG based system is fused with the IBVS system to perform elbow motions when an object is visible to the camera. Experiments are carried out using a virtual prosthesis to validate the proposed task planning method.

In this paper, an image-based impedance control strategy for force tracking of an unmanned aerial manipulator (UAM) is presented. Firstly, image features with nice decoupling characteristics are designed and the relationship between the camera motion and the image features is derived. Then, a two-stage strategy is proposed to achieve force tracking of the UAM on a planar object in an arbitrary pose. The first stage drives the end-effector perpendicular to the object’s planer surface by pure visual servoing. To achieve force tracking under the visual guidance, an adaptive visual impedance control method which adjusts the target stiffness according to the force tracking error and the visual feature error is proposed in the second stage. The closed-loop system is proved asymptotically stable by means of Lyapunov analysis. Further, the stability in free flight phase of the stage two is also analyzed and ensured. Finally, experiments were carried out including a whiteboard cleaning task in different poses. The experimental results illustrate the validity and effectiveness of the proposed approach. Note to Practitioners—This work is motivated by the contact force tracking problem of an unmanned aerial manipulator (UAM) without the position measurement. In recent years, impedance control is widely used in force tracking problems. However, position measurement is needed both for the robot and the object in most studies. To generate desired force under visual guidance, image-based visual servoing is combined with a novel impedance controller with variable stiffness. The idea is intuitive. For a human in contact with a wall, the contact force is controlled by adjusting his arm stiffness, which means the stiffness is adapted to the difference between the desired and actual contact force. Furthermore, the stiffness is also adapted to the visual tracking error, removing the assumption that no tracking error is in the inner loop of the impedance controller in the previous works. The proposed strategy is a promising solution for real applications and is validated by a board cleaning experiment in this paper.

Image-based visual servoing (IBVS) can reach a desired position for a relatively stationary target using continuous visual feedback. Proper feature extraction and appropriate servoing control laws are essential to performance for IBVS. IBVS control can be interrupted or interfered abruptly if no features are extracted when the observed object is occluded. To address the problem of missing feature points in current images during a visual navigation task, a homography method that uses a priori visual information is proposed to predict all of the missing feature points and to ensure the execution of IBVS. The mixture parameter for the image Jacobian matrix can also affect the control of IBVS. The settings for the mixture parameter are heuristic so there is no a systematic approach for most IBVS applications. An adaptive control approach is proposed to determine the mixture parameter. The proposed method uses a reinforcement learning (RL) method to adaptively adjust the mixture parameter during the robot movement, which allows more efficient control than a constant parameter. A logarithmic interval state-space partition for RL is used to ensure efficient learning. The integrated visual servoing control system is validated by several experiments that involve wheeled mobile robots reaching a target with a desired configuration. The results for simulation and experiment demonstrate that the proposed method has a faster convergence rate than other methods.

In this paper, a practical discrete-time control method with adaptive image feature prediction for the image-based visual servoing (IBVS) scheme is presented. In the discrete-time IBVS inner-loop/outer-loop control architecture, the time delay caused by image capture and computation is noticed. Considering the dynamic characteristics of a 6-DOF manipulator velocity input system, we propose a linear dynamic model to describe the motion of a robot end effector. Furthermore, for better estimation of image features and smoothing of the robot's velocity input, we propose an adaptive image feature prediction method that employs past image feature data and real robot velocity data to adopt the prediction parameters. The experimental results on a 6-DOF robotic arm demonstrate that the proposed method can ensure system stability and accelerate system convergence.