Optimized hybrid decoupled visual servoing with supervised learning

This study proposes a hybrid visual servoing technique that is optimised to tackle the shortcomings of classical 2D, 3D and hybrid visual servoing approaches. These shortcomings are mostly the convergence issues, image and robot singularities, and unreachable trajectories for the robot. To address these deficiencies, 3D estimation of the visual features was used to control the translations in Z-axis as well as all rotations. To speed up the visual servoing (VS) operation, adaptive gains were used. Damped Least Square (DLS) approach was used to reduce the robot singularities and smooth out the discontinuities. Finally, manipulability was established as a secondary task, and the redundancy of the robot was resolved using the classical projection operator. The proposed approach is compared with the classical 2D, 3D and hybrid visual servoing methods in both simulation and real-world. The approach offers more efficient trajectories for the robot, with shorter camera paths than 2D image-based and classical hybrid VS methods. In comparison with the traditional position-based approach, the proposed method is less likely to lose the object from the camera scene, and it is more robust to the camera calibrations. Moreover, the proposed approach offers greater robot controllability (higher manipulability) than other approaches.

A review on visual servoing for underwater vehicle manipulation systems automatic control and case study

This paper presents a method to combine voice control and hybrid visual servoing for a continuum robot. In this design, the continuum robot autonomously tracks the element of interest and maintains its position in the safe region based on image information. A method of hybrid visual servoing is proposed to integrate 2D image-based visual servoing (IBVS) to ensure the elements of interests under visibility constraints and 3D position-based visual servoing (PBVS) to adjust the robot to a desired pose in the constraint region. The robotic camera thus provides stable image while tracking the element of interest. In this design, the user can move the continuum robot to acquire the desired view by voice commands. In voice control, the system can recognize connecting words to control the robot. All software modules are integrated using Robot Operating System (ROS) for controlling the continuum robot. Experimental results obtained from a prototype continuum robot verify the effectiveness of the proposed method.

This paper considers image-guided assembly for large composite panels. By using fiducial markers on the panels and robot gripper mounted cameras, we are able to use an industrial robot to align the panels to sub-millimeter accuracy. We considered two commonly used visual servoing schemes: position-based visual servoing (PBVS) and image-based visual servoing (IBVS). It has been noted that IBVS possesses superior robustness with respect to the camera calibration accuracy. However, we have found that in our case, PBVS is both faster and slightly more accurate than IBVS. This result is due to the fact that the visual servoing target in the image plane is derived from a reference target, which depends on the accuracy of the camera model. This additional dependency essentially nullifies the robustness advantage of IBVS. We also implemented a simple scheme to combine inputs from multiple cameras to improve the visual servoing accuracy. Both simulation and experimental results are included to show the effectiveness of visual servoing in an industrial setting.

U ovom preglednom radu su prikazane metode vizuelnog upravljanja robotskih sistema, sa primarnim fokusom na mobilne robote sa diferencijalnim pogonom. Analizirane su standardne metode vizuelnog upravljanja bazirane na (i) greškama u parametrima slike (engl. Image-Based Visual Servoing - IBVS) i (ii) izdvojenim karakteristikama sa slike neophodnim za estimaciju položaja izabranog objekta (engl. Position-Based Visual Servoing - PBVS) i poređene sa novom metodom direktnog vizuelnog upravljanja (engl. Direct Visual Servoing - DVS). U poređenju sa IBVS i PBVS metodama, DVS metod se odlikuje višom tačnošću, ali i manjim domenom konvergencije. Zbog ovog razloga je DVS metod upravljanja pogodan za integraciju u hibridne sisteme vizuelnog upravljanja. Takođe, predstavljeni su radovi koji unapređuju sistem vizuelnog upravljanja korišćenjem stereo sistema (sistem sa dve kamere). Stereo sistem, u poređenju sa alternativnim metodama, omogućava tačniju ocenu dubine karakterističnih objekata sa slike, koja je neophodna za zadatke vizuelnog upravljanja. Predmet analize su i radovi koji integrišu tehnike veštačke inteligencije u sistem vizuelnog upravljanja. Ovim tehnikama sistemi vizuelnog upravljanja dobijaju mogućnost da uče, čime se njihov domen primene znatno proširuje. Na kraju, napominje se i mogućnost integracije vizuelne odometrije u sisteme vizuelnog upravljanja, što prouzrokuje povećanje robusnosti čitavog robotskog sistema.

This paper presents a review of characteristics of the two main visual servoing methods: image-based visual servoing and position-based visual servoing. First, a generic structure is defined with an emphasis on the same system inputs and outputs for the two methods. The stability, robustness, sensitivity, and other general characteristics of the methods are presented. To overcome the dependency of both methods on a known target model, a midpoint triangulation method is proposed to estimate the unknown target model for robot visual servoing.

This thesis deals with Visual Servoing and its strictly connected disciplines like projective geometry, image processing, robotics and non-linear control. More specifically the work addresses the problem to control a robotic manipulator through one of the largely used Visual Servoing techniques: the Image Based Visual Servoing (IBVS). In Image Based Visual Servoing the robot is driven by on-line performing a feedback control loop that is closed directly in the 2D space of the camera sensor. The work considers the case of a monocular system with the only camera mounted on the robot end effector (eye in hand configuration). Through IBVS the system can be positioned with respect to a 3D fixed target by minimizing the differences between its initial view and its goal view, corresponding respectively to the initial and the goal system configurations: the robot Cartesian Motion is thus generated only by means of visual informations. However, the execution of a positioning control task by IBVS is not straightforward because singularity problems may occur and local minima may be reached where the reached image is very close to the target one but the 3D positioning task is far from being fulfilled: this happens in particular for large camera displacements, when the the initial and the goal target views are noticeably different. To overcame singularity and local minima drawbacks, maintaining the good properties of IBVS robustness with respect to modeling and camera calibration errors, an opportune image path planning can be exploited. This work deals with the problem of generating opportune image plane trajectories for tracked points of the servoing control scheme (a trajectory is made of a path plus a time law). The generated image plane paths must be feasible i.e. they must be compliant with rigid body motion of the camera with respect to the object so as to avoid image jacobian singularities and local minima problems. In addition, the image planned trajectories must generate camera velocity screws which are smooth and within the allowed bounds of the robot. We will show that a scaled 3D motion planning algorithm can be devised in order to generate feasible image plane trajectories. Since the paths in the image are off-line generated it is also possible to tune the planning parameters so as to maintain the target inside the camera field of view even if, in some unfortunate cases, the feature target points would leave the camera images due to 3D robot motions. To test the validity of the proposed approach some both experiments and simulations results have been reported taking also into account the influence of noise in the path planning strategy. The experiments have been realized with a 6DOF anthropomorphic manipulator with a fire-wire camera installed on its end effector: the results demonstrate the good performances and the feasibility of the proposed approach.

This paper aims to develop a visual servo control of a robotic manipulator for cherry tomato harvesting. In the robotic manipulator, an RGB-depth camera was mounted to the end effector to acquire the poses of the target cherry tomatoes in space. The eye-in-hand-based visual servo controller guides the end effector to implement eye–hand coordination to harvest the target cherry tomatoes, in which a hybrid visual servo control method (HVSC) with the fuzzy dynamic control parameters was proposed by combining position-based visual servo (PBVS) control and image-based visual servo (IBVS) control for the tradeoff of both performances. In addition, a novel cutting and clipping integrated mechanism was designed to pick the target cherry tomatoes. The proposed tomato-harvesting robotic manipulator with HVSC was validated and evaluated in a laboratory testbed based on harvesting implementation. The results show that the developed robotic manipulator using HVSC has an average harvesting time of 9.40 s/per and an average harvesting success rate of 96.25% in picking cherry tomatoes.

The objective of visual servoing aims to control an object's motion with visual feedbacks and becomes popular recently. Problems of complex modeling and instability always exist in visual servoing methods. Moreover, there are few research works on selection of the servoing gain in image-based visual servoing (IBVS) methods. This paper proposes an IBVS method with Q -Learning, where the learning rate is adjusted by a fuzzy system. Meanwhile, a synthetic preprocess is introduced to perform feature extraction. The extraction method is actually a combination of a color-based recognition algorithm and an improved contour-based recognition algorithm. For dealing with underactuated dynamics of the unmanned aerial vehicles (UAVs), a decoupled controller is designed, where the velocity and attitude are decoupled through attenuating the effects of underactuation in roll and pitch and two independent servoing gains, for linear and angular motion servoing, respectively, are designed in place of single servoing gain in traditional methods. For further improvement in convergence and stability, a reinforcement learning method, Q -Learning, is taken for adaptive servoing gain adjustment. The Q -Learning is composed of two independent learning agents for adjusting two serving gains, respectively. In order to improve the performance of the Q -Learning, a fuzzy-based method is proposed for tuning the learning rate. The results of simulations and experiments on control of UAVs demonstrate that the proposed method has better properties in stability and convergence than the competing methods.

Since wheeled mobile robots play an important role in the field of domestic medical service, this paper studies the problem of guaranteed cost control for visual servoing systems of wheeled mobile robots with input constraints. First, a cascade model is derived for a class of mobile robot via image-based visual servoing method, in which the first stage subsystem considers the angular velocity as the input, and the second stage subsystem recognizes the angular velocity as the system parameter. Since the angular velocity of the system is a bounded time-varying parameter, the angular velocity is treated as an uncertain parameter in the second stage subsystem. In this case, the second stage subsystem is transferred into a linear time-invariant system with parameter uncertainties. Sufficient conditions are obtained by choosing a proper Lyapunov function, ensuring that the closed-loop system is asymptotically stable. Furthermore, a minimization problem with three linear matrix inequalities (LMIs) is proposed to obtain a state feedback controller. Finally, simulations and experiments are provided to show the effectiveness of the proposed method.

Visual servoing is a method to achieve precise positioning and motion control of objects by visual feedback, and it is widely applied in the fields of robotics and unmanned aerial vehicles (UAVs) in recent years. This paper presents a novel image-based visual servoing (IBVS) control method for UAVs based on fuzzy logic to effectively solve the problem under field of view constraint and improve the control efficiency. In this paper, a fuzzy logic of servo gain is designed for the control input of visual servoing, which solves the problem of feature loss in IBVS and improves the efficiency. Meanwhile, a deep computing method based on known data is proposed to solve the unknown depth of Jacobian matrix, which makes the control easier to converge. The effectiveness of the proposed method is verified by the simulation of a quadrotor UAV equipped with a monocular camera.

In this paper, a novel image-based visual servoing (IBVS) method is proposed for the case that the servoing target randomly and slowly tilts. With the basic IBVS function and virtual camera plane method, a four-dimensional image kinematics is derived. Then a transformation based homography reconstruction method is presented to solve the problem of tilting target plane by defining a new virtual camera plane. Subsequently the IBVS control laws are designed according to the quadrotor dynamics and transformed image kinematics. Finally, simulations show the validity and priority of the new method compared with the classic dynamic IBVS method.

Since quadrotors are becoming more popular, several techniques appeared to control robustly the Unmanned Aerial Vehicle. This article proposes an intelligent image-based visual servoing (IBVS) control method for an unmanned aerial vehicle (UAV). This work consists of computing the proportional controller gain by smoothing the error between the desired and actual feature vectors values, as opposed to the classical visual servoing (VS) scheme. The weak handling of this gain, affect the maintaining of tracked features in the area of view. Unlike the classical VS scheme that requires a known value of gain, selected in a heuristic way, which lead the controller to fall in the velocity convergence problem. We adjust an adaptive gain value by the use of a fuzzy logic controller. The proposed controller exploits the error between feature vectors and its derivative as inputs. Furthermore, we design an intelligent based-fuzzy logic method for gain tuning of PD controllers applied for handling the heading and position of the quadrotor. The proposed controllers are applied to the inner and outer loop, offer several advantages in dealing the uncertainty occurred within trajectory tracking control. Various simulations are developed in MATLAB, shows the outperformance of the proposed method against the classical one.

This paper presents a visual servoing method for a redundant manipulator to control quickly and accurately by image error in a complicated environment. The proposed approach combines two parts, one is position-based visual servoing, and another is avoiding obstacles by using virtual repulsive torque. The image features are found and mapped by Oriented FAST and Rotated BRIEF (ORB) and the image errors are calculated. Furthermore, the object pose is estimated in the world coordinate by the image errors and the current pose of the manipulator and the target EOAT pose compute will be obtained. After confirming the EOAT pose, this method introduces a virtual repulsive torque for solving generalized inverse kinematics problems by analytical and numerical methods. The proposed approach uses the property of the redundant manipulator to maintain the distance between links and obstacles. For the applicability, we transform the programs to the nodes and use Robot Operating System as the core to build the communication protocol for exchanging information. The validity and efficiency of the proposed method are measured in simulations, by which it verifies that the method is not only feasible to constrained motions but also has good performances in collision avoidance.

We propose a novel geometry-constrained learning-based method for camera-in-hand visual servoing systems that eliminates the need for camera intrinsic parameters, depth information, and the robot's kinematic model. Our method uses a cerebellar model articulation controller (CMAC) to execute online Jacobian estimation within the control framework. Specifically, we introduce a fixed-dimension, uniform-magnitude error function based on the projective homography matrix. The fixed-dimension error function ensures a constant Jacobian size regardless of the number of feature points, thereby reducing computational complexity. By not relying on individual feature points, the approach maintains robustness even when some features are occluded. The uniform magnitude of the error vector elements simplifies neural network input normalization, thereby enhancing online training efficiency. Furthermore, we incorporate geometric constraints between feature points (such as collinearity preservation) into the network update process, ensuring that model predictions conform to the fundamental principles of projective geometry and eliminating physically impossible control outputs. Experimental and simulation results demonstrate that our approach achieves superior robustness and faster learning rates compared to other model-free image-based visual servoing methods.