Real-Time Visual Servo Control of a Cartesian Robot

This paper aims to develop a visual servo control of a robotic manipulator for cherry tomato harvesting. In the robotic manipulator, an RGB-depth camera was mounted to the end effector to acquire the poses of the target cherry tomatoes in space. The eye-in-hand-based visual servo controller guides the end effector to implement eye–hand coordination to harvest the target cherry tomatoes, in which a hybrid visual servo control method (HVSC) with the fuzzy dynamic control parameters was proposed by combining position-based visual servo (PBVS) control and image-based visual servo (IBVS) control for the tradeoff of both performances. In addition, a novel cutting and clipping integrated mechanism was designed to pick the target cherry tomatoes. The proposed tomato-harvesting robotic manipulator with HVSC was validated and evaluated in a laboratory testbed based on harvesting implementation. The results show that the developed robotic manipulator using HVSC has an average harvesting time of 9.40 s/per and an average harvesting success rate of 96.25% in picking cherry tomatoes.

At the beginning of its application, arm-robot manipulator worked blindly in completing the assigned task, and along with the improvement of camera technology as it is getting cheaper and smaller, the application of visual sensor lead to visual servoing control which utilized target detection as the input to control/move the end-effector of an arm-robot. The application of camera in arm-robot manipulator is divided into eye-in-hand where a camera is mounted to the robot, and eye-to-hand where the camera is fixed elsewhere. Visual control involves kinematics modeling and target position in image plane provided by image processing to ensure the motion accuracy in grasping and harvesting the fruit. The end-effector movement for harvesting relies heavily on the accuracy of target position detection that can be obtained by image processing method. This paper is a review study that discusses the design of visual servoing control of an arm-robot manipulator applied in agriculture starting from kinematics and inverse kinematics for controlling the motion and trajectory of the robot, and image processing for target detecting. An example of an image processing method is given by using Edge Detection method for detecting a dragon fruit simulated in SCILAB.

SummaryThe goal of target tracking is to compute motion strategies for a robot equipped with visual sensors, so that it can effectively track a moving target despite obstruction by obstacles. It is an important problem with many applications in robotics. Existing work focuses mostly on the 2-D version of the problem, partly due to the complexity of dealing with 3-D visibility relationships. This paper proposes an online algorithm for 3-D target tracking among obstacles, using only local geometric information available to a robot’s visual sensors. Key to this new algorithm is the definition and efficient computation of a risk function, which tries to capture a target’s ability in escaping from the robot sensors’ visibility region in both short and long terms. The robot then moves to minimize this risk function locally in a greedy fashion. In the absence of occlusion by obstacles, the standard tracking algorithm based on visual servo control can be considered a special case of our algorithm. Experiments show that the new algorithm generated interesting tracking behaviors in three dimensions and performed substantially better than visual servo control in simulation.

The use of unmanned aerial vehicles (UAVs) for military, scientific, and civilian sectors are increasing drastically in recent years. This study presents algorithms for the visual-servo control of an UAV, in which a quadrotor helicopter has been stabilized with visual information through the control loop. Unlike previous study that use pose estimation approach which is time consuming and subject to various errors, the visual-servo control is more reliable and fast. The method requires a camera on-board the vehicle, which is already available on various UAV systems. The UAV with a camera behaves like an eye-in-hand visual servoing system. In this study the controller was designed by using two different approaches; image based visual servo control method and hybrid visual servo control method. Various simulations are developed on Matlab, in which the quadrotor aerial vehicle has been visual-servo controlled. In order to show the effectiveness of the algorithms, experiments were performed on a model quadrotor UAV, which suggest successful performance.

With the growing interest of integrating robotics into everyday life and industry, the requirements towards the quality and quantity of applications grows equally hard. This trend is profoundly recognized in applications involving visual perception. Whereas visual sensing in home environments tend to be mainly used for recognition and localization, safety becomes the driving factor for developing intelligent visual control algorithms. More specifically, a robot operating in a human environment should not collide with obstacles and executed motion should be as smooth as possible. Furthermore, as the environment is not known on beforehand, a high demand on the robustness of visual processing is a necessity. On the other hand, in an industrial setting, the environment is known on beforehand and safety is mainly guaranteed by excluding a human operator. Despite these reasons, and the fact that visual servoing has gained much attention from industry to become a standard solution for robotic automation tasks, applications are highly simplified. For example, methods such as visual fault detection are already a mature technique in industrial manufacturing, where a fixed camera observes a product (e.g., on a conveyor belt) and checks whether it meets certain requirements. These operations can be executed at a fairly high rate due to the simplicity of the system (e.g., static camera) and the simplification of the processing task (e.g., binary images). For both areas the identified difficulties are similar. Foremost, this is the slow nature of (robust) visual processing, in respect to the ever growing demand of increasing speed and reducing delay. These two application areas with analogous limitations motivate the design of more direct approaches of vision in visual control systems. Therefore, in order to meet the requirements for next generation visual control systems, this thesis presents approaches which employ visual measurements as a direct feedback to design constrained motion. First, for industrial robotics, in order to obtain the required positioning accuracy, the measurement and fixation system have to be highly rigid and welldesigned, implying high cost and long design time. By measuring the position of objects directly with a camera, instead of indirectly by motor encoders, the requirements of the measurement and fixation system are less demanding. Moreover, this motivates the miniaturization of the complete control system. The approach is validated in experiments on a simplified 2D planar stage (i.e., considerable friction, poor fixation), which attains similar performance compared to encoder-based positioning systems. Secondly, in a human-centred environment, this direct sensing can improve traditional visual control systems, when subject to certain disturbances. More specifically, a method is proposed that uses an image-based feedforward controller on top of traditional position-based visual servo control to overcome disturbances such as friction or poorly designed local motor controllers. This visual feedforward control action is only active when an image-based error is present and vanishes when that error goes to zero. The method is validated on an anthropomorphic robotic manipulator with 7 degrees of freedom, intended for operation in the human care environment. Third, sensing the product directly gives rise to designing motion directly. Whereas in traditional approaches the motion trajectory is designed offline and can not be changed at runtime, direct trajectory generation computes the motion of the next step based on current state and events. This means that at any instance in time, the trajectory of a motion system can be altered with respect to certain desired kinematic or dynamic constraints. For industrial applications this makes manufacturing on near-repetitive or non-rigid structures (e.g. flexible displays) possible. When applied to a robotic manipulator, this enables obstacle avoidance to no longer be on path planning level, but on trajectory planning level, where kinematic or dynamic constraints can be taken into account. This results in a motion that is smoother than when obstacle avoidance with path planning is employed. For both application areas this direct trajectory generation method is implemented and shows high flexibility in constrained motion trajectory design.

Optical biopsy using probe‐based confocal laser endomicroscopy (pCLE) and optical coherence tomography (OCT) technologies performs in vivo tissue scanning of organ surfaces and plays a crucial role in early cancer diagnosis. Herein, a flexible endomicroscopic scanning instrument based on a 3‐degree‐of‐freedom flexible hinge structure and visual servoing control is presented. The instrument integrates OCT and pCLE. OCT information compensates for the imaging distance of pCLE, which can combine the advantages of the two technologies. The mechanism allows for tissue surface scanning with a wire‐driven active bending part and dynamic adjustment of the probe‐tissue distance using a combination of wire and spring‐driven, which satisfy large‐area scanning while maintaining consistence tissue contacts. Local optimization‐based visual servoing control is proposed to optimize the tissue scanning. Through the scanning experiments, high‐quality mosaics of uneven surfaces are obtained with an effective area that is larger than 4 mm2. In vivo animal trial further verifies the feasibility and clinical potential of the instrument.

In this paper we study image based visual servo (IBVS) control of a four rotor vertical take-off and landing (VTOL) craft known as an X4-flyer. The approach taken is tailored to planar visual targets and is based on a reformulation and extension of 2 1/2 D image-based visual servo design to the control of dynamic under-actuated systems. Exponential stability of the closed loop system is proved and simulation results are presented.

Designing a real-time visual tracking system to catch goldfishes is a very complex task because a fish’s precise relative 3D positions inside a water tank are difficult to determine. In this paper, we present a new visual servo control system using stereo vision to acquire the 3D positions of goldfishes. The continuously adaptive mean shift (CAMSHIFT) algorithm and optical flow vision tracking methods are applied to provide feedback of a tracked fish’s real-time position and motion trajectory in a water tank at a high frame-rate. Two low-cost, well calibrated cameras are used to track the goldfishes and a 5-axis robot manipulator is used to catch a specified goldfish. The designed visual tracking and servo system is less sensitive to lighting influences and thus performs more efficiently. Experiments of the proposed method yielded very good results as the real-time 3D vision successfully tracked two fishes and guided the manipulator to catching one of them in a net attached to its end-effector.

In this brief, the robust position-based visual servoing control problem is addressed for a quadrotor with a down-facing monocular camera to perform ground target tracking tasks in outdoor environments. The object is to regulate the position of the quadrotor relative to the ground target by using a minimum sensor set, including an inertial measurement unit, an ultrasonic sensor, and a vision sensor. The robust compensators are designed in the visual servoing control system to enhance the robustness against nonlinearities, couplings, and uncertainties, including parameter uncertainties and external disturbances in the translational and rotational quadrotor dynamics. Theoretical analysis and experimental results demonstrate the effectiveness of the proposed robust visual servoing control method.

This paper investigates the object following control problem for a low-cost quadrotor unmanned aerial vehicle (UAV). An object tracking-based approach is proposed. In this approach, the techniques of object tracking and visual servo control are efficiently combined to achieve the control objectives. To eliminate the impact of tracking target rotation during tracking model, an online learning method is proposed by learning the object size over different view at the desired distance. On the basis of this, an image-based visual servo control law is then designed to ensure that the quadrotor can track the moving target in real-time and keep the desired distance away from the object. The applicability of the proposed scheme is verified by flight experiments. It is demonstrated that the proposed algorithm is robust to rotary movement.

This paper is oriented to the transmission line inspection environment, aiming to solve the problem of how to control the robotic arm to quickly and accurately grasp the target object, and work on the planning of the motion path and the position-based visual servo control method. For the spatial trajectory planning of the robotic arm, linear trajectory planning and circular arc trajectory planning are carried out respectively. Between the reaction speed problem of the traditional RRT motion algorithm in collision-free motion planning, a two-way RRT-Connect path planning method is proposed. Then the visual servo control algorithm of the UAV robotic arm is proposed. The position-based visual servo control method is adopted, while the EK-SVSF position estimation algorithm and the OWA-based stepwise data fusion algorithm are combined for optimal position estimation of the target during transmission line inspection. It is verified in simulation experiments that the improved RRT algorithm can obtain the shortest path planning in a shorter time with better performance. Finally, the grasping experiment is carried out in a real robotic arm transmission line inspection scene. The robotic arm path planning algorithm and visual servo control proposed in this paper are combined to carry out grasping experiments in a static cluttered environment. The experimental results show that the algorithm proposed in this paper has the best grasping success rate, servo success rate, and grasping similarity. In addition, the average errors in x-axis, y-axis and z-axis of the proposed method are 0.19cm, 1.11cm and 0.18cm, respectively, which show a good grasping accuracy, indicating that the control method proposed in this paper can be applied to the real transmission line inspection work scenarios.

This paper proposes a new geometric control method for homography-based visual servo control of Underactuated UAVs. In order to solve the application difficulties of geometric control in HBVS and explore a visual servo control technology that can be applied to aerial detection operations, this paper integrates the geometric control into the visual servoing framework and design a new homography-based geometric visual servoing controller. The outer loop is used as feedback information using the virtual homography matrix between the two images. The inner loop controls the orientation of the UAVs through geometric control. The stability of the proposed controller is proved based on Lyapunov’s theory. The proposed method has better transient performance and dynamic performance than the conventional visual servo method. The excellent performance of the controller has been proven by a large number of experiments. In addition, the application of the controller on an unmanned aerial manipulator is demonstrated.

Visual servo control is one of the central issues in the research field of intelligent robot. In order to obtain more effective information from the environment, visual sensor is used in robot control. This paper sets up the suitable model of the robot, with mathematical formulas derivation of visual servo control. Fuzzy adaptive PID is introduced to IBVS, in order to get a better performance. The simulation result shows that control strategy is effective.

This paper describes a feature based visual servo control considering the object distortion on the image plane. In the visual servo control of the manipulator with the hand/eye system realizing the stable convergence to target feature points with visual distortion, it is necessary to control not only the tip position but also the direction of the hand/eye system. To control both motions at the same time, the redundant manipulator is useful. In the proposed method, the rotational motion is controlled by null space input to reduce the visual distortion. Furthermore, to realize the stable motion convergence, the null space input gain is adjusted by fuzzy logic. To confirm the validity of the proposed method, simulation and experimental results are shown.

There is significant motivation to provide robotic systems with improved autonomy as a means to significantly accelerate deactivation and decommissioning (D&D) operations while also reducing the associated costs, removing human operators from hazardous environments, and reducing the required burden and skill of human operators. To achieve improved autonomy, this project focused on the basic science challenges leading to the development of visual servo controllers. The challenge in developing these controllers is that a camera provides 2-dimensional image information about the 3-dimensional Euclidean-space through a perspective (range dependent) projection that can be corrupted by uncertainty in the camera calibration matrix and by disturbances such as nonlinear radial distortion. Disturbances in this relationship (i.e., corruption in the sensor information) propagate erroneous information to the feedback controller of the robot, leading to potentially unpredictable task execution. This research project focused on the development of a visual servo control methodology that targets compensating for disturbances in the camera model (i.e., camera calibration and the recovery of range information) as a means to achieve predictable response by the robotic system operating in unstructured environments. The fundamental idea is to use nonlinear Lyapunov-based techniques along with photogrammetry methods to overcome the complex control issues and alleviate many of the restrictive assumptions that impact current robotic applications. The outcome of this control methodology is a plug-and-play visual servoing control module that can be utilized in conjunction with current technology such as feature recognition and extraction to enable robotic systems with the capabilities of increased accuracy, autonomy, and robustness, with a larger field of view (and hence a larger workspace). The developed methodology has been reported in numerous peer-reviewed publications and the performance and enabling capabilities of the resulting visual servo control modules have been demonstrated on mobile robot and robot manipulator platforms.