Visual Servoing Method Using Camera Self-calibration

The objective of visual servoing aims to control an object's motion with visual feedbacks and becomes popular recently. Problems of complex modeling and instability always exist in visual servoing methods. Moreover, there are few research works on selection of the servoing gain in image-based visual servoing (IBVS) methods. This paper proposes an IBVS method with Q -Learning, where the learning rate is adjusted by a fuzzy system. Meanwhile, a synthetic preprocess is introduced to perform feature extraction. The extraction method is actually a combination of a color-based recognition algorithm and an improved contour-based recognition algorithm. For dealing with underactuated dynamics of the unmanned aerial vehicles (UAVs), a decoupled controller is designed, where the velocity and attitude are decoupled through attenuating the effects of underactuation in roll and pitch and two independent servoing gains, for linear and angular motion servoing, respectively, are designed in place of single servoing gain in traditional methods. For further improvement in convergence and stability, a reinforcement learning method, Q -Learning, is taken for adaptive servoing gain adjustment. The Q -Learning is composed of two independent learning agents for adjusting two serving gains, respectively. In order to improve the performance of the Q -Learning, a fuzzy-based method is proposed for tuning the learning rate. The results of simulations and experiments on control of UAVs demonstrate that the proposed method has better properties in stability and convergence than the competing methods.

This paper presents a review of characteristics of the two main visual servoing methods: image-based visual servoing and position-based visual servoing. First, a generic structure is defined with an emphasis on the same system inputs and outputs for the two methods. The stability, robustness, sensitivity, and other general characteristics of the methods are presented. To overcome the dependency of both methods on a known target model, a midpoint triangulation method is proposed to estimate the unknown target model for robot visual servoing.

The visual servoing technique has been studied extensively as a robust method for navigating robots toward goal positions and orientations reliably. Unfortunately, the existing visual servoing methods require the calibration of cameras and robots, which is time-consuming. Thus, in this paper, we propose a visual servoing method, which does not require the calibration of cameras and robots. In particular, we show that we can navigate uncalibrated robots to goal positions properly by using the projective reconstruction based on the abstract projection of control values.

Over the past few decades, vision based alignment has been accepted as an important technique to achieve higher economic benefits for precision manufacturing and measurement applications. Also referred to as visual servoing, this technique basically applies the vision feedback information and drives the moving parts to the desired target location using some appropriate control laws. Although recently rapid development of advanced image processing algorithms and hardware have made this alignment process an easier task, some fundamental issues including inevitable system constraints and singularities, still remain as a challenging research topic for further investigation. This paper aims to develop a visual servoing method for automatic alignment system using model predictive control (MPC). The reason for using this optimal control for visual servoing design is because of its capability of handling constraints such as motor and image constraints in precision alignment systems. In particular, a microassembly system for peg and hole alignment application is adopted to illustrate the design process. The goal is to perform visual tracking of two image feature points based on a XYθ motor-stage system. From the viewpoint of MPC, this is an optimization problem that minimizes feature errors under given constraints. Therefore, a dynamic model consisting of camera parameters and motion stage dynamics is first derived to build the prediction model and set up the cost function. At each sample step the control command is obtained by solving a quadratic programming optimization problem. Finally, simulation results with comparison to a conventional image based visual servoing method demonstrate the effectiveness and potential use of this method.

A review on visual servoing for underwater vehicle manipulation systems automatic control and case study

This paper presents a novel control strategy to force a vertical take‐off and landing (VTOL) aircraft to accomplish the pinpoint landing task. The control development is based on the image‐based visual servoing method and the back‐stepping technique; its design differs from the existing methods because the controller maps the image errors onto the actuator space via a visual model which does not contain the depth information of the feature point. The novelty of the proposed method is to extend the image‐based visual servoing technique to the VTOL aircraft control. In addition, the Lyapunov theory is used to prove the asymptotic stability of the VTOL aircraft visual servoing system, while the image error can converge to zero. Furthermore, simulations have been also conducted to demonstrate the performances of the proposed method.

This study proposes a hybrid visual servoing technique that is optimised to tackle the shortcomings of classical 2D, 3D and hybrid visual servoing approaches. These shortcomings are mostly the convergence issues, image and robot singularities, and unreachable trajectories for the robot. To address these deficiencies, 3D estimation of the visual features was used to control the translations in Z-axis as well as all rotations. To speed up the visual servoing (VS) operation, adaptive gains were used. Damped Least Square (DLS) approach was used to reduce the robot singularities and smooth out the discontinuities. Finally, manipulability was established as a secondary task, and the redundancy of the robot was resolved using the classical projection operator. The proposed approach is compared with the classical 2D, 3D and hybrid visual servoing methods in both simulation and real-world. The approach offers more efficient trajectories for the robot, with shorter camera paths than 2D image-based and classical hybrid VS methods. In comparison with the traditional position-based approach, the proposed method is less likely to lose the object from the camera scene, and it is more robust to the camera calibrations. Moreover, the proposed approach offers greater robot controllability (higher manipulability) than other approaches.

This paper has two purposes: investigating a featureless visual servoing approach based on mutation analysis and proposing a visual servo control method for nanomanipulations. For the first purpose, the featureless visual servoing method is needed because traditional visual servoing relies heavily on robust feature extraction and tracking, which are very difficult in natural environment. The mutation analysis based approach in this paper considers the image as a set, and designs a controller to make the distance between the initial and goal image sets converge to zero, thereby steering the initial image to the goal image. For the second purpose, atomic force microscopic (AFM) based nanomanipulations with subnanometer accuracy are very difficult because the position sensor cannot provide valuable feedback due to large noises at this precision level. We propose to use the images obtained by AFM and perform a visual servo control. This method, independent of external sensors, can directly perform control on the AFM end tip's position. The featureless controller is successfully validated on AFM images and the results suggest a potential precision enhancement for nanomanipulations.

The conventional image-based visual servoing leads to image singularities that might cause control instabilities. To avoid this problem, in this paper, the image moments are used as features for visual servoing, where the Jacobian matrix is full rank and upper triangular. Thus, it has the maximal decoupled structure and simplifies the controller. The general analytical form of the interaction matrix or the Jacobian matrix considering the camera parameters related to any image moments is derived in this paper. As a servoing controller, an optimal visual PD controller is presented to improve the performance of the visual servoing system instead of the P controller, which is the method extensively used in visual servoing. A genetic algorithm-based PD parameters tuning method is applied to obtain the optimal parameters. The method proposed is used to align the micropeg and hole, and the simulation results show that the object can reach its desired position faster and more smoothly.

This paper showcases the integration of the Interfacing Toolbox for Robotic Arms (ITRA) with our newly developed hybrid Visual Servoing (VS) methods to automate the disassembly of electric vehicle batteries, thereby advancing sustainability and fostering a circular economy. ITRA enhances collaboration between industrial robotic arms, server computers, sensors, and actuators, meeting the intricate demands of robotic disassembly, including the essential real-time tracking of components and robotic arms. We demonstrate the effectiveness of our hybrid VS approach, combined with ITRA, in the context of Electric Vehicle (EV) battery disassembly across two robotic testbeds. The first employs a KUKA KR10 robot for precision tasks, while the second utilizes a KUKA KR500 for operations needing higher payload capacity. Conducted in T1 (Manual Reduced Velocity) mode, our experiments underscore a swift communication protocol that links low-level and high-level control systems, thus enabling rapid object detection and tracking. This allows for the efficient completion of disassembly tasks, such as removing the EV battery’s top case in 27 s and disassembling a stack of modules in 32 s. The demonstrated success of our framework highlights its extensive applicability in robotic manufacturing sectors that demand precision and adaptability, including medical robotics, extreme environments, aerospace, and construction.

When facing large-size targets, for the problem that the target imaging field of monocular camera is small and the target imaging is not clear at a long distance, this paper proposes an uncalibrated visual servoing method using multiple cameras for large targets. On the basis of image-based visual servo control, an uncalibrated visual servo model of an arrayed multiple cameras is constructed, and the nonlinear mapping relationship between image features of multi-view fusion and robot end six-degree-of-freedom motion velocity is constructed by using image moments. Visual servoing operation for large-size targets by predicting robot end-motion velocity using a neural network based on genetic particle swarm optimization without calibrating the camera parameters and hand-eye relationship. The experimental results show that the proposed method can effectively realize multi-view fusion visual servoing operation for large-size targets with high operation accuracy and robustness.

This study proposes an optimized hybrid visual servoing approach to overcome the imperfections of classical two-dimensional, three-dimensional and hybrid visual servoing methods. These imperfections are mostly convergence issues, non-optimized trajectories, expensive calculations and singularities. The proposed method provides more efficient optimized trajectories with shorter camera path for the robot than image-based and classical hybrid visual servoing methods. Moreover, it is less likely to lose the object from the camera field of view, and it is more robust to camera calibration than the classical position-based and hybrid visual servoing methods. The drawbacks in two-dimensional visual servoing are mostly related to the camera retreat and rotational motions. To tackle these drawbacks, rotations and translations in Z-axis have been separately controlled from three-dimensional estimation of the visual features. The pseudo-inverse of the proposed interaction matrix is approximated by a neuro-fuzzy neural network called local linear model tree. Using local linear model tree, the controller avoids the singularities and ill-conditioning of the proposed interaction matrix and makes it robust to image noises and camera parameters. The proposed method has been compared with classical image-based, position-based and hybrid visual servoing methods, both in simulation and in the real world using a 7-degree-of-freedom arm robot.

This paper presents a method to combine voice control and hybrid visual servoing for a continuum robot. In this design, the continuum robot autonomously tracks the element of interest and maintains its position in the safe region based on image information. A method of hybrid visual servoing is proposed to integrate 2D image-based visual servoing (IBVS) to ensure the elements of interests under visibility constraints and 3D position-based visual servoing (PBVS) to adjust the robot to a desired pose in the constraint region. The robotic camera thus provides stable image while tracking the element of interest. In this design, the user can move the continuum robot to acquire the desired view by voice commands. In voice control, the system can recognize connecting words to control the robot. All software modules are integrated using Robot Operating System (ROS) for controlling the continuum robot. Experimental results obtained from a prototype continuum robot verify the effectiveness of the proposed method.

We propose a novel geometry-constrained learning-based method for camera-in-hand visual servoing systems that eliminates the need for camera intrinsic parameters, depth information, and the robot's kinematic model. Our method uses a cerebellar model articulation controller (CMAC) to execute online Jacobian estimation within the control framework. Specifically, we introduce a fixed-dimension, uniform-magnitude error function based on the projective homography matrix. The fixed-dimension error function ensures a constant Jacobian size regardless of the number of feature points, thereby reducing computational complexity. By not relying on individual feature points, the approach maintains robustness even when some features are occluded. The uniform magnitude of the error vector elements simplifies neural network input normalization, thereby enhancing online training efficiency. Furthermore, we incorporate geometric constraints between feature points (such as collinearity preservation) into the network update process, ensuring that model predictions conform to the fundamental principles of projective geometry and eliminating physically impossible control outputs. Experimental and simulation results demonstrate that our approach achieves superior robustness and faster learning rates compared to other model-free image-based visual servoing methods.

In this paper, a novel visual servoing technique for a 5-dof mobile manipulator with an eye-to-hand camera configuration is introduced. The proposed technique can be categorized as an image based (or 2D) visual servoing using a fixed camera in conjunction with a conic mirror (aka, a Catadioptric camera system) providing panoramic vision. Two fictitious landmarks mounted on robot's end-effector along with their mirror reflections, when viewed by the camera, provide enough information for 3D reasoning based on the four points viewed on the image plane. Instead of directly using the image features associated with these four points, five new image features are chosen to make the image Jacobian rank efficient. A dual estimation/control strategy based on Extended Kalman Filter (EKF) is utilized to (1) estimate camera's intrinsic and extrinsic parameters, and (2) track the coordinates of the landmarks and their reflections on the image plane. The relationship between the translational and rotational velocity of a frame attached to the robot's end-effector and the rate of change of the proposed image features are fully formulated. The robustness of the proposed technique in translational and rotational servoing of a 5-dof holonomic mobile manipulator is illustrated through computer simulations.