A study on real-time implementation of visual feedback control of robot manipulator

We present an approach to visual feedback control using image-based visual servoing with stereo vision. In order to control the position and orientation of a robot with respect to an object, a technique is proposed using binocular stereo vision. The stereo vision enables us to calculate an exact image Jacobian not only at around a desired location but also at the other locations. The suggested technique can guide a robot manipulator to the desired location without giving such priori knowledge as the relative distance to the desired location or the model of an object even if the initial positioning error is large. The paper describes a model of stereo vision and how to generate feedback commands. The performance of the proposed visual servoing system is illustrated by the simulation and experimental results and compared with the case of conventional method for an assembling robot.

Visual servoing of robot manipulators is a key technique where the appearance of an object in the image plane is used to control the velocity of the end-effector such that the desired position is reached in the scene. The vast majority of visual servoing methods proposed so far uses calibrated robots in conjunction with calibrated cameras. It has been shown that the behavior of visual control loops does not degrade too much in the presence of calibration errors. Nevertheless, camera and robot calibration are complex and time-consuming processes requiring special-purpose mechanical devices, such as theodolites and calibration jigs. In this paper, we suggest formulating a visual servoing control loop in a nonmetric space, which in our case amounts to the projective space in which a triangulation of the scene using an uncalibrated stereo rig is expressed. The major consequence of controlling the robot in nonmetric space rather than in Euclidean space is that both the robot’s direct kinematic map and the robot’s Jacobian matrix must be defined in this space as well. The elementary joint-space motions that can be performed by a robot manipulator are pure rotations and pure translations. Traditionally, these motions are represented as Euclidean transformations. Since these motions are observed with an uncalibrated stereo rig, it will be convenient to represent them as projective transformations (homographies) rather than Euclidean transformations. Indeed, it will be shown that rotations and translations can be parameterized as special cases of homographies, which will be called projective rotations and projective translations. The algebraic properties of this nonmetric representation of elementary motions will be thoroughly investigated allowing us to characterize the direct kinematic map and the Jacobian matrix of a manipulator. Therefore, we introduce the concepts of projective kinematics and a projective Jacobian matrix. Unlike the classical robot Jacobian matrix of a manipulator that relates the robot joint-velocities to the kinematic screw associated with the end-effector, we establish a direct relationship between joint-velocities and image-plane velocities. The latter are velocities associated with image points arising from the 3-D to 2-D projection of end-effector points. Finally, we provide a practical method to estimate the projec tive kinematic model and we describe some preliminary simulated experiments that use this nonmetric model to perform stereo-based servoing. Nevertheless, in-depth analysis of projective control will be the topic of a forthcoming paper.

With the growing interest of integrating robotics into everyday life and industry, the requirements towards the quality and quantity of applications grows equally hard. This trend is profoundly recognized in applications involving visual perception. Whereas visual sensing in home environments tend to be mainly used for recognition and localization, safety becomes the driving factor for developing intelligent visual control algorithms. More specifically, a robot operating in a human environment should not collide with obstacles and executed motion should be as smooth as possible. Furthermore, as the environment is not known on beforehand, a high demand on the robustness of visual processing is a necessity. On the other hand, in an industrial setting, the environment is known on beforehand and safety is mainly guaranteed by excluding a human operator. Despite these reasons, and the fact that visual servoing has gained much attention from industry to become a standard solution for robotic automation tasks, applications are highly simplified. For example, methods such as visual fault detection are already a mature technique in industrial manufacturing, where a fixed camera observes a product (e.g., on a conveyor belt) and checks whether it meets certain requirements. These operations can be executed at a fairly high rate due to the simplicity of the system (e.g., static camera) and the simplification of the processing task (e.g., binary images). For both areas the identified difficulties are similar. Foremost, this is the slow nature of (robust) visual processing, in respect to the ever growing demand of increasing speed and reducing delay. These two application areas with analogous limitations motivate the design of more direct approaches of vision in visual control systems. Therefore, in order to meet the requirements for next generation visual control systems, this thesis presents approaches which employ visual measurements as a direct feedback to design constrained motion. First, for industrial robotics, in order to obtain the required positioning accuracy, the measurement and fixation system have to be highly rigid and welldesigned, implying high cost and long design time. By measuring the position of objects directly with a camera, instead of indirectly by motor encoders, the requirements of the measurement and fixation system are less demanding. Moreover, this motivates the miniaturization of the complete control system. The approach is validated in experiments on a simplified 2D planar stage (i.e., considerable friction, poor fixation), which attains similar performance compared to encoder-based positioning systems. Secondly, in a human-centred environment, this direct sensing can improve traditional visual control systems, when subject to certain disturbances. More specifically, a method is proposed that uses an image-based feedforward controller on top of traditional position-based visual servo control to overcome disturbances such as friction or poorly designed local motor controllers. This visual feedforward control action is only active when an image-based error is present and vanishes when that error goes to zero. The method is validated on an anthropomorphic robotic manipulator with 7 degrees of freedom, intended for operation in the human care environment. Third, sensing the product directly gives rise to designing motion directly. Whereas in traditional approaches the motion trajectory is designed offline and can not be changed at runtime, direct trajectory generation computes the motion of the next step based on current state and events. This means that at any instance in time, the trajectory of a motion system can be altered with respect to certain desired kinematic or dynamic constraints. For industrial applications this makes manufacturing on near-repetitive or non-rigid structures (e.g. flexible displays) possible. When applied to a robotic manipulator, this enables obstacle avoidance to no longer be on path planning level, but on trajectory planning level, where kinematic or dynamic constraints can be taken into account. This results in a motion that is smoother than when obstacle avoidance with path planning is employed. For both application areas this direct trajectory generation method is implemented and shows high flexibility in constrained motion trajectory design.

Visual servoing has been around for decades, but time delay is still one of the most troublesome problems to achieve target tracking. To circumvent the problem, in this paper, the Kalman filter is employed to estimate the future position of the object. In order to introduce the Kalman filter, accurate time delays, which include the processing lag and the motion lag, need to be obtained. Thus, the delays of the visual control servoing systems are discussed and a generic timing model for the system is provided. Then, we present a current statistical model for a moving target. A fuzzy adaptive Kalman filter, which is evolved from the Kalman filter, is introduced based on the current statistical model. Finally, a two DOF visual controller based variable structure control law for micro-manipulation is presented. The results show that the proposed adaptive Kalman filter can improve the ability to track moving targets.

Robotics has been a paradigm for science in the last few decades. At first, scientists’ efforts were devoted to the solution of the problem of planning and control of themotion of robot manipulators. However, the motion control of robot manipulators in unstructured environments is today an attractive scientific problem. An interesting solution for motion control is the use of sensor information, such as computer vision, in the system’s feedback. Several works and tools have been developed in recent years in this field (Corke; 2005), (Chaumette and Hutchinson; 2006). The more typical approaches consider visual perception for servoing and for the so called look and move (Hutchinson et al.; 1996). Visual servoing (Kelly et al.; 2000) can be classified into two approaches: camera-in-hand or camera-to-hand (Flandin et al.; 2000). In camera-to-hand robotic systems, multiple cameras or a single camera fixed in the worldcoordinate frame capture images of both, the robot and its environment. The tracking of the object with visual feedback can be made in 2D or 3D. An interesting solution for the visual servoing of camera-in hand robot manipulators in 2D can be found in e. g. (Bonfe et al.; 2002) and (Hernandez et al.; 2008) where stability demonstration of a decoupled controller have been presented. For 3D tracking some solutions reported are look and move controller, with one camera (Sim et al.; 2002) or more than one camera (Xie et al.; 2005). A 3D visual servoing with stability analysis in continuous time is presented by (Hernandez et al.; 2008a) and (Kelly et al.; 2006) present a direct visual servoing with transpose Jacobian control technique for regulation of robot manipulators in the 3D Cartesian space. In a similar vein (Enescu et al.; 2006) present mobile robot navigation for person tracking using a stereo head-camera. In this chapter, we consider the control problem of camera-in-hand robot manipulators in 2D and 3D. In both cases only one camera is mounted on the robot’s arm, which supplies visual information of the environment, with the aim of moving the manipulator by maintaining the image of the tracked object (a sphere) in the centre of the image plane, despite the possible movements of the object. In the 3D control the constant radius is used as a feature too. In this work, the proposed control system considers two loops in cascade, an internal loop solving the robots’ joint control, and an external loop implementing a dynamic look and move visual controller. A stability analysis in discrete time is developed under the conditions that it is possible to approximate the dynamic effect of the internal loop as an external loop time delay (Corke; 1996) and (Bonfe et al.; 2002). The more classic presentations of servovisual control are velocity controllers, based in the term feature Jacobian,(Chaumette and Hutchinson; 2006) 2

The conventional image-based visual servoing leads to image singularities that might cause control instabilities. To avoid this problem, in this paper, the image moments are used as features for visual servoing, where the Jacobian matrix is full rank and upper triangular. Thus, it has the maximal decoupled structure and simplifies the controller. The general analytical form of the interaction matrix or the Jacobian matrix considering the camera parameters related to any image moments is derived in this paper. As a servoing controller, an optimal visual PD controller is presented to improve the performance of the visual servoing system instead of the P controller, which is the method extensively used in visual servoing. A genetic algorithm-based PD parameters tuning method is applied to obtain the optimal parameters. The method proposed is used to align the micropeg and hole, and the simulation results show that the object can reach its desired position faster and more smoothly.

This paper aims to develop a visual servo control of a robotic manipulator for cherry tomato harvesting. In the robotic manipulator, an RGB-depth camera was mounted to the end effector to acquire the poses of the target cherry tomatoes in space. The eye-in-hand-based visual servo controller guides the end effector to implement eye–hand coordination to harvest the target cherry tomatoes, in which a hybrid visual servo control method (HVSC) with the fuzzy dynamic control parameters was proposed by combining position-based visual servo (PBVS) control and image-based visual servo (IBVS) control for the tradeoff of both performances. In addition, a novel cutting and clipping integrated mechanism was designed to pick the target cherry tomatoes. The proposed tomato-harvesting robotic manipulator with HVSC was validated and evaluated in a laboratory testbed based on harvesting implementation. The results show that the developed robotic manipulator using HVSC has an average harvesting time of 9.40 s/per and an average harvesting success rate of 96.25% in picking cherry tomatoes.

This paper presents a new nonlinear adaptive vision-based observer to estimate position and linear velocity information for closed-loop position-based visual servo control of an aerial robot in GPS-denied environments. Specifically, the observer determines the position and linear velocity of the robot for closed-loop control by observing using a low-cost on-board camera at least two feature points fixed in the world frame. The nonlinear adaptive observer takes advantage of the geometry of perspective projection, and is designed to update position and velocity information in real-time. Thus, there are no constraints or assumptions on the depth and initial estimation errors. Furthermore, the proposed parameter estimator addresses the challenge in situations where GPS signals may be weak, unreliable, or nonexistent, such in valleys, canyons, and between tall buildings, or inside of a building and under dense canopy. For closed-loop tracking control using the estimated position and velocity information, a backstepping controller is employed for the underactuated aerial robot system. The Lyapunov method is used to show stability of the closed-loop system. Simulation and experimental results are presented that validate the performance of the observer and control system for hovering and tracking a circular trajectory, where both are defined in the world (lab) frame.

Research has focused on developing control strategies that allow to work with imprecisely calibrated or even uncalibrated robot systems. A popular approach is to close the control loop with visual feedback (visual servoing), even though this requires continuous, high rate image processing. In contrast, we developed a strategy that can compensate calibration errors by integrating sparse visual feedback asynchronously. This strategy was motivated by neuroscientific studies which show that human grasping definitely requires only sparse asynchronous visual feedback, even with disturbed visual input. We analyze the performance of our control strategy on a robotic hand-eye system consisting of a 6 d.o.f. manipulator and a pan-tilt head with a stereo camera system. To analyze thoroughly which calibration errors can be compensated by sparse feedback, we furthermore built a complete simulation model of the real hand-eye system, including the vision modules. Taken together, experimental and simulation results show that calibration errors can be compensated with a very moderate amount of (asynchronous) visual feedback. Thus, the proposed control strategy can indeed be seen as a serious alternative to visual servoing for real robotics applications. This shows that robotics can profit from taking a closer look at the results of neuroscience.

This paper presents a dynamic simulation model for visual servoing of a robot manipulator using image based strategy. The modeling takes into account the dynamics of the camera velocity inner loop as well as a model of the visual sensor that consists of the camera and feature extraction block. Considering the inner loop and the visual sensor models, the dynamic simulation model of the visual servo open loop was developed. Using this open loop model, an image based controller was derived for the visual servoing system. Finally, the developed dynamic simulation model has been tested, including open-loop validation, closed-loop validation and comparisons between simulation results and real images obtained with a 6 DOF ABB robot having an eye-in-hand camera configuration.

This study presents the design and validation of a novel sampled-time state feedback controller for a multi-robotic system composed of a Cartesian robot and a Delta robot, providing five degrees of freedom – two from the Cartesian and three from the Delta robot. The novelty of this work lies in the integration of a machine learning-based visual servoing strategy with the discrete-time control design, where the controller's sampling period is carefully selected based on the computational demands of the vision-based feedback. The end-effector's pose is estimated in real time using a modified eye-to-hand YOLOv8 algorithm embedded within a neural network visual servoing system. This approach enables the robotic system to continuously update its control actions based on visual feedback, enhancing trajectory tracking accuracy during task execution. A key contribution of this work is the investigation of the impact of sampling time on the convergence quality of the end-effector's tracking error. The proposed control architecture was rigorously evaluated in a virtual environment, where the end-effector pose estimated by the visual system was compared against ground truth data. The experimental results demonstrate that adjusting the sampling time significantly improves control accuracy, thereby justifying the proposed discrete-time controller design. This integrated machine learning and sampled-time control approach shows promising potential for real-time, vision-guided robotic manipulation. Highlights A synthetic database for the proposed controller using the Segment Anything Model (SAM) for image processing. A CNN algorithm to detect the position and orientation of the end-effector of a composite robotic manipulator based on a sequential configuration of an open-closed kinematic chain. A sampled controller that drives the position/orientation of the composite robotic manipulator to time-varying specific target configurations, including its convergence analysis.

Robotic assembly of smartphone back shells with eye-in-hand visual servoing

The separate design of vision algorithms and motion control algorithms in existing vision servo systems limits their application in high-speed and high-precision industrial scenarios. To address this problem, a vision-motion codesign method is proposed for integrating visual servo systems with motion controllers via real-time Ethernet connections. During the engineering design phase of our proposal, we developed a control kernel architecture to clarify the system design logic. Then, we proposed a runtime module for real-time trajectory generation to facilitate the unified deployment of different planning methods in the kernel. On this basis, we implemented a self-adaptive trajectory generation synthesis method capable of dynamically adjusting the motion resources for real-time interpolation. The experimental results demonstrate that in visual servo static tracking experiments with the SCARA robot, our proposed method has a faster response time and smaller impact than other mainstream trajectory generation methods in 1 ms real-time interpolation cycles.

U ovom preglednom radu su prikazane metode vizuelnog upravljanja robotskih sistema, sa primarnim fokusom na mobilne robote sa diferencijalnim pogonom. Analizirane su standardne metode vizuelnog upravljanja bazirane na (i) greškama u parametrima slike (engl. Image-Based Visual Servoing - IBVS) i (ii) izdvojenim karakteristikama sa slike neophodnim za estimaciju položaja izabranog objekta (engl. Position-Based Visual Servoing - PBVS) i poređene sa novom metodom direktnog vizuelnog upravljanja (engl. Direct Visual Servoing - DVS). U poređenju sa IBVS i PBVS metodama, DVS metod se odlikuje višom tačnošću, ali i manjim domenom konvergencije. Zbog ovog razloga je DVS metod upravljanja pogodan za integraciju u hibridne sisteme vizuelnog upravljanja. Takođe, predstavljeni su radovi koji unapređuju sistem vizuelnog upravljanja korišćenjem stereo sistema (sistem sa dve kamere). Stereo sistem, u poređenju sa alternativnim metodama, omogućava tačniju ocenu dubine karakterističnih objekata sa slike, koja je neophodna za zadatke vizuelnog upravljanja. Predmet analize su i radovi koji integrišu tehnike veštačke inteligencije u sistem vizuelnog upravljanja. Ovim tehnikama sistemi vizuelnog upravljanja dobijaju mogućnost da uče, čime se njihov domen primene znatno proširuje. Na kraju, napominje se i mogućnost integracije vizuelne odometrije u sisteme vizuelnog upravljanja, što prouzrokuje povećanje robusnosti čitavog robotskog sistema.

A technology based on Kalman filtering method combined with multi-channel gain training reinforcement learning for uncalibrated camera visual servo tasks is proposed in this paper. First, a dynamic system with state variables formed from the elements of the image Jacobian matrix is constructed to describe the mapping relationship between two-dimensional images and three-dimensional poses. Kalman filter is used to estimate the state variables of the constructed system online. Next, the Jacobian matrix estimation and depth determination strategy gradient (DDPG) methods are combined to jointly train multi-channel gains by setting a reasonable segmented reward and punishment mechanism. Through training, a more effective gain decision can be obtained. The robustness of Kalman filtering to interference to a certain extent reduces the precise dependence of reinforcement learning models, thereby achieving higher robustness in intelligent visual servo control. Finally, the effectiveness and advantages of the Kalman-DDPG method have been demonstrated through simulation comparison and six-degree-of-freedom (DOF) uncalibrated manipulator physical experiments.