Visual Servoing of Robot Manipulators Part I: Projective Kinematics

In this paper, we propose a dynamic visual servo controller for precise tracking of robot manipulator. For the controller, we used a dual-loop structure which includes a vision controller and dynamic controller with different sampling speed. The vision controller is accommodated in the outer control loop with the aid of eye-in-hand camera, implementing the image based visual servoing method. As an inner loop controller, an adaptive backstepping controller is derived for position control of joint angle together with PID controller. The performance of proposed visual servo controller has been verified through computer simulation with V-REP simulation package.

In this paper we consider the tracking control problem of planar robots manipulators via visual servoing in the presence of parametric uncertainties associated with the robot dynamics and the camera parameters. An image–based visual servo control is developed using a Takagi–Sugeno (T–S) fuzzy model. The design includes an image coordinate velocity observer. To the best knowledge of the authors such a proposal remains hitherto unpublished. The new scheme is compared experimentally with two different non–fuzzy algorithms showing the good performance and advantages of the complete system.

In this paper, the uncalibrated visual servoing problem of robot manipulators with motor dynamics will be addressed for the fixed-camera configuration. A new adaptive image-space visual servoing strategy is presented to handle uncertainties in the camera intrinsic and extrinsic parameters, robot kinematic and dynamic parameters, and motor dynamic parameters. To deal with the nonlinear dependence of image Jacobian matrix on the unknown parameters, the proposed scheme is developed based on the concept of depth-independent interaction matrix. In this way, the camera parameters and the robot kinematic parameters in the closed-loop dynamics can be linearly parameterized such that adaptive laws can be designed to estimate them on-line. Adaptive algorithms are also developed to provide estimation of unknown robot dynamic and motor dynamic parameters. Stability analysis will be performed to show asymptotic convergence of image errors using Lyapunov theory based on both rigid-link robot dynamics and full motor dynamics. Simulation results based on a two-link planar robot manipulators will be given to illustrate the performance of the proposed scheme.

Real-time collision avoidance is a critical challenge in visual servoing control for safe robot manipulation. This article presents a model-free framework that integrates reinforcement learning (RL) with real-time collision detection (RCD), enabling secure and smooth visual servoing in dynamic environments. The reward function in the RL framework is carefully designed to simultaneously minimize tracking errors, promote motion smoothness, and enforce collision avoidance. Beyond policy optimization through RL, a two-step real-time trajectory blending (RTB) scheme, consisting of a Butterworth filter (BF) and polynomial interpolation, is developed to further refine the generated trajectories and ensure smooth joint velocity profiles. Furthermore, the soft actor-critic (SAC) algorithm is employed for policy learning, and the performance is compared with two widely used methods, namely proximal policy optimization (PPO) and twin delayed deep deterministic policy gradient (TD3). Experimental results on a 6-degree-of-freedom (6-DoF) robot manipulator demonstrate the effectiveness of the proposed approach in achieving safe and smooth visual servoing in real time.

Predictive Control Architecture for Visual Servoing of Robot Manipulators

This paper describes a whole new framework to quickly develop dynamic visual servoing systems embedded in an FPGA. Parallel design of the algorithms increases the precision of this kind of controllers, whereas minimizes the response time. Embedding dynamic visual servoing algorithms converts these systems into real-time systems. Additionally, an optimal control framework to dynamically visual control robot arms is proposed. The architecture re-configurability is tested with the implementation of two dynamic visual servoing controllers. One of the implemented controllers is derived from the proposed control framework and the other is the well-known image transpose Jacobian controller. From these experiments, FPGA resources utilization and timeline of the pipeline show the optimization of the proposed framework.

The conventional image-based visual servoing leads to image singularities that might cause control instabilities. To avoid this problem, in this paper, the image moments are used as features for visual servoing, where the Jacobian matrix is full rank and upper triangular. Thus, it has the maximal decoupled structure and simplifies the controller. The general analytical form of the interaction matrix or the Jacobian matrix considering the camera parameters related to any image moments is derived in this paper. As a servoing controller, an optimal visual PD controller is presented to improve the performance of the visual servoing system instead of the P controller, which is the method extensively used in visual servoing. A genetic algorithm-based PD parameters tuning method is applied to obtain the optimal parameters. The method proposed is used to align the micropeg and hole, and the simulation results show that the object can reach its desired position faster and more smoothly.

A technology based on Kalman filtering method combined with multi-channel gain training reinforcement learning for uncalibrated camera visual servo tasks is proposed in this paper. First, a dynamic system with state variables formed from the elements of the image Jacobian matrix is constructed to describe the mapping relationship between two-dimensional images and three-dimensional poses. Kalman filter is used to estimate the state variables of the constructed system online. Next, the Jacobian matrix estimation and depth determination strategy gradient (DDPG) methods are combined to jointly train multi-channel gains by setting a reasonable segmented reward and punishment mechanism. Through training, a more effective gain decision can be obtained. The robustness of Kalman filtering to interference to a certain extent reduces the precise dependence of reinforcement learning models, thereby achieving higher robustness in intelligent visual servo control. Finally, the effectiveness and advantages of the Kalman-DDPG method have been demonstrated through simulation comparison and six-degree-of-freedom (DOF) uncalibrated manipulator physical experiments.

Presents an approach to real time implementation of visual feedback control for a robotic manipulator. There are mainly two ways to put the visual feedback into practice. One is called look-and-move and the other is visual servoing. The former is the method which transforms the position and orientation of an object obtained by a visual sensor into those in the world frame fixed to an environment and guides the arm of the manipulator to a desired location in the world frame. Visual servoing uses the Jacobian matrix which relates the displacement of an image feature to the displacement of a camera motion and performs a closed-loop control regarding the feature as a scale of the state. Therefore, we can construct a servo system based only on the image and can have a robust control against the calibration error because there is no need to calculate the corresponding location in the world frame. In order to control the position and orientation of a robot with respect to an object, a new technique is proposed using binocular stereo vision. The stereo vision enables us to calculate an exact image Jacobian not only around a desired location but also at other locations. The suggested technique can guide a robot manipulator to the desired location without giving such priori knowledge as the relative distance to the desired location or the model of an object even if the initial positioning error is large. The performance of the proposed visual servoing system is illustrated by the experimental results and compared with the case of conventional method for an assembling robot with four joints.

With the growing interest of integrating robotics into everyday life and industry, the requirements towards the quality and quantity of applications grows equally hard. This trend is profoundly recognized in applications involving visual perception. Whereas visual sensing in home environments tend to be mainly used for recognition and localization, safety becomes the driving factor for developing intelligent visual control algorithms. More specifically, a robot operating in a human environment should not collide with obstacles and executed motion should be as smooth as possible. Furthermore, as the environment is not known on beforehand, a high demand on the robustness of visual processing is a necessity. On the other hand, in an industrial setting, the environment is known on beforehand and safety is mainly guaranteed by excluding a human operator. Despite these reasons, and the fact that visual servoing has gained much attention from industry to become a standard solution for robotic automation tasks, applications are highly simplified. For example, methods such as visual fault detection are already a mature technique in industrial manufacturing, where a fixed camera observes a product (e.g., on a conveyor belt) and checks whether it meets certain requirements. These operations can be executed at a fairly high rate due to the simplicity of the system (e.g., static camera) and the simplification of the processing task (e.g., binary images). For both areas the identified difficulties are similar. Foremost, this is the slow nature of (robust) visual processing, in respect to the ever growing demand of increasing speed and reducing delay. These two application areas with analogous limitations motivate the design of more direct approaches of vision in visual control systems. Therefore, in order to meet the requirements for next generation visual control systems, this thesis presents approaches which employ visual measurements as a direct feedback to design constrained motion. First, for industrial robotics, in order to obtain the required positioning accuracy, the measurement and fixation system have to be highly rigid and welldesigned, implying high cost and long design time. By measuring the position of objects directly with a camera, instead of indirectly by motor encoders, the requirements of the measurement and fixation system are less demanding. Moreover, this motivates the miniaturization of the complete control system. The approach is validated in experiments on a simplified 2D planar stage (i.e., considerable friction, poor fixation), which attains similar performance compared to encoder-based positioning systems. Secondly, in a human-centred environment, this direct sensing can improve traditional visual control systems, when subject to certain disturbances. More specifically, a method is proposed that uses an image-based feedforward controller on top of traditional position-based visual servo control to overcome disturbances such as friction or poorly designed local motor controllers. This visual feedforward control action is only active when an image-based error is present and vanishes when that error goes to zero. The method is validated on an anthropomorphic robotic manipulator with 7 degrees of freedom, intended for operation in the human care environment. Third, sensing the product directly gives rise to designing motion directly. Whereas in traditional approaches the motion trajectory is designed offline and can not be changed at runtime, direct trajectory generation computes the motion of the next step based on current state and events. This means that at any instance in time, the trajectory of a motion system can be altered with respect to certain desired kinematic or dynamic constraints. For industrial applications this makes manufacturing on near-repetitive or non-rigid structures (e.g. flexible displays) possible. When applied to a robotic manipulator, this enables obstacle avoidance to no longer be on path planning level, but on trajectory planning level, where kinematic or dynamic constraints can be taken into account. This results in a motion that is smoother than when obstacle avoidance with path planning is employed. For both application areas this direct trajectory generation method is implemented and shows high flexibility in constrained motion trajectory design.

Adaptive fault-tolerant control for uncalibrated camera–robot system with multiple uncertainties

In this paper, a novel 3-UPU (P and U stand for prismatic and universal joints, respectively) parallel mechanism (PM) and its variant PM are proposed. Both of them have two rotational and one translational (2R1T) degrees of freedom (DOFs) without involving any parasitic motion. Mobility analysis shows that the three constraint forces provided by three limbs of the mechanism are located on the same plane and the mobile platform can translate perpendicular to this plane and rotate around any axis on it. Analysis of the mechanism’s motion characteristics demonstrates that the mobile platform outputs either pure rotation or pure translation. Moreover, the rotational axis can be fixed during the rotation process, which means no parasitic motion is involved. The causes of the motion characteristics are analyzed by the combination of an overall Jacobian matrix, a statistical method, and a geometric method. The PMs only need to translate or rotate once to move from the initial configuration to the final configuration, which allows for easy control of speeds. The relationship between mechanism parameters and singularity is analyzed. A speed control method for the PMs is proposed and a prototype is designed and made. Experiments are conducted to verify the determined motion characteristics, the speed control method, and the singularity analysis.

U ovom preglednom radu su prikazane metode vizuelnog upravljanja robotskih sistema, sa primarnim fokusom na mobilne robote sa diferencijalnim pogonom. Analizirane su standardne metode vizuelnog upravljanja bazirane na (i) greškama u parametrima slike (engl. Image-Based Visual Servoing - IBVS) i (ii) izdvojenim karakteristikama sa slike neophodnim za estimaciju položaja izabranog objekta (engl. Position-Based Visual Servoing - PBVS) i poređene sa novom metodom direktnog vizuelnog upravljanja (engl. Direct Visual Servoing - DVS). U poređenju sa IBVS i PBVS metodama, DVS metod se odlikuje višom tačnošću, ali i manjim domenom konvergencije. Zbog ovog razloga je DVS metod upravljanja pogodan za integraciju u hibridne sisteme vizuelnog upravljanja. Takođe, predstavljeni su radovi koji unapređuju sistem vizuelnog upravljanja korišćenjem stereo sistema (sistem sa dve kamere). Stereo sistem, u poređenju sa alternativnim metodama, omogućava tačniju ocenu dubine karakterističnih objekata sa slike, koja je neophodna za zadatke vizuelnog upravljanja. Predmet analize su i radovi koji integrišu tehnike veštačke inteligencije u sistem vizuelnog upravljanja. Ovim tehnikama sistemi vizuelnog upravljanja dobijaju mogućnost da uče, čime se njihov domen primene znatno proširuje. Na kraju, napominje se i mogućnost integracije vizuelne odometrije u sisteme vizuelnog upravljanja, što prouzrokuje povećanje robusnosti čitavog robotskog sistema.

In R. A. Van Gorder, “General rotating quantum vortex filaments in the low-temperature Svistunov model of the local induction approximation,” Phys. Fluids 26, 065105 (2014) I discussed properties of generalized vortex filaments exhibiting purely rotational motion under the low-temperature Svistunov model of the local induction approximation. Such solutions are stationary in terms of translational motion. In the Comment [N. Hietala, “Comment on ‘General rotating quantum vortex filaments in the low-temperature Svistunov model of the local induction approximation’ [Phys. Fluids 26, 065105 (2014)],” Phys. Fluids 26, 119101 (2014)], the author criticizes my paper for not including translational motion (although it was clearly stated that the filament motion was assumed rotational). As it turns out, if one is interested in studying the geometric structure of solutions (which was the point of my paper), one obtains the needed qualitative results on the structure of such solutions by studying the purely rotational case. Nevertheless, in this Response I shall discuss the vortex filaments that have both rotational and translational motions. I then briefly discuss why one might want to study such generalized rotating filament solutions, in contrast to simple the standard helical or planar examples (which are really special cases). I also discuss how one can study the time evolution of filaments which exhibit more complicated dynamics than pure translation and rotation. Doing this, one can study non-stationary solutions which initially appear purely rotational and gradually display other dynamics as the filaments evolve.

The visual servoing technique has been studied extensively as a robust method for navigating robots toward goal positions and orientations reliably. Unfortunately, the existing visual servoing methods require the calibration of cameras and robots, which is time-consuming. Thus, in this paper, we propose a visual servoing method, which does not require the calibration of cameras and robots. In particular, we show that we can navigate uncalibrated robots to goal positions properly by using the projective reconstruction based on the abstract projection of control values.