Predictive Control Architecture for Visual Servoing of Robot Manipulators

This paper deals with real-time implementation of visual-motor control of a 7 degree of freedom (DOF) robot manipulator using self-organized map (SOM) based learning approach. The robot manipulator considered here is a 7 DOF PowerCube manipulator from Amtec Robotics. The primary objective is to reach a target point in the task space using only a single step movement from any arbitrary initial configuration of the robot manipulator. A new clustering algorithm using Kohonen SOM lattice has been proposed that maintains the fidelity of training data. Two different approaches have been proposed to find an inverse kinematic solution without using any orientation feedback. In the first approach, the inverse Jacobian matrices are learnt from the training data using function decomposition. It is shown that function decomposition leads to significant improvement in accuracy of inverse kinematic solution. In the second approach, a concept called sub-clustering in configuration space is suggested to provide multiple solutions for the inverse kinematic problem. Redundancy is resolved at position level using several criteria. A redundant manipulator is dexterous owing to the availability of multiple configurations for a given end-effector position. However, existing visual motor coordination schemes provide only one inverse kinematic solution for every target position even when the manipulator is kinematically redundant. Thus, the second approach provides a learning architecture that can capture redundancy from the training data. The training data are generated using explicit kinematic model of the combined robot manipulator and camera configuration. The training is carried out off-line and the trained network is used on-line to compute the joint angle vector to reach a target position in a single step only. The accuracy attained is better than the current state of art.

Visual servoing of robot manipulators is a key technique where the appearance of an object in the image plane is used to control the velocity of the end-effector such that the desired position is reached in the scene. The vast majority of visual servoing methods proposed so far uses calibrated robots in conjunction with calibrated cameras. It has been shown that the behavior of visual control loops does not degrade too much in the presence of calibration errors. Nevertheless, camera and robot calibration are complex and time-consuming processes requiring special-purpose mechanical devices, such as theodolites and calibration jigs. In this paper, we suggest formulating a visual servoing control loop in a nonmetric space, which in our case amounts to the projective space in which a triangulation of the scene using an uncalibrated stereo rig is expressed. The major consequence of controlling the robot in nonmetric space rather than in Euclidean space is that both the robot’s direct kinematic map and the robot’s Jacobian matrix must be defined in this space as well. The elementary joint-space motions that can be performed by a robot manipulator are pure rotations and pure translations. Traditionally, these motions are represented as Euclidean transformations. Since these motions are observed with an uncalibrated stereo rig, it will be convenient to represent them as projective transformations (homographies) rather than Euclidean transformations. Indeed, it will be shown that rotations and translations can be parameterized as special cases of homographies, which will be called projective rotations and projective translations. The algebraic properties of this nonmetric representation of elementary motions will be thoroughly investigated allowing us to characterize the direct kinematic map and the Jacobian matrix of a manipulator. Therefore, we introduce the concepts of projective kinematics and a projective Jacobian matrix. Unlike the classical robot Jacobian matrix of a manipulator that relates the robot joint-velocities to the kinematic screw associated with the end-effector, we establish a direct relationship between joint-velocities and image-plane velocities. The latter are velocities associated with image points arising from the 3-D to 2-D projection of end-effector points. Finally, we provide a practical method to estimate the projec tive kinematic model and we describe some preliminary simulated experiments that use this nonmetric model to perform stereo-based servoing. Nevertheless, in-depth analysis of projective control will be the topic of a forthcoming paper.

In this paper, the uncalibrated visual servoing problem of robot manipulators with motor dynamics will be addressed for the fixed-camera configuration. A new adaptive image-space visual servoing strategy is presented to handle uncertainties in the camera intrinsic and extrinsic parameters, robot kinematic and dynamic parameters, and motor dynamic parameters. To deal with the nonlinear dependence of image Jacobian matrix on the unknown parameters, the proposed scheme is developed based on the concept of depth-independent interaction matrix. In this way, the camera parameters and the robot kinematic parameters in the closed-loop dynamics can be linearly parameterized such that adaptive laws can be designed to estimate them on-line. Adaptive algorithms are also developed to provide estimation of unknown robot dynamic and motor dynamic parameters. Stability analysis will be performed to show asymptotic convergence of image errors using Lyapunov theory based on both rigid-link robot dynamics and full motor dynamics. Simulation results based on a two-link planar robot manipulators will be given to illustrate the performance of the proposed scheme.

In this paper we consider the tracking control problem of planar robots manipulators via visual servoing in the presence of parametric uncertainties associated with the robot dynamics and the camera parameters. An image–based visual servo control is developed using a Takagi–Sugeno (T–S) fuzzy model. The design includes an image coordinate velocity observer. To the best knowledge of the authors such a proposal remains hitherto unpublished. The new scheme is compared experimentally with two different non–fuzzy algorithms showing the good performance and advantages of the complete system.

Real-time collision avoidance is a critical challenge in visual servoing control for safe robot manipulation. This article presents a model-free framework that integrates reinforcement learning (RL) with real-time collision detection (RCD), enabling secure and smooth visual servoing in dynamic environments. The reward function in the RL framework is carefully designed to simultaneously minimize tracking errors, promote motion smoothness, and enforce collision avoidance. Beyond policy optimization through RL, a two-step real-time trajectory blending (RTB) scheme, consisting of a Butterworth filter (BF) and polynomial interpolation, is developed to further refine the generated trajectories and ensure smooth joint velocity profiles. Furthermore, the soft actor-critic (SAC) algorithm is employed for policy learning, and the performance is compared with two widely used methods, namely proximal policy optimization (PPO) and twin delayed deep deterministic policy gradient (TD3). Experimental results on a 6-degree-of-freedom (6-DoF) robot manipulator demonstrate the effectiveness of the proposed approach in achieving safe and smooth visual servoing in real time.

SUMMARYIn most dynamic adaptive control simulation of robotic manipulators, the Langrange–Euler (L–E) dynamic equations are first piecewise linearized about the desired reference and then discretized and rewritten in a state space form. This makes things very complicated and it is easy to make errors. What is more is that with a different reference this work must be done again. A new simulation scheme – Backward Recursive Self-Tuning Adaptive (BRSTA) – as it will be called, is suggested in this paper for adaptive controller design of robot manipulators. A two degree of freedom robot manipulator is used to verify the scheme in the condition of highly nonlinear and highly coupled system. A one degree of freedom robot manipulator is used for comparing both the forward and backward methods. The main advantages of this scheme include that it can be used for evaluating the self-tuning adaptive control laws and provide the initial process parameters for real-time control. And it is concluded here that the Newton–Euler (N–E) dynamic equations are equally well qualified as the Langrange–Euler (L-E) equations for the simulation of self-tuning adaptive control of robot manipulators.

In this paper, we propose a dynamic visual servo controller for precise tracking of robot manipulator. For the controller, we used a dual-loop structure which includes a vision controller and dynamic controller with different sampling speed. The vision controller is accommodated in the outer control loop with the aid of eye-in-hand camera, implementing the image based visual servoing method. As an inner loop controller, an adaptive backstepping controller is derived for position control of joint angle together with PID controller. The performance of proposed visual servo controller has been verified through computer simulation with V-REP simulation package.

In this study, trajectory tracking control of six–degrees of freedom robotic manipulator has been performed using the proposed fractional-order adaptive integral sliding mode control scheme. The proposed method is first composed by fractional-order and adaptive integral sliding mode control to achieve the finite-time convergence, chattering-free control inputs, better tracking performance and robustness for the robotic manipulator. Furthermore, an adaptive method has been utilized to evaluate and compensate the uncertain and unknown dynamics of the system without relying on the prior knowledge of the upper bounds. To illustrate the efficiency of the proposed fractional-order adaptive integral sliding mode control method, the real-time experimental studies have been performed for the six–degrees of freedom industrial robot manipulator. The experimental outcomes strongly verified that the proposed controller gives quite well trajectory tracking response and faster convergence compared with the classical integral sliding mode control under the external payload.

Increasing demands of flexible automating manufacturing processes made robots with vision systems to be frequently used due to their ability to adapt to new environments and various conditions. This paper presents a visual servoing system for real time picking and placing of dynamic objects. The system is integrated into a manufacturing cell and its structure consists in a CCD camera with SIFT detector, an image based predictive controller and a robot manipulator. Using a Service Oriented Architecture (SOA), the visual servoing system is implemented and different experimental testing of the vision-guided robot predictive control architecture is conducted.

Visual servoing control of robot manipulators represents a natural option in unstructured environments and it is specially attractive when the task to achieve can be expressed directly in image coordinates. In the present work a control law is developed for trajectory tracking of three degrees of freedom robot manipulators. The main characteristic of the proposed algorithm is that it does not need any dynamic model of the system and image coordinates are employed directly for feedback, while an observer is designed for velocity estimation. To carry out 3D movements no stereoscopic vision is used. Rather, a simpler configuration employing two fixed cameras is proposed which makes unnecessary their calibration and helps avoid occlusions for most of the robots workspace. The developed theory was successfully tested in an industrial manipulator.

This thesis deals with Visual Servoing and its strictly connected disciplines like projective geometry, image processing, robotics and non-linear control. More specifically the work addresses the problem to control a robotic manipulator through one of the largely used Visual Servoing techniques: the Image Based Visual Servoing (IBVS). In Image Based Visual Servoing the robot is driven by on-line performing a feedback control loop that is closed directly in the 2D space of the camera sensor. The work considers the case of a monocular system with the only camera mounted on the robot end effector (eye in hand configuration). Through IBVS the system can be positioned with respect to a 3D fixed target by minimizing the differences between its initial view and its goal view, corresponding respectively to the initial and the goal system configurations: the robot Cartesian Motion is thus generated only by means of visual informations. However, the execution of a positioning control task by IBVS is not straightforward because singularity problems may occur and local minima may be reached where the reached image is very close to the target one but the 3D positioning task is far from being fulfilled: this happens in particular for large camera displacements, when the the initial and the goal target views are noticeably different. To overcame singularity and local minima drawbacks, maintaining the good properties of IBVS robustness with respect to modeling and camera calibration errors, an opportune image path planning can be exploited. This work deals with the problem of generating opportune image plane trajectories for tracked points of the servoing control scheme (a trajectory is made of a path plus a time law). The generated image plane paths must be feasible i.e. they must be compliant with rigid body motion of the camera with respect to the object so as to avoid image jacobian singularities and local minima problems. In addition, the image planned trajectories must generate camera velocity screws which are smooth and within the allowed bounds of the robot. We will show that a scaled 3D motion planning algorithm can be devised in order to generate feasible image plane trajectories. Since the paths in the image are off-line generated it is also possible to tune the planning parameters so as to maintain the target inside the camera field of view even if, in some unfortunate cases, the feature target points would leave the camera images due to 3D robot motions. To test the validity of the proposed approach some both experiments and simulations results have been reported taking also into account the influence of noise in the path planning strategy. The experiments have been realized with a 6DOF anthropomorphic manipulator with a fire-wire camera installed on its end effector: the results demonstrate the good performances and the feasibility of the proposed approach.

A new evolutionary-based algorithm is proposed to solve the robot manipulator optimal path generation problem. The following scenario is considered: given a learnt joint path describing a robot manipulator simple task in the Cartesian space, an optimal path is calculated when a different initial joint configuration is considered. The optimization problem is formulated as the minimization of both the end-effector pose error and the total joint displacement so as to ensure convergence towards the learnt path and a smooth joint motion. To solve the optimization problem an algorithm based on an evolutionary method called Differential Evolution (DE) is used. DE is a stochastic direct search optimization method based on the evolution of a candidate solution population in an iterative process of mutation, recombination, and selection. Since the algorithm does not require the use of the Jacobian matrix during the kinematic inversion, singularities problems are overcome. Results on the optimal path generation of a six degrees of freedom robot manipulator are also presented.

In the impedance control scheme in robotic manipulation applications, the interaction force can be indirectly controlled by acting on the manipulator reference position trajectory. However, the reference position depends on the environment parameters, like location and stiffness, which in practice are not exactly known. To deal with those uncertainties, this paper presents a fuzzy adaptive approach which determines the appropriate reference (target) positions to an impedance controller in order to obtain the desired force/position trajectory. Moreover, only an estimate of the environment location is necessary, and so the fuzzy adaptive controller corrects this estimated position based on the force error. The fuzzy adaptive controller is a kind of self organizing controller, which consists of a direct fuzzy controller and an adaptation mechanism, (the fuzzy inverse model) which adjusts the membership functions of the consequent parts of the direct fuzzy controller. To validate the proposed control scheme, simulation results with a two degree of freedom robot manipulator are presented where it is shown a good force/tracking performance with uncertainties on the environment location and on the stiffness value.

In the paper fuzzy logic controller for rehabilitation robot manipulator is presented. The control object is two degree of freedom robot manipulator. In this case there are two main problems. The first is, that the object is for rehabilitation process so the quality of control system is very important problem. The next problem is with the highly nonlinear dynamics of a robot manipulator. Suggest fuzzy logic control system for that reasons. The model-based Takagi-Sugeno fuzzy control strategy is applied. Described model have two inputs (error and derivation) and one output. The first part of research, was the simulation tests. For this part simulation model was prepare. The dynamic equations was used to build a model in Matlab. During this research the main problem was suitable control quality. It is very important because control quality is connected with rehabilitation process. The control algorithms enable to movement realization and obtain set position given by the physiotherapist. The tests results show that the fuzzy logic control have good control quality.

This paper presents a decentralized control scheme, based on a recurrent neural identifier with a block control structure, and its application to robotics manipulators. A local joint controller is proposed for each joint, using only local angular position and velocity measurements. These very simple local joint controllers allow trajectory tracking, with reduced computations. The applicability of the proposed scheme is illustrated, via simulations, first by the applications to a two degree of freedom robotic manipulator and then to a seven degree of freedom one.