Adaptive Control of Robots Using Visual Feedback

With the growing interest of integrating robotics into everyday life and industry, the requirements towards the quality and quantity of applications grows equally hard. This trend is profoundly recognized in applications involving visual perception. Whereas visual sensing in home environments tend to be mainly used for recognition and localization, safety becomes the driving factor for developing intelligent visual control algorithms. More specifically, a robot operating in a human environment should not collide with obstacles and executed motion should be as smooth as possible. Furthermore, as the environment is not known on beforehand, a high demand on the robustness of visual processing is a necessity. On the other hand, in an industrial setting, the environment is known on beforehand and safety is mainly guaranteed by excluding a human operator. Despite these reasons, and the fact that visual servoing has gained much attention from industry to become a standard solution for robotic automation tasks, applications are highly simplified. For example, methods such as visual fault detection are already a mature technique in industrial manufacturing, where a fixed camera observes a product (e.g., on a conveyor belt) and checks whether it meets certain requirements. These operations can be executed at a fairly high rate due to the simplicity of the system (e.g., static camera) and the simplification of the processing task (e.g., binary images). For both areas the identified difficulties are similar. Foremost, this is the slow nature of (robust) visual processing, in respect to the ever growing demand of increasing speed and reducing delay. These two application areas with analogous limitations motivate the design of more direct approaches of vision in visual control systems. Therefore, in order to meet the requirements for next generation visual control systems, this thesis presents approaches which employ visual measurements as a direct feedback to design constrained motion. First, for industrial robotics, in order to obtain the required positioning accuracy, the measurement and fixation system have to be highly rigid and welldesigned, implying high cost and long design time. By measuring the position of objects directly with a camera, instead of indirectly by motor encoders, the requirements of the measurement and fixation system are less demanding. Moreover, this motivates the miniaturization of the complete control system. The approach is validated in experiments on a simplified 2D planar stage (i.e., considerable friction, poor fixation), which attains similar performance compared to encoder-based positioning systems. Secondly, in a human-centred environment, this direct sensing can improve traditional visual control systems, when subject to certain disturbances. More specifically, a method is proposed that uses an image-based feedforward controller on top of traditional position-based visual servo control to overcome disturbances such as friction or poorly designed local motor controllers. This visual feedforward control action is only active when an image-based error is present and vanishes when that error goes to zero. The method is validated on an anthropomorphic robotic manipulator with 7 degrees of freedom, intended for operation in the human care environment. Third, sensing the product directly gives rise to designing motion directly. Whereas in traditional approaches the motion trajectory is designed offline and can not be changed at runtime, direct trajectory generation computes the motion of the next step based on current state and events. This means that at any instance in time, the trajectory of a motion system can be altered with respect to certain desired kinematic or dynamic constraints. For industrial applications this makes manufacturing on near-repetitive or non-rigid structures (e.g. flexible displays) possible. When applied to a robotic manipulator, this enables obstacle avoidance to no longer be on path planning level, but on trajectory planning level, where kinematic or dynamic constraints can be taken into account. This results in a motion that is smoother than when obstacle avoidance with path planning is employed. For both application areas this direct trajectory generation method is implemented and shows high flexibility in constrained motion trajectory design.

This paper aims to develop a visual servo control of a robotic manipulator for cherry tomato harvesting. In the robotic manipulator, an RGB-depth camera was mounted to the end effector to acquire the poses of the target cherry tomatoes in space. The eye-in-hand-based visual servo controller guides the end effector to implement eye–hand coordination to harvest the target cherry tomatoes, in which a hybrid visual servo control method (HVSC) with the fuzzy dynamic control parameters was proposed by combining position-based visual servo (PBVS) control and image-based visual servo (IBVS) control for the tradeoff of both performances. In addition, a novel cutting and clipping integrated mechanism was designed to pick the target cherry tomatoes. The proposed tomato-harvesting robotic manipulator with HVSC was validated and evaluated in a laboratory testbed based on harvesting implementation. The results show that the developed robotic manipulator using HVSC has an average harvesting time of 9.40 s/per and an average harvesting success rate of 96.25% in picking cherry tomatoes.

This paper presents a novel combination of visual servoing (VS) control and neural network (NN) learning on humanoid dual-arm robot. A VS control system is built by using stereo vision to obtain the 3D point cloud of a target object. A least square-based method is proposed to reduce the stochastic error in workspace calibration. An NN controller is designed to compensate for the effect of uncertainties in payload and other parameters (both internal and external) during the tracking control. In contrast to the conventional NN controller, a deterministic learning technique is utilized in this work, to enable the learned neural knowledge to be reused before current dynamics changes. A skill transfer mechanism is also developed to apply the neural learned knowledge from one arm to the other, to increase the neural learning efficiency. Tracked trajectory of object is used to provide target position to the coordinated dual arms of a Baxter robot in the experimental study. Robotic implementations has demonstrated the efficiency of the developed VS control system and has verified the effectiveness of the proposed NN controller with knowledge-reuse and skill transfer features.

In this paper, we propose a dynamic visual servo controller for precise tracking of robot manipulator. For the controller, we used a dual-loop structure which includes a vision controller and dynamic controller with different sampling speed. The vision controller is accommodated in the outer control loop with the aid of eye-in-hand camera, implementing the image based visual servoing method. As an inner loop controller, an adaptive backstepping controller is derived for position control of joint angle together with PID controller. The performance of proposed visual servo controller has been verified through computer simulation with V-REP simulation package.

At the beginning of its application, arm-robot manipulator worked blindly in completing the assigned task, and along with the improvement of camera technology as it is getting cheaper and smaller, the application of visual sensor lead to visual servoing control which utilized target detection as the input to control/move the end-effector of an arm-robot. The application of camera in arm-robot manipulator is divided into eye-in-hand where a camera is mounted to the robot, and eye-to-hand where the camera is fixed elsewhere. Visual control involves kinematics modeling and target position in image plane provided by image processing to ensure the motion accuracy in grasping and harvesting the fruit. The end-effector movement for harvesting relies heavily on the accuracy of target position detection that can be obtained by image processing method. This paper is a review study that discusses the design of visual servoing control of an arm-robot manipulator applied in agriculture starting from kinematics and inverse kinematics for controlling the motion and trajectory of the robot, and image processing for target detecting. An example of an image processing method is given by using Edge Detection method for detecting a dragon fruit simulated in SCILAB.

Purpose Commercial bioprinting systems are typically characterized by high costs and limited adaptability, presenting a significant barrier for academic research. This study aims to introduce and empirically validate an integral prototyping methodology for developing a low-cost, accessible robotic bioprinters. The primary objective is to provide a holistic framework that integrates mechanical design, electronics and control systems, aimed at addressing key engineering challenges. This work also addresses the critical issue of maintaining high positional accuracy in a system with a variable payload mass. Design/methodology/approach Following the Integral Prototyping framework, a multi-link robotic manipulator bioprinter was developed through an iterative process that integrated mechanical, electronic and control subsystems. A direct-drive extruder was implemented to minimize material waste by embedding cartridges into the end-effector, a design choice that created the central challenge of a variable payload. This study develops a payload-agnostic visual servoing controller. The system’s adaptability and the controller’s performance were validated by testing with five biomaterials and two 50:50 composites with diverse viscosities (gelatin-based hydrogel, lipid-rich emulsion, ground pork, starch-based hydrogel and composites), confirming the framework’s control strategy and the overall adaptability to heterogeneous biomaterials. Findings Application of the Integral Prototyping methodology resulted in a robotic bioprinter capable of sustaining high positional accuracy despite variable payloads. The visual servoing controller improved the system’s precision, reducing average positional deviations for key joint angles (e.g. from 13.18% to 6.62% for q1 and from 6.74% to 4.08% for q2), demonstrating effective compensation for mass variability introduced by the direct-drive extruder. The control system interfaces with standardized G-code to ensure both accuracy and accessibility. This control robustness translated into accurate fabrication outcomes, with dimensional deviations below 2.0% and scaffold fidelity deviations between 2.0% and 3.3%. Scaffold integrity was partially constrained given the configuration of the extrusion module. Results confirm that the control system successfully decouples positional precision from payload variability. These outcomes validate Integral Prototyping as a framework that not only integrates design, electronics and software but also resolves emergent engineering challenges central to accessible bioprinting. Originality/value The primary contribution is a reproducible Integral Prototyping methodology for developing low-cost, adaptable robotic manufacturing bioprinting platform that remain robust under variable operating condition. This work contributes in uniting design, electronics and control into a single framework while providing a validated visual servoing solution to the long-standing robotics challenge of variable end-effector mass. Beyond the specific controller, this study establishes a generalizable blueprint for rapid prototyping and tissue engineering, demonstrating that cost and flexibility barriers in bioprinting can be addressed through integrative, methodology-driven development.

The use of unmanned aerial vehicles (UAVs) for military, scientific, and civilian sectors are increasing drastically in recent years. This study presents algorithms for the visual-servo control of an UAV, in which a quadrotor helicopter has been stabilized with visual information through the control loop. Unlike previous study that use pose estimation approach which is time consuming and subject to various errors, the visual-servo control is more reliable and fast. The method requires a camera on-board the vehicle, which is already available on various UAV systems. The UAV with a camera behaves like an eye-in-hand visual servoing system. In this study the controller was designed by using two different approaches; image based visual servo control method and hybrid visual servo control method. Various simulations are developed on Matlab, in which the quadrotor aerial vehicle has been visual-servo controlled. In order to show the effectiveness of the algorithms, experiments were performed on a model quadrotor UAV, which suggest successful performance.

In this paper we study image based visual servo (IBVS) control of a four rotor vertical take-off and landing (VTOL) craft known as an X4-flyer. The approach taken is tailored to planar visual targets and is based on a reformulation and extension of 2 1/2 D image-based visual servo design to the control of dynamic under-actuated systems. Exponential stability of the closed loop system is proved and simulation results are presented.

PurposeThe purpose of this paper is to present the control of six degrees of freedom (PUMA560) robotic arm using visual servoing, based upon linear matrix inequality (LMI). The aim lies in developing such a method that neither involves camera calibration parameters nor inverse kinematics. The approach adopted in this paper includes transpose Jacobian control; thus, inverse of the Jacobian matrix is no longer required. By invoking the Lyapunov's direct method, closed‐loop stability of the system is ensured. Simulation results are shown for three different cases, which exhibit the system stability and convergence even in the presence of large errors.Design/methodology/approachThe paper presents LMI‐based visual servo control of PUMA560 robotic arm.FindingsThe proposed method is implementable in the dynamic environment due to its independence to camera and object model.Research limitations/implicationsVisibility constraint is not included during servoing – this may cause features to leave the camera field of view (fov).Originality/valueLMI optimization is employed for visual servo control in an uncalibrated environment. Lyapunov's direct method is utilized which ensures system stability and convergence.

U ovom preglednom radu su prikazane metode vizuelnog upravljanja robotskih sistema, sa primarnim fokusom na mobilne robote sa diferencijalnim pogonom. Analizirane su standardne metode vizuelnog upravljanja bazirane na (i) greškama u parametrima slike (engl. Image-Based Visual Servoing - IBVS) i (ii) izdvojenim karakteristikama sa slike neophodnim za estimaciju položaja izabranog objekta (engl. Position-Based Visual Servoing - PBVS) i poređene sa novom metodom direktnog vizuelnog upravljanja (engl. Direct Visual Servoing - DVS). U poređenju sa IBVS i PBVS metodama, DVS metod se odlikuje višom tačnošću, ali i manjim domenom konvergencije. Zbog ovog razloga je DVS metod upravljanja pogodan za integraciju u hibridne sisteme vizuelnog upravljanja. Takođe, predstavljeni su radovi koji unapređuju sistem vizuelnog upravljanja korišćenjem stereo sistema (sistem sa dve kamere). Stereo sistem, u poređenju sa alternativnim metodama, omogućava tačniju ocenu dubine karakterističnih objekata sa slike, koja je neophodna za zadatke vizuelnog upravljanja. Predmet analize su i radovi koji integrišu tehnike veštačke inteligencije u sistem vizuelnog upravljanja. Ovim tehnikama sistemi vizuelnog upravljanja dobijaju mogućnost da uče, čime se njihov domen primene znatno proširuje. Na kraju, napominje se i mogućnost integracije vizuelne odometrije u sisteme vizuelnog upravljanja, što prouzrokuje povećanje robusnosti čitavog robotskog sistema.

This paper proposes a new geometric control method for homography-based visual servo control of Underactuated UAVs. In order to solve the application difficulties of geometric control in HBVS and explore a visual servo control technology that can be applied to aerial detection operations, this paper integrates the geometric control into the visual servoing framework and design a new homography-based geometric visual servoing controller. The outer loop is used as feedback information using the virtual homography matrix between the two images. The inner loop controls the orientation of the UAVs through geometric control. The stability of the proposed controller is proved based on Lyapunov’s theory. The proposed method has better transient performance and dynamic performance than the conventional visual servo method. The excellent performance of the controller has been proven by a large number of experiments. In addition, the application of the controller on an unmanned aerial manipulator is demonstrated.

This paper proposes an uncalibrated adaptive visual servoing (VS) control framework based on dual-camera fusion to address key technical challenges in robotic visual servoing systems, including real-time state estimation, multi-space coordination, and dynamic target tracking. By combining the complementary advantages of “eye-in-hand” and “eye-to-hand” camera configurations, an adaptive switching mechanism is designed to achieve coordinated control between image space and Cartesian space, addressing convergence problems of traditional methods during target occlusion or field-of-view loss. Key features of the framework include: an uncalibrated control method based on the image Jacobian matrix; adaptive parameter adjustment based on Kalman filtering (KF); and a dual-camera fusion switching strategy. Experiments show the method achieves a positioning accuracy of 1.197 mm and an orientation accuracy of 0.149° in a representative static positioning task; demonstrates effective performance in scenarios involving out-of-view target acquisition and occlusion recovery; and reduces tracking errors by 13%–28% while shortening convergence time by 5%–32% in dynamic tracking tasks. This framework provides a practical technical approach for visual servoing systems in complex environments, showing potential for broad industrial applications.

This paper has two purposes: investigating a featureless visual servoing approach based on mutation analysis and proposing a visual servo control method for nanomanipulations. For the first purpose, the featureless visual servoing method is needed because traditional visual servoing relies heavily on robust feature extraction and tracking, which are very difficult in natural environment. The mutation analysis based approach in this paper considers the image as a set, and designs a controller to make the distance between the initial and goal image sets converge to zero, thereby steering the initial image to the goal image. For the second purpose, atomic force microscopic (AFM) based nanomanipulations with subnanometer accuracy are very difficult because the position sensor cannot provide valuable feedback due to large noises at this precision level. We propose to use the images obtained by AFM and perform a visual servo control. This method, independent of external sensors, can directly perform control on the AFM end tip's position. The featureless controller is successfully validated on AFM images and the results suggest a potential precision enhancement for nanomanipulations.

This paper is oriented to the transmission line inspection environment, aiming to solve the problem of how to control the robotic arm to quickly and accurately grasp the target object, and work on the planning of the motion path and the position-based visual servo control method. For the spatial trajectory planning of the robotic arm, linear trajectory planning and circular arc trajectory planning are carried out respectively. Between the reaction speed problem of the traditional RRT motion algorithm in collision-free motion planning, a two-way RRT-Connect path planning method is proposed. Then the visual servo control algorithm of the UAV robotic arm is proposed. The position-based visual servo control method is adopted, while the EK-SVSF position estimation algorithm and the OWA-based stepwise data fusion algorithm are combined for optimal position estimation of the target during transmission line inspection. It is verified in simulation experiments that the improved RRT algorithm can obtain the shortest path planning in a shorter time with better performance. Finally, the grasping experiment is carried out in a real robotic arm transmission line inspection scene. The robotic arm path planning algorithm and visual servo control proposed in this paper are combined to carry out grasping experiments in a static cluttered environment. The experimental results show that the algorithm proposed in this paper has the best grasping success rate, servo success rate, and grasping similarity. In addition, the average errors in x-axis, y-axis and z-axis of the proposed method are 0.19cm, 1.11cm and 0.18cm, respectively, which show a good grasping accuracy, indicating that the control method proposed in this paper can be applied to the real transmission line inspection work scenarios.

This paper proposes a new controller for dynamic visual servoing of robotic manipulators. In this work the image is acquired from a CCD camera mounted on planar robotic manipulator end-effector, and processed in a standard PC based vision system in order to close the visual servoing loop with PD controllers. First, we propose an extension to dynamic visual servoing based on a pose estimation algorithm and then compare it with another Lyapunov stable controller. The proposed dynamic controller uses only information from the image and robot encoders. This approach is inspired on classical 3D visual servoing and the Jacobian transpose controller approach. An experimental setup developed for visual servoing experiments, including software and hardware is presented. Experimental results for a planar robotic manipulator are presented.