Dynamics-Based Visual Tracking of Robot Pose With Camera Extrinsic Online Calibration

This work presents different proposals to deal with common problems in robot visual servoing based on the application of discontinuous control methods. The feasibility and effectiveness of the proposed approaches are substantiated by simulation results and real experiments using a 6R industrial manipulator. The main contributions are:\n\n- Geometric invariance using sliding mode control (Chapter 3): the defined higher-order invariance is used by the proposed approaches to tackle problems in visual servoing. Proofs of invariance condition are presented.\n\n- Fulfillment of constraints in visual servoing (Chapter 4): the proposal uses sliding mode methods to satisfy mechanical and visual constraints in visual servoing, while a secondary task is considered to properly track the target object. The main advantages of the proposed approach are: low computational cost, robustness and fully utilization of the allowed space for the constraints.\n\n- Robust auto tool change for industrial robots using visual servoing (Chapter 4): visual servoing and the proposed method for constraints fulfillment are applied to an automated solution for tool changing in industrial robots. The robustness of the proposed method is due to the control law of the visual servoing, which uses the information acquired by the vision system to close a feedback control loop. Furthermore, sliding mode control is simultaneously used in a prioritized level to satisfy the aforementioned constraints. Thus, the global control accurately places the tool in the warehouse, but satisfying the robot constraints.\n\n- Sliding mode controller for reference tracking (Chapter 5): an approach based on sliding mode control is proposed for reference tracking in robot visual servoing using industrial robot manipulators. The novelty of the proposal is the introduction of a sliding mode controller that uses a high-order discontinuous control signal, i.e., joint accelerations or joint jerks, in order to obtain a smoother behavior and ensure the robot system stability, which is demonstrated with a theoretical proof.\n\n- PWM and PFM for visual servoing in fully decoupled approaches (Chapter 6): discontinuous control based on pulse width and pulse frequency modulation is proposed for fully decoupled position based visual servoing approaches, in order to get the same convergence time for camera translation and rotation.\n\nMoreover, other results obtained in visual servoing applications are also described.

This paper presents a review of characteristics of the two main visual servoing methods: image-based visual servoing and position-based visual servoing. First, a generic structure is defined with an emphasis on the same system inputs and outputs for the two methods. The stability, robustness, sensitivity, and other general characteristics of the methods are presented. To overcome the dependency of both methods on a known target model, a midpoint triangulation method is proposed to estimate the unknown target model for robot visual servoing.

In this paper, the uncalibrated visual servoing problem of robot manipulators with motor dynamics will be addressed for the fixed-camera configuration. A new adaptive image-space visual servoing strategy is presented to handle uncertainties in the camera intrinsic and extrinsic parameters, robot kinematic and dynamic parameters, and motor dynamic parameters. To deal with the nonlinear dependence of image Jacobian matrix on the unknown parameters, the proposed scheme is developed based on the concept of depth-independent interaction matrix. In this way, the camera parameters and the robot kinematic parameters in the closed-loop dynamics can be linearly parameterized such that adaptive laws can be designed to estimate them on-line. Adaptive algorithms are also developed to provide estimation of unknown robot dynamic and motor dynamic parameters. Stability analysis will be performed to show asymptotic convergence of image errors using Lyapunov theory based on both rigid-link robot dynamics and full motor dynamics. Simulation results based on a two-link planar robot manipulators will be given to illustrate the performance of the proposed scheme.

Initialization and choice of adequate motion models are two important but seldom discussed problems in 3D model-based pose (position and orientation) tracking. In this paper, we propose an automatic initialization approach suitable for textured objects. In addition, we define, study and experimentally evaluate three motion models commonly used in visual servoing and augmented reality

We propose and demonstrate an efficient pose estimation method, specifically small tilt angle variation, for visual servo control system design using a monocular camera, eliminating the need for iterative computations. This paper addresses the challenges associated with accurately estimating the relative pose between a chaser, equipped with a monocular camera, and a target that provides visual cues, particularly in scenarios where the captured image shows only minimal variations in the target's tilt angle. Traditional approaches often rely on direct feedback methods from image data or designs for pose estimation and control separately, necessitating computationally expensive iterations for pose estimation or complex implementation of visual servoing. To address these challenges, we adopt an extended Kalman filter, known as a robust nonlinear observer, and propose a non‐iterative approach that integrates motion dynamics models and control inputs into pose estimation from image data. This approach enables efficient integration of pose estimation and control solely relying on a monocular camera, reducing computational costs and improving robustness. Experimental validation using the ball‐and‐plate system demonstrates the effectiveness of our proposed method. In our setup, the camera is deliberately positioned directly above the pose estimation target, creating a scenario with minimal image variation in the tilt angle. Despite this condition, our approach achieves efficient and accurate pose estimation, demonstrating its feasibility for high‐speed pose estimation and control without iterative computations.

U ovom preglednom radu su prikazane metode vizuelnog upravljanja robotskih sistema, sa primarnim fokusom na mobilne robote sa diferencijalnim pogonom. Analizirane su standardne metode vizuelnog upravljanja bazirane na (i) greškama u parametrima slike (engl. Image-Based Visual Servoing - IBVS) i (ii) izdvojenim karakteristikama sa slike neophodnim za estimaciju položaja izabranog objekta (engl. Position-Based Visual Servoing - PBVS) i poređene sa novom metodom direktnog vizuelnog upravljanja (engl. Direct Visual Servoing - DVS). U poređenju sa IBVS i PBVS metodama, DVS metod se odlikuje višom tačnošću, ali i manjim domenom konvergencije. Zbog ovog razloga je DVS metod upravljanja pogodan za integraciju u hibridne sisteme vizuelnog upravljanja. Takođe, predstavljeni su radovi koji unapređuju sistem vizuelnog upravljanja korišćenjem stereo sistema (sistem sa dve kamere). Stereo sistem, u poređenju sa alternativnim metodama, omogućava tačniju ocenu dubine karakterističnih objekata sa slike, koja je neophodna za zadatke vizuelnog upravljanja. Predmet analize su i radovi koji integrišu tehnike veštačke inteligencije u sistem vizuelnog upravljanja. Ovim tehnikama sistemi vizuelnog upravljanja dobijaju mogućnost da uče, čime se njihov domen primene znatno proširuje. Na kraju, napominje se i mogućnost integracije vizuelne odometrije u sisteme vizuelnog upravljanja, što prouzrokuje povećanje robusnosti čitavog robotskog sistema.

Adaptive Control of Robots Using Visual Feedback

Clothing structure design is a key link in the clothing manufacturing process, and the traditional design methods rely on the experience and skills of designers. With the development and application of digital technology in the apparel industry, it has greatly promoted the innovation and development of apparel manufacturing. In this paper, we combine the depth estimation and vision controller methods to construct an image Jacobi matrix to realize the control of the hand-eye mapping relationship of the robot's visual servo, which represents the mapping relationship between the robot's joint speed to the end speed. Using Kalman filtering algorithm, the image Jacobi matrix to be estimated is used as the system state to estimate the system state, so as to achieve the control of the stitch and displacement of the garment sewing, and, at the same time, capture the visual information contained in the garment to optimize the design of the garment structure. For the optimized designed garments using the Jacobi matrix, 4# has the highest mean comfort rating of 4.5 and above. The mean value of satisfaction evaluation for ease of movement and overall comfort of the optimized garment went up to 4. It is evident that the overall comfort of the garment optimized by the image Jacobi matrix algorithm has been significantly improved.

Visual servoing is very useful for navigating robots to specific positions and orientations accurately and reliably. Unfortunately, the existing methods require accurate camera calibration tasks that are troublesome. Whenever a camera is changed or intrinsic parameters become different, we should carry out camera calibration tasks. Methods in order to cope with this problem is so called self-calibration techniques. Until now, self-calibration techniques just have considered the case of static self-calibration where these estimated intrinsic parameters are not used to control the robot. Only a few researchers have recently developed the case of dynamic self-calibration where estimated intrinsic parameters are used to control the robot. In this paper, a visual servoing procedure which is performed in parallel to a self-calibration algorithm based on simplified kruppa equations is proposed. Simulation results show that estimation of both constant and varying intrinsic parameters is efficient and the robot converge to the desired positions

Vision allows a robotic system to obtain a lot of information on the surrounding environment to be used for motion planning and control. When the control is based on feedback of visual information is called Visual Servoing. Visual Servoing is a powerful tool which allows a robot to increase its interaction capabilities and tasks complexity. In this chapter we describe the architecture of the Robotenis system in order to design two different control strategies to carry out tracking tasks. Robotenis is an experimental stage that is formed of a parallel robot and vision equipment. The system was designed to test joint control and Visual Servoing algorithms and the main objective is to carry out tasks in three dimensions and dynamical environments. As a result the mechanical system is able to interact with objects which move close to 2m=s. The general architecture of control strategies is composed by two intertwined control loops: The internal loop is faster and considers the information from the joins, its sample time is 0:5ms. Second loop represents the visual Servoing system and it is an external loop to the first mentioned. The second loop represents the main study purpose, it is based in the prediction of the object velocity that is obtained from visual information and its sample time is 8:3ms. The robot workspace analysis plays an important role in Visual Servoing tasks, by this analysis is possible to bound the movements that the robot is able to reach. In this article the robot jacobian is obtained by two methods. First method uses velocity vector-loop equations and the second is calculated from the time derivate of the kinematical model of the robot. First jacobian requires calculating angles from the kinematic model. Second jacobian instead, depends on physical parameters of the robot and can be calculated directly. Jacobians are calculated from two different kinematic models, the first one determines the angles each element of the robot. Fist jacobian is used in the graphic simulator of the system due to the information that can be obtained from it. Second jacobian is used to determine off-line the work space of the robot and it is used in the joint and visual controller of the robot (in real time). The work space of the robot is calculated from the condition number of the jacobian (this is a topic that is not studied in article). The dynamic model of the mechanical system is based on Lagrange multipliers, and it uses forearms and end effector platform of non-negligible inertias for the

In this paper we consider the tracking control problem of planar robots manipulators via visual servoing in the presence of parametric uncertainties associated with the robot dynamics and the camera parameters. An image–based visual servo control is developed using a Takagi–Sugeno (T–S) fuzzy model. The design includes an image coordinate velocity observer. To the best knowledge of the authors such a proposal remains hitherto unpublished. The new scheme is compared experimentally with two different non–fuzzy algorithms showing the good performance and advantages of the complete system.

Visual servoing is a method to achieve precise positioning and motion control of objects by visual feedback, and it is widely applied in the fields of robotics and unmanned aerial vehicles (UAVs) in recent years. This paper presents a novel image-based visual servoing (IBVS) control method for UAVs based on fuzzy logic to effectively solve the problem under field of view constraint and improve the control efficiency. In this paper, a fuzzy logic of servo gain is designed for the control input of visual servoing, which solves the problem of feature loss in IBVS and improves the efficiency. Meanwhile, a deep computing method based on known data is proposed to solve the unknown depth of Jacobian matrix, which makes the control easier to converge. The effectiveness of the proposed method is verified by the simulation of a quadrotor UAV equipped with a monocular camera.

With the growing interest of integrating robotics into everyday life and industry, the requirements towards the quality and quantity of applications grows equally hard. This trend is profoundly recognized in applications involving visual perception. Whereas visual sensing in home environments tend to be mainly used for recognition and localization, safety becomes the driving factor for developing intelligent visual control algorithms. More specifically, a robot operating in a human environment should not collide with obstacles and executed motion should be as smooth as possible. Furthermore, as the environment is not known on beforehand, a high demand on the robustness of visual processing is a necessity. On the other hand, in an industrial setting, the environment is known on beforehand and safety is mainly guaranteed by excluding a human operator. Despite these reasons, and the fact that visual servoing has gained much attention from industry to become a standard solution for robotic automation tasks, applications are highly simplified. For example, methods such as visual fault detection are already a mature technique in industrial manufacturing, where a fixed camera observes a product (e.g., on a conveyor belt) and checks whether it meets certain requirements. These operations can be executed at a fairly high rate due to the simplicity of the system (e.g., static camera) and the simplification of the processing task (e.g., binary images). For both areas the identified difficulties are similar. Foremost, this is the slow nature of (robust) visual processing, in respect to the ever growing demand of increasing speed and reducing delay. These two application areas with analogous limitations motivate the design of more direct approaches of vision in visual control systems. Therefore, in order to meet the requirements for next generation visual control systems, this thesis presents approaches which employ visual measurements as a direct feedback to design constrained motion. First, for industrial robotics, in order to obtain the required positioning accuracy, the measurement and fixation system have to be highly rigid and welldesigned, implying high cost and long design time. By measuring the position of objects directly with a camera, instead of indirectly by motor encoders, the requirements of the measurement and fixation system are less demanding. Moreover, this motivates the miniaturization of the complete control system. The approach is validated in experiments on a simplified 2D planar stage (i.e., considerable friction, poor fixation), which attains similar performance compared to encoder-based positioning systems. Secondly, in a human-centred environment, this direct sensing can improve traditional visual control systems, when subject to certain disturbances. More specifically, a method is proposed that uses an image-based feedforward controller on top of traditional position-based visual servo control to overcome disturbances such as friction or poorly designed local motor controllers. This visual feedforward control action is only active when an image-based error is present and vanishes when that error goes to zero. The method is validated on an anthropomorphic robotic manipulator with 7 degrees of freedom, intended for operation in the human care environment. Third, sensing the product directly gives rise to designing motion directly. Whereas in traditional approaches the motion trajectory is designed offline and can not be changed at runtime, direct trajectory generation computes the motion of the next step based on current state and events. This means that at any instance in time, the trajectory of a motion system can be altered with respect to certain desired kinematic or dynamic constraints. For industrial applications this makes manufacturing on near-repetitive or non-rigid structures (e.g. flexible displays) possible. When applied to a robotic manipulator, this enables obstacle avoidance to no longer be on path planning level, but on trajectory planning level, where kinematic or dynamic constraints can be taken into account. This results in a motion that is smoother than when obstacle avoidance with path planning is employed. For both application areas this direct trajectory generation method is implemented and shows high flexibility in constrained motion trajectory design.

PurposeThe purpose of this paper is to present the control of six degrees of freedom (PUMA560) robotic arm using visual servoing, based upon linear matrix inequality (LMI). The aim lies in developing such a method that neither involves camera calibration parameters nor inverse kinematics. The approach adopted in this paper includes transpose Jacobian control; thus, inverse of the Jacobian matrix is no longer required. By invoking the Lyapunov's direct method, closed‐loop stability of the system is ensured. Simulation results are shown for three different cases, which exhibit the system stability and convergence even in the presence of large errors.Design/methodology/approachThe paper presents LMI‐based visual servo control of PUMA560 robotic arm.FindingsThe proposed method is implementable in the dynamic environment due to its independence to camera and object model.Research limitations/implicationsVisibility constraint is not included during servoing – this may cause features to leave the camera field of view (fov).Originality/valueLMI optimization is employed for visual servo control in an uncalibrated environment. Lyapunov's direct method is utilized which ensures system stability and convergence.

In this paper, a novel visual servoing technique for a 5-dof mobile manipulator with an eye-to-hand camera configuration is introduced. The proposed technique can be categorized as an image based (or 2D) visual servoing using a fixed camera in conjunction with a conic mirror (aka, a Catadioptric camera system) providing panoramic vision. Two fictitious landmarks mounted on robot's end-effector along with their mirror reflections, when viewed by the camera, provide enough information for 3D reasoning based on the four points viewed on the image plane. Instead of directly using the image features associated with these four points, five new image features are chosen to make the image Jacobian rank efficient. A dual estimation/control strategy based on Extended Kalman Filter (EKF) is utilized to (1) estimate camera's intrinsic and extrinsic parameters, and (2) track the coordinates of the landmarks and their reflections on the image plane. The relationship between the translational and rotational velocity of a frame attached to the robot's end-effector and the rate of change of the proposed image features are fully formulated. The robustness of the proposed technique in translational and rotational servoing of a 5-dof holonomic mobile manipulator is illustrated through computer simulations.