Automatic Tuning Methodology of Visual Servoing System Using Predictive Approach

This work presents different proposals to deal with common problems in robot visual servoing based on the application of discontinuous control methods. The feasibility and effectiveness of the proposed approaches are substantiated by simulation results and real experiments using a 6R industrial manipulator. The main contributions are:\n\n- Geometric invariance using sliding mode control (Chapter 3): the defined higher-order invariance is used by the proposed approaches to tackle problems in visual servoing. Proofs of invariance condition are presented.\n\n- Fulfillment of constraints in visual servoing (Chapter 4): the proposal uses sliding mode methods to satisfy mechanical and visual constraints in visual servoing, while a secondary task is considered to properly track the target object. The main advantages of the proposed approach are: low computational cost, robustness and fully utilization of the allowed space for the constraints.\n\n- Robust auto tool change for industrial robots using visual servoing (Chapter 4): visual servoing and the proposed method for constraints fulfillment are applied to an automated solution for tool changing in industrial robots. The robustness of the proposed method is due to the control law of the visual servoing, which uses the information acquired by the vision system to close a feedback control loop. Furthermore, sliding mode control is simultaneously used in a prioritized level to satisfy the aforementioned constraints. Thus, the global control accurately places the tool in the warehouse, but satisfying the robot constraints.\n\n- Sliding mode controller for reference tracking (Chapter 5): an approach based on sliding mode control is proposed for reference tracking in robot visual servoing using industrial robot manipulators. The novelty of the proposal is the introduction of a sliding mode controller that uses a high-order discontinuous control signal, i.e., joint accelerations or joint jerks, in order to obtain a smoother behavior and ensure the robot system stability, which is demonstrated with a theoretical proof.\n\n- PWM and PFM for visual servoing in fully decoupled approaches (Chapter 6): discontinuous control based on pulse width and pulse frequency modulation is proposed for fully decoupled position based visual servoing approaches, in order to get the same convergence time for camera translation and rotation.\n\nMoreover, other results obtained in visual servoing applications are also described.

Robots have now far more impact in humans life then ten years ago. Vacuum cleaning robots are already well known. Making today's robots to work unassisted requires appropriate visual servoing architecture. In the past, a lot of efforts were directed towards designing controllers that relies exclusively on image data. Still most robots are servoed kinematically using joint data. Visual servoing architecture has applications not only in robotics. Video cameras are often mounted on platforms that can move like rovers, booms, gantries and aircrafts. People can operate such platforms to capture desired views of a scene or a target. To avoid collisions, with the environment and occlusions, such platforms demands much skill. Visual-servoing some degrees-of-freedom may reduce the operator burden and improve tracking. We call this concept human-in-the-loop visual servoing. Human-in-the-loop systems involve an operator who manipulates a device for desired tasks based on feedback from the device and environment. For example, devices like rovers gantries and aircrafts possess a video camera. The task is to control maneuver the vehicle and position the camera to obtain desired fields of view. To overcome joint limits, avoid collisions and ensure occlusion-free views, these devices are typically equipped with redundant degrees-of-freedom. Tracking moving subjects with such systems is a challenging task and requires a well skilled operator. In this approach, we use computer vision techniques to visually servo the camera. The net effect is that the operator just focuses on safely manipulating the boom and dolly while computer-control automatically servos the camera.

Visual servoing (VS) is an automatic control technique which uses the feedback of visual information to guide the robot motion to converge to the desired location. This paper proposes new moment-based features for robust VS with spheres, especially in cameras obeying the unified model. Specifically, we introduce a virtual plane to reload contours of image projections of spheres, in the process of which image noises will however be greatly magnified and induce large converge error. To solve this, new moment-based features are proposed and extracted on this virtual plane to enhance the robustness of VS to calibration errors and image noises. Simulation results verify the effectiveness of the proposed scheme and demonstrate the high robustness of these set of new features to image noises. The altering trends of converge errors versus increasing noise levels are randomly tested to present the obvious advantage of new moment-based features over the usual ones. At last, a path-planning based VS application is performed with fisheye views of some spheres, benefiting from the robustness and an interpolation-based tracking scheme.

In this paper, we describe an experimental facility for studying robotic grasping of objects in space. This problem arises in several applications, including on-orbit servicing of satellites and removal of space debris. The facility is based on a novel concept for experimental evaluation of robotic capture of freefloating objects. The central idea behind it is to use a small helium airship to emulate a free-floating object. Over the past year, a facility has been developed at McGill University to implement this concept in a laboratory setting. The main components of our facility are: a seven-degree-of-freedom robot, a spherical helium airship 5 ft in diameter, a stereo-based vision system and control hardware. One key issue that had to be addressed for the airship is how to make it balanced and neutrally buoyant. Control architecture has been developed allowing the airship to fly under computer control. The function of the robot is to intercept the airship and grab it by a grapple fixture. A ‘look then move’ visual servoing architecture has been implemented allowing the robot to follow a target. The paper describes the main components of the facility and the algorithms implemented for balloon balancing, robot control and visual servoing.

Visual servoing has been around for decades, but time delay is still one of the most troublesome problems to achieve target tracking. To circumvent the problem, in this paper, the Kalman filter is employed to estimate the future position of the object. In order to introduce the Kalman filter, accurate time delays, which include the processing lag and the motion lag, need to be obtained. Thus, the delays of the visual control servoing systems are discussed and a generic timing model for the system is provided. Then, we present a current statistical model for a moving target. A fuzzy adaptive Kalman filter, which is evolved from the Kalman filter, is introduced based on the current statistical model. Finally, a two DOF visual controller based variable structure control law for micro-manipulation is presented. The results show that the proposed adaptive Kalman filter can improve the ability to track moving targets.

For soft continuum arms, visual servoing is a popular control strategy that relies on visual feedback to close the control loop. However, robust visual servoing is challenging as it requires reliable feature extraction from the image, accurate control models and sensors to perceive the shape of the arm, both of which can be hard to implement in a soft robot. This letter circumvents these challenges by presenting a deep neural network-based method to perform smooth and robust 3D positioning tasks on a soft arm by visual servoing using a camera mounted at the distal end of the arm. A convolutional neural network is trained to predict the actuations required to achieve the desired pose in a structured environment. Integrated and modular approaches for estimating the actuations from the image are proposed and are experimentally compared. A proportional control law is implemented to reduce the error between the desired and current image as seen by the camera. The model together with the proportional feedback control makes the described approach robust to several variations such as new targets, lighting, loads, and diminution of the soft arm. Furthermore, the model lends itself to be transferred to a new environment with minimal effort.

Visual servoing systems are designed to solve pose alignment problems by providing the necessary linear and angular velocities using data extracted from images. Among the difficulties encountered by the traditional visual servoing approaches, there are feature detection and tracking, camera calibration, scene complexity, and robotic system constraints. Part of these problems can be solved if Convolutional Neural Networks (CNNs) are added to a visual servoing architecture. The main advantage of CNNs is the capability of understanding both the overall structure and specific details of the images corresponding to the current and desired layouts. To take a step further the state-of-the-art architectures, in this paper, we show how extra input data can improve the visual servoing behaviour. The extra data result from maps of regions induced by the feature points' positions, without the necessity of employing tracking. The results obtained on relevant data sets show that the proposed input fusion-based CNN provides an improved approximation of the linear and angular visual servoing velocities.

Presents an approach to real time implementation of visual feedback control for a robotic manipulator. There are mainly two ways to put the visual feedback into practice. One is called look-and-move and the other is visual servoing. The former is the method which transforms the position and orientation of an object obtained by a visual sensor into those in the world frame fixed to an environment and guides the arm of the manipulator to a desired location in the world frame. Visual servoing uses the Jacobian matrix which relates the displacement of an image feature to the displacement of a camera motion and performs a closed-loop control regarding the feature as a scale of the state. Therefore, we can construct a servo system based only on the image and can have a robust control against the calibration error because there is no need to calculate the corresponding location in the world frame. In order to control the position and orientation of a robot with respect to an object, a new technique is proposed using binocular stereo vision. The stereo vision enables us to calculate an exact image Jacobian not only around a desired location but also at other locations. The suggested technique can guide a robot manipulator to the desired location without giving such priori knowledge as the relative distance to the desired location or the model of an object even if the initial positioning error is large. The performance of the proposed visual servoing system is illustrated by the experimental results and compared with the case of conventional method for an assembling robot with four joints.

The research on acceleration-level visual servoing of manipulators is crucial yet insufficient, which restricts the potential application range of visual servoing. To address this issue, this paper proposes a quadratic programming-based acceleration-level image-based visual servoing (AIVS) scheme, which considers joint constraints. Besides, aiming to address the unknown problems in visual servoing systems, a data-driven learning algorithm is proposed to facilitate estimating structural information. Building upon this foundation, a data-driven acceleration-level image-based visual servoing (DAIVS) scheme is proposed, integrating learning and control capabilities. Subsequently, a recurrent neural network (RNN) is developed to tackle the DAIVS scheme, followed by theoretical analyses substantiating its stability. Afterwards, simulations and experiments on a Franka Emika Panda manipulator with eye-in-hand structure and comparisons among the existing methods are provided. The obtained results demonstrate the feasibility and practicality of the proposed schemes and highlight the superior learning and control ability of the proposed RNN. This method is particularly well-suited for visual servoing applications of manipulators with unknown structure.

A PC-based visual-servoing automatic microassembly system was developed and tested. With compliant PU microgripper, regional-edge-statistics algorithm, and PD controller, a visual servoing system was implemented for gripping micro object, gluing adhesive, and operating adhesive bonding. The main specifications of the system are: gripping range of 60~80 mum , working space of 7 mmtimes5.74 mmtimes15 mm, system bandwidth of 15 Hz. In the performance test, a copper wire with diameter 60 mum was automatically gripped and transported for adhesive bonding with a copper wire of 380 mum.

In this paper, a visual servoing robotic control scheme of fast convergence and calibration-free is proposed. The objective of this paper is to achieve a better convergence around the equilibrium for uncalibrated visual servoing systems. Moreover, there are threefold unknown parameters, the visual kinematic parameters, the dynamic parameters and the parameters of the feature points, considered in the design of the proposed control scheme. In order to achieve the above control objective, the finite-time tracking controller and three adaptive laws are proposed. The adaptability to the unknown parameters is guaranteed by the online adaptive laws. The finite-time convergence is achieved by the continuously non-smooth fractional order function in the controller. The rigorously mathematic proof of stability is given by homogeneous theory and the Lyapunov function formalism. Three real-time experiments are conducted to demonstrate the practical performance of the proposed scheme.

The separate design of vision algorithms and motion control algorithms in existing vision servo systems limits their application in high-speed and high-precision industrial scenarios. To address this problem, a vision-motion codesign method is proposed for integrating visual servo systems with motion controllers via real-time Ethernet connections. During the engineering design phase of our proposal, we developed a control kernel architecture to clarify the system design logic. Then, we proposed a runtime module for real-time trajectory generation to facilitate the unified deployment of different planning methods in the kernel. On this basis, we implemented a self-adaptive trajectory generation synthesis method capable of dynamically adjusting the motion resources for real-time interpolation. The experimental results demonstrate that in visual servo static tracking experiments with the SCARA robot, our proposed method has a faster response time and smaller impact than other mainstream trajectory generation methods in 1 ms real-time interpolation cycles.

Prosthesis is a device that replaces a missing body part. These devices are used to reinstate the lost functionality due to an amputation. Above elbow amputees are equipped with trans-humeral prostheses. The existing prostheses use biological signals such as electromyography (EMG) in combine with external sensors for the controlling purposes. Vision is the latest addition to external sensor domain to eliminate the drawbacks of existing hybrid prostheses. This paper proposes a vision based task planning method in combination with available EMG signals to plan reach-to-grasp tasks of a trans-humeral prosthesis. The visual servoing system consists of an Image Based Visual Servoing (IBVS) system in conjunction with an Artificial Neural Network (ANN) to center the object in the image frame. Moreover, a gradient descent algorithm is proposed for the elbow motion to reach the palm towards the object of interest. EMG based system is capable of controlling the prosthetic elbow motion with available muscles' signals. EMG based system is fused with the IBVS system to perform elbow motions when an object is visible to the camera. Experiments are carried out using a virtual prosthesis to validate the proposed task planning method.

This paper presents an application of feature-based visual servoing to achieve accurate and robust 3D filament alignment. Two orthogonal cameras are used to localize the five degrees of freedom of an axi-symmetric filament. An algorithm for precisely estimating the center and orientation features of a filament in a camera view is described. The use of feature-based servoing overcomes the difficulty of 3D camera calibration in a factory environment. These features drive a PID control loop for each view. The two views are coupled in one degree of freedom, and the multiple-view controller switches between two single-view controllers to achieve 3D servoing. The overall accuracy achieved is in thousandths of an inch. Experimental results on the performance of the control algorithm are discussed. Models explaining the system behavior are presented. The use of visual servoing in this application far exceeds the positioning accuracy and the repeatability of human operators.

Intra-operative imaging is widely used to provide visual feedback to a clinician when he/she performs a procedure. In visual servoing, surgical instruments and parts of tissue/body are tracked by processing the acquired images. This information is then used within a control loop to manoeuvre a robotic manipulator during a procedure. A comprehensive search of electronic databases was completed for the period 2000-2013 to provide a survey of the visual servoing applications in medical robotics. The focus is on medical applications where image-based tracking is used for closed-loop control of a robotic system. Detailed classification and comparative study of various contributions in visual servoing using endoscopic or direct visual images are presented and summarized in tables and diagrams. The main challenges in using visual servoing for medical robotic applications are identified and potential future directions are suggested. 'Supervised automation of medical robotics' is found to be a major trend in this field.