IBVS and EMG based reach-to-grasp task planning method for a trans-humeral prosthesis

This paper presents a self-memory prediction model to mitigate the effects of image based visual servoing (IBVS) system under uncertainty. The performance of IBVS system is easily influenced by different tasks, diverse environments and uncertain disturbances. Through building a self-memory prediction model to keep previous movement tendency in the every current movement, the framework of a self-memory is proposed for incremental self-learning capability. The main challenge in IBVS system is avoid abrupt change in the operation of the system. The proposed approach is processed from state pattern, and introduces the memory function which contain multiple historical observed values. According to past events, predict the current state and avoid abrupt changes in the every current movement. Simulations on IBVS system illustrate the effectiveness of the proposed method.

This thesis deals with Visual Servoing and its strictly connected disciplines like projective geometry, image processing, robotics and non-linear control. More specifically the work addresses the problem to control a robotic manipulator through one of the largely used Visual Servoing techniques: the Image Based Visual Servoing (IBVS). In Image Based Visual Servoing the robot is driven by on-line performing a feedback control loop that is closed directly in the 2D space of the camera sensor. The work considers the case of a monocular system with the only camera mounted on the robot end effector (eye in hand configuration). Through IBVS the system can be positioned with respect to a 3D fixed target by minimizing the differences between its initial view and its goal view, corresponding respectively to the initial and the goal system configurations: the robot Cartesian Motion is thus generated only by means of visual informations. However, the execution of a positioning control task by IBVS is not straightforward because singularity problems may occur and local minima may be reached where the reached image is very close to the target one but the 3D positioning task is far from being fulfilled: this happens in particular for large camera displacements, when the the initial and the goal target views are noticeably different. To overcame singularity and local minima drawbacks, maintaining the good properties of IBVS robustness with respect to modeling and camera calibration errors, an opportune image path planning can be exploited. This work deals with the problem of generating opportune image plane trajectories for tracked points of the servoing control scheme (a trajectory is made of a path plus a time law). The generated image plane paths must be feasible i.e. they must be compliant with rigid body motion of the camera with respect to the object so as to avoid image jacobian singularities and local minima problems. In addition, the image planned trajectories must generate camera velocity screws which are smooth and within the allowed bounds of the robot. We will show that a scaled 3D motion planning algorithm can be devised in order to generate feasible image plane trajectories. Since the paths in the image are off-line generated it is also possible to tune the planning parameters so as to maintain the target inside the camera field of view even if, in some unfortunate cases, the feature target points would leave the camera images due to 3D robot motions. To test the validity of the proposed approach some both experiments and simulations results have been reported taking also into account the influence of noise in the path planning strategy. The experiments have been realized with a 6DOF anthropomorphic manipulator with a fire-wire camera installed on its end effector: the results demonstrate the good performances and the feasibility of the proposed approach.

This paper considers image-guided assembly for large composite panels. By using fiducial markers on the panels and robot gripper mounted cameras, we are able to use an industrial robot to align the panels to sub-millimeter accuracy. We considered two commonly used visual servoing schemes: position-based visual servoing (PBVS) and image-based visual servoing (IBVS). It has been noted that IBVS possesses superior robustness with respect to the camera calibration accuracy. However, we have found that in our case, PBVS is both faster and slightly more accurate than IBVS. This result is due to the fact that the visual servoing target in the image plane is derived from a reference target, which depends on the accuracy of the camera model. This additional dependency essentially nullifies the robustness advantage of IBVS. We also implemented a simple scheme to combine inputs from multiple cameras to improve the visual servoing accuracy. Both simulation and experimental results are included to show the effectiveness of visual servoing in an industrial setting.

A review on visual servoing for underwater vehicle manipulation systems automatic control and case study

This paper addresses the challenges of choosing proper image features for planar symmetric shape objects and designing visual servoing controller to enhance the tracking performance in image-based visual servoing (IBVS). Six image moments are chosen as the image features and the analytical image interaction matrix related to the image features are derived. A controller is designed to efficiently increase the robustness of the visual servoing system. Experimental results on a 6-DOF robot visual servoing system are provided to illustrate the effectiveness of the proposed method.

Image-based visual servoing (IBVS) can reach a desired position for a relatively stationary target using continuous visual feedback. Proper feature extraction and appropriate servoing control laws are essential to performance for IBVS. IBVS control can be interrupted or interfered abruptly if no features are extracted when the observed object is occluded. To address the problem of missing feature points in current images during a visual navigation task, a homography method that uses a priori visual information is proposed to predict all of the missing feature points and to ensure the execution of IBVS. The mixture parameter for the image Jacobian matrix can also affect the control of IBVS. The settings for the mixture parameter are heuristic so there is no a systematic approach for most IBVS applications. An adaptive control approach is proposed to determine the mixture parameter. The proposed method uses a reinforcement learning (RL) method to adaptively adjust the mixture parameter during the robot movement, which allows more efficient control than a constant parameter. A logarithmic interval state-space partition for RL is used to ensure efficient learning. The integrated visual servoing control system is validated by several experiments that involve wheeled mobile robots reaching a target with a desired configuration. The results for simulation and experiment demonstrate that the proposed method has a faster convergence rate than other methods.

A key problem of Image Based Visual Servo (IBVS) System is to track objects in image sequences. Thus, the tracking algorithm plays an important role in improving the efficiency of IBVS systems. In this paper, a novel tracking algorithm called Modified CamShift Guided Particle Filter (MCAMSGPF) is proposed, which interpolated Speeded-Up Robust Features (SURF) into the framework of conventional CamShift Guided Particle Filter (CAMSGPF) tracking method. This new algorithm outperforms conventional CAMSGPF and other baseline trackers with respect to tracking robustness in the clutter background of similar colors and occlusions. We also proposed a new system model to implement and test the new algorithm in a real time moving IBVS system, which is applied in a mobile robot with an on-board camera.

U ovom preglednom radu su prikazane metode vizuelnog upravljanja robotskih sistema, sa primarnim fokusom na mobilne robote sa diferencijalnim pogonom. Analizirane su standardne metode vizuelnog upravljanja bazirane na (i) greškama u parametrima slike (engl. Image-Based Visual Servoing - IBVS) i (ii) izdvojenim karakteristikama sa slike neophodnim za estimaciju položaja izabranog objekta (engl. Position-Based Visual Servoing - PBVS) i poređene sa novom metodom direktnog vizuelnog upravljanja (engl. Direct Visual Servoing - DVS). U poređenju sa IBVS i PBVS metodama, DVS metod se odlikuje višom tačnošću, ali i manjim domenom konvergencije. Zbog ovog razloga je DVS metod upravljanja pogodan za integraciju u hibridne sisteme vizuelnog upravljanja. Takođe, predstavljeni su radovi koji unapređuju sistem vizuelnog upravljanja korišćenjem stereo sistema (sistem sa dve kamere). Stereo sistem, u poređenju sa alternativnim metodama, omogućava tačniju ocenu dubine karakterističnih objekata sa slike, koja je neophodna za zadatke vizuelnog upravljanja. Predmet analize su i radovi koji integrišu tehnike veštačke inteligencije u sistem vizuelnog upravljanja. Ovim tehnikama sistemi vizuelnog upravljanja dobijaju mogućnost da uče, čime se njihov domen primene znatno proširuje. Na kraju, napominje se i mogućnost integracije vizuelne odometrije u sisteme vizuelnog upravljanja, što prouzrokuje povećanje robusnosti čitavog robotskog sistema.

We present an approach to visual feedback control using image-based visual servoing with stereo vision. In order to control the position and orientation of a robot with respect to an object, a technique is proposed using binocular stereo vision. The stereo vision enables us to calculate an exact image Jacobian not only at around a desired location but also at the other locations. The suggested technique can guide a robot manipulator to the desired location without giving such priori knowledge as the relative distance to the desired location or the model of an object even if the initial positioning error is large. The paper describes a model of stereo vision and how to generate feedback commands. The performance of the proposed visual servoing system is illustrated by the simulation and experimental results and compared with the case of conventional method for an assembling robot.

The research on acceleration-level visual servoing of manipulators is crucial yet insufficient, which restricts the potential application range of visual servoing. To address this issue, this paper proposes a quadratic programming-based acceleration-level image-based visual servoing (AIVS) scheme, which considers joint constraints. Besides, aiming to address the unknown problems in visual servoing systems, a data-driven learning algorithm is proposed to facilitate estimating structural information. Building upon this foundation, a data-driven acceleration-level image-based visual servoing (DAIVS) scheme is proposed, integrating learning and control capabilities. Subsequently, a recurrent neural network (RNN) is developed to tackle the DAIVS scheme, followed by theoretical analyses substantiating its stability. Afterwards, simulations and experiments on a Franka Emika Panda manipulator with eye-in-hand structure and comparisons among the existing methods are provided. The obtained results demonstrate the feasibility and practicality of the proposed schemes and highlight the superior learning and control ability of the proposed RNN. This method is particularly well-suited for visual servoing applications of manipulators with unknown structure.

Visual servoing is a method to achieve precise positioning and motion control of objects by visual feedback, and it is widely applied in the fields of robotics and unmanned aerial vehicles (UAVs) in recent years. This paper presents a novel image-based visual servoing (IBVS) control method for UAVs based on fuzzy logic to effectively solve the problem under field of view constraint and improve the control efficiency. In this paper, a fuzzy logic of servo gain is designed for the control input of visual servoing, which solves the problem of feature loss in IBVS and improves the efficiency. Meanwhile, a deep computing method based on known data is proposed to solve the unknown depth of Jacobian matrix, which makes the control easier to converge. The effectiveness of the proposed method is verified by the simulation of a quadrotor UAV equipped with a monocular camera.

This paper develops a robust vision-based mobile manipulation system for wheeled mobile robots. In particular, the paper addresses the retention of visual features in the field of view of the camera, which is an important robustness issue in visual servoing. First, the classical approach of Image-Based Visual Servoing (IBVS) for fixed-base manipulators is extended to wheeled mobile robots. Second, in order to guarantee visibility of visual features, an innovative controller with machine learning using Q-learning is proposed, which can learn its behavior policy and autonomously improve its performance. Third, a hybrid controller for robust mobile manipulation is developed to integrate the IBVS controller and the Q-learning controller through a rule-based arbitrator. This is thought to be the first work that integrates Reinforcement Learning or Q-learning with visual servoing to achieve robust operation. Experiments are carried out to validate the approaches developed in this paper. The experimental results show that the new hybrid controller developed here possesses the capabilities of self-learning and fast response, and provides a balanced performance with respect to robustness and accuracy.

In this report, we improve the motion discrimination method from electromyogram (EMG) for a prosthetic hand and propose prosthetic hand control. In the past, we proved that a motion discrimination method using conic models could discriminate three hand motions without the incorrect discriminations that the elbow motions cause. In this research, to increase discrimination accuracy of motion discrimination using conic models, we propose a feature extraction method using quadratic polynomials. Additionally, because many prosthetic hands using motion discrimination have constant motion speed that can't be controlled, we propose an angular velocity generation method using multiple regression models. We verified these methods by controlling the 3D hand model. In the experiment, the proposed method could discriminate five motions at a rate of above 90 percent without the incorrect discriminations that elbow motions cause. Moreover, the wrist joint angle of the 3D hand model could be controlled by standard variation of 3[deg] or less.

In this article, a robust nonlinear model predictive control (NMPC) scheme is proposed for the visual servoing of quadrotors subject to external disturbances. By using the virtual camera approach, the image moments are defined in the virtual camera plane and adopted as visual features to derive the decoupled image kinematics. As a result, the image-based visual servoing (IBVS) system model is established by integrating the image kinematics and quadrotor dynamics. To handle the visibility constraint, a robust NMPC scheme is developed for the IBVS of the quadrotor such that the visual target can stay within the field of view of the camera. In addition, based on the Lipschitz condition, the tightened state constraints are constructed to tackle external disturbances. The sufficient conditions on guaranteeing recursive feasibility of the proposed NMPC algorithm are derived. Furthermore, we theoretically show that the tracking error will converge to a small set around the origin in finite time under some derived conditions. Finally, simulation studies and experimental tests are conducted to verify the efficacy of the proposed method.

This paper addresses the adaptive visual servoing control for robotic manipulator under uncalibrated eye-to-hand configuration with unknown actuator backlash. Most existing control approaches to visual servoing manipulator system ignore actuator backlash constraint, which is inevitable in mechanical robot and will significantly affect the dynamic performance. To alleviate the deleterious and undesirable effect caused by the unknown actuator backlash, a smooth adaptive inverse is proposed to compensate the backlash effect Based on the visual servoing system, a novel updating law is presented to estimate the backlash model parameters. Moreover, an adaptive image-based visual servoing(IBVS) controller is constructed to guarantee the global stability of closed-loop system and the boundary of parameters. The stability of dynamic system and convergence of the image error are proved by Lyapunov method. Lastly, simulations are carried out to validate the availability of the proposed controller.