Unified Passivity-Based Visual Control for Moving Object Tracking

This paper aims to develop a visual servo control of a robotic manipulator for cherry tomato harvesting. In the robotic manipulator, an RGB-depth camera was mounted to the end effector to acquire the poses of the target cherry tomatoes in space. The eye-in-hand-based visual servo controller guides the end effector to implement eye–hand coordination to harvest the target cherry tomatoes, in which a hybrid visual servo control method (HVSC) with the fuzzy dynamic control parameters was proposed by combining position-based visual servo (PBVS) control and image-based visual servo (IBVS) control for the tradeoff of both performances. In addition, a novel cutting and clipping integrated mechanism was designed to pick the target cherry tomatoes. The proposed tomato-harvesting robotic manipulator with HVSC was validated and evaluated in a laboratory testbed based on harvesting implementation. The results show that the developed robotic manipulator using HVSC has an average harvesting time of 9.40 s/per and an average harvesting success rate of 96.25% in picking cherry tomatoes.

With the growing interest of integrating robotics into everyday life and industry, the requirements towards the quality and quantity of applications grows equally hard. This trend is profoundly recognized in applications involving visual perception. Whereas visual sensing in home environments tend to be mainly used for recognition and localization, safety becomes the driving factor for developing intelligent visual control algorithms. More specifically, a robot operating in a human environment should not collide with obstacles and executed motion should be as smooth as possible. Furthermore, as the environment is not known on beforehand, a high demand on the robustness of visual processing is a necessity. On the other hand, in an industrial setting, the environment is known on beforehand and safety is mainly guaranteed by excluding a human operator. Despite these reasons, and the fact that visual servoing has gained much attention from industry to become a standard solution for robotic automation tasks, applications are highly simplified. For example, methods such as visual fault detection are already a mature technique in industrial manufacturing, where a fixed camera observes a product (e.g., on a conveyor belt) and checks whether it meets certain requirements. These operations can be executed at a fairly high rate due to the simplicity of the system (e.g., static camera) and the simplification of the processing task (e.g., binary images). For both areas the identified difficulties are similar. Foremost, this is the slow nature of (robust) visual processing, in respect to the ever growing demand of increasing speed and reducing delay. These two application areas with analogous limitations motivate the design of more direct approaches of vision in visual control systems. Therefore, in order to meet the requirements for next generation visual control systems, this thesis presents approaches which employ visual measurements as a direct feedback to design constrained motion. First, for industrial robotics, in order to obtain the required positioning accuracy, the measurement and fixation system have to be highly rigid and welldesigned, implying high cost and long design time. By measuring the position of objects directly with a camera, instead of indirectly by motor encoders, the requirements of the measurement and fixation system are less demanding. Moreover, this motivates the miniaturization of the complete control system. The approach is validated in experiments on a simplified 2D planar stage (i.e., considerable friction, poor fixation), which attains similar performance compared to encoder-based positioning systems. Secondly, in a human-centred environment, this direct sensing can improve traditional visual control systems, when subject to certain disturbances. More specifically, a method is proposed that uses an image-based feedforward controller on top of traditional position-based visual servo control to overcome disturbances such as friction or poorly designed local motor controllers. This visual feedforward control action is only active when an image-based error is present and vanishes when that error goes to zero. The method is validated on an anthropomorphic robotic manipulator with 7 degrees of freedom, intended for operation in the human care environment. Third, sensing the product directly gives rise to designing motion directly. Whereas in traditional approaches the motion trajectory is designed offline and can not be changed at runtime, direct trajectory generation computes the motion of the next step based on current state and events. This means that at any instance in time, the trajectory of a motion system can be altered with respect to certain desired kinematic or dynamic constraints. For industrial applications this makes manufacturing on near-repetitive or non-rigid structures (e.g. flexible displays) possible. When applied to a robotic manipulator, this enables obstacle avoidance to no longer be on path planning level, but on trajectory planning level, where kinematic or dynamic constraints can be taken into account. This results in a motion that is smoother than when obstacle avoidance with path planning is employed. For both application areas this direct trajectory generation method is implemented and shows high flexibility in constrained motion trajectory design.

There is significant motivation to provide robotic systems with improved autonomy as a means to significantly accelerate deactivation and decommissioning (D&D) operations while also reducing the associated costs, removing human operators from hazardous environments, and reducing the required burden and skill of human operators. To achieve improved autonomy, this project focused on the basic science challenges leading to the development of visual servo controllers. The challenge in developing these controllers is that a camera provides 2-dimensional image information about the 3-dimensional Euclidean-space through a perspective (range dependent) projection that can be corrupted by uncertainty in the camera calibration matrix and by disturbances such as nonlinear radial distortion. Disturbances in this relationship (i.e., corruption in the sensor information) propagate erroneous information to the feedback controller of the robot, leading to potentially unpredictable task execution. This research project focused on the development of a visual servo control methodology that targets compensating for disturbances in the camera model (i.e., camera calibration and the recovery of range information) as a means to achieve predictable response by the robotic system operating in unstructured environments. The fundamental idea is to use nonlinear Lyapunov-based techniques along with photogrammetry methods to overcome the complex control issues and alleviate many of the restrictive assumptions that impact current robotic applications. The outcome of this control methodology is a plug-and-play visual servoing control module that can be utilized in conjunction with current technology such as feature recognition and extraction to enable robotic systems with the capabilities of increased accuracy, autonomy, and robustness, with a larger field of view (and hence a larger workspace). The developed methodology has been reported in numerous peer-reviewed publications and the performance and enabling capabilities of the resulting visual servo control modules have been demonstrated on mobile robot and robot manipulator platforms.

A mobile manipulator system is developed to work as a guard robot in a modern building. Several prototypes of advanced task-oriented service robots have been reported. However, a breakthrough to mobile manipulation is still required on the fact that most mobile robots can only work on a single floor of the building. It is desirable that the robot can perform various tasks in every floor of a modern building. Consequently, the robot must be able to take conventional elevators. It is well recognized that techniques of visual servo are key factors for successful mobile manipulation. This paper presents a neural network for real-time clustering of objects in an image for tracking control. The system can successfully recognize each individual button on the elevator panel. Then a novel window-based visual tracking control scheme is proposed for guiding the mobile manipulator to push the correct panel button. The new method uses two one-dimensional windows for visual servo control Experimental results reveal that this method performs much better than the standard techniques.

For a visual servo system, there usually exists the problem of time-delay likely caused by long image processing and data transmission. The visual servo system of our robot is subject to two main limitations stemming from the specific commercial mobile manipulator, of which one is the large time-delay due to image transmission, whereas the other is failure to directly command each joint velocity as the visual servo's control input. In this paper, we propose a novel visual servo system to compensate the large time-delay, and we use integral of velocity which incorporating the image estimation and prediction with the kinematic model so as to achieve indirect velocity control with position based control mode joint. Our proposed visual servo system controls the neck joints of robot and keeps the object within the field of view of the camera during the approaching phase. We evaluate the proposed approach by experiment on a real wheeled humanoid robot and find the resulting performance quite promising, showing that our method is feasible and practical.

Wheeled Mobile robots (WMRs) are very popular because they are appropriate for applications with relatively low mechanical complexity and energy consumption. WMRs are used in many military and civilian applications. These applications include surveillance, transportation, inspection, exploration, and many other applications either in structured or unstructured environments. This paper provides a review on the developments of the wheeled mobile robots (WMRs) control. It reviews the kinematic and dynamic modeling of the robots, the path following and tracking, the avoiding obstacles control algorithms, and finally a review on robot vision and visual servoing control system is presented including the main challenges in each field of research.

This paper presents a novel combination of visual servoing (VS) control and neural network (NN) learning on humanoid dual-arm robot. A VS control system is built by using stereo vision to obtain the 3D point cloud of a target object. A least square-based method is proposed to reduce the stochastic error in workspace calibration. An NN controller is designed to compensate for the effect of uncertainties in payload and other parameters (both internal and external) during the tracking control. In contrast to the conventional NN controller, a deterministic learning technique is utilized in this work, to enable the learned neural knowledge to be reused before current dynamics changes. A skill transfer mechanism is also developed to apply the neural learned knowledge from one arm to the other, to increase the neural learning efficiency. Tracked trajectory of object is used to provide target position to the coordinated dual arms of a Baxter robot in the experimental study. Robotic implementations has demonstrated the efficiency of the developed VS control system and has verified the effectiveness of the proposed NN controller with knowledge-reuse and skill transfer features.

This paper is oriented to the transmission line inspection environment, aiming to solve the problem of how to control the robotic arm to quickly and accurately grasp the target object, and work on the planning of the motion path and the position-based visual servo control method. For the spatial trajectory planning of the robotic arm, linear trajectory planning and circular arc trajectory planning are carried out respectively. Between the reaction speed problem of the traditional RRT motion algorithm in collision-free motion planning, a two-way RRT-Connect path planning method is proposed. Then the visual servo control algorithm of the UAV robotic arm is proposed. The position-based visual servo control method is adopted, while the EK-SVSF position estimation algorithm and the OWA-based stepwise data fusion algorithm are combined for optimal position estimation of the target during transmission line inspection. It is verified in simulation experiments that the improved RRT algorithm can obtain the shortest path planning in a shorter time with better performance. Finally, the grasping experiment is carried out in a real robotic arm transmission line inspection scene. The robotic arm path planning algorithm and visual servo control proposed in this paper are combined to carry out grasping experiments in a static cluttered environment. The experimental results show that the algorithm proposed in this paper has the best grasping success rate, servo success rate, and grasping similarity. In addition, the average errors in x-axis, y-axis and z-axis of the proposed method are 0.19cm, 1.11cm and 0.18cm, respectively, which show a good grasping accuracy, indicating that the control method proposed in this paper can be applied to the real transmission line inspection work scenarios.

Recent progress in bistatic radar techniques can be used to improve performances of classical monostatic radar. A prominent limitation of coherent radar is its inability to measure the complete velocity vector (magnitude and direction) of a detected target. A single coherent detection can provide range-rate only. At least two detections, separated in time, are needed to estimate the target's velocity vector. This study discusses how the velocity vector can be determined by two simultaneous detections spaced in distance. The second detection is obtained by an auxiliary distant bistatic coherent receiver; an approach proposed in the 1990s to enhance meteorological radar. Being a very simple case of a distributed radar system allows for a simple demonstration of how to calculate the target's position and velocity vector and how to analyse the estimation accuracy, including geometric dilution of precision plots of the velocity error. Also discussed are two methods to identify correct data association when more than one target is detected.

The document proposes the development of a 3D virtual environment, oriented to the common work activities between a unicycle type mobile manipulator robot and a human operator, in collaborative tasks. This strategy is focuses on the incorporation of virtual reality (VR), in which the operator will have access to visualize in an immersive way the behavior of the mobile manipulator robot in common tasks where the human being and the mobile manipulator robot interact. For the interaction between the human operator and the mobile manipulator robot, the graphic engine Unity 3D is used, which exchanges information with the mathematical software Matlab, in order to execute the control algorithm through the use of shared memories. The Novint Falcon haptic device allows human-robot interaction, which provides the operator with force feedback on what is happening in the virtual environment generated by the Unity 3D software and the interaction it has with the mobile manipulator. The HTC Vive immersion device allows the operator to visualize the virtual environment created for the execution of the task. In this work, the design and simulation of the locomotion system of a mobile manipulator robot is carried out for manipulation and object transfer tasks together with the human operator. Finally, the simulation results that validate the proposed control strategy are presented and discussed.

This thesis deals with Visual Servoing and its strictly connected disciplines like projective geometry, image processing, robotics and non-linear control. More specifically the work addresses the problem to control a robotic manipulator through one of the largely used Visual Servoing techniques: the Image Based Visual Servoing (IBVS). In Image Based Visual Servoing the robot is driven by on-line performing a feedback control loop that is closed directly in the 2D space of the camera sensor. The work considers the case of a monocular system with the only camera mounted on the robot end effector (eye in hand configuration). Through IBVS the system can be positioned with respect to a 3D fixed target by minimizing the differences between its initial view and its goal view, corresponding respectively to the initial and the goal system configurations: the robot Cartesian Motion is thus generated only by means of visual informations. However, the execution of a positioning control task by IBVS is not straightforward because singularity problems may occur and local minima may be reached where the reached image is very close to the target one but the 3D positioning task is far from being fulfilled: this happens in particular for large camera displacements, when the the initial and the goal target views are noticeably different. To overcame singularity and local minima drawbacks, maintaining the good properties of IBVS robustness with respect to modeling and camera calibration errors, an opportune image path planning can be exploited. This work deals with the problem of generating opportune image plane trajectories for tracked points of the servoing control scheme (a trajectory is made of a path plus a time law). The generated image plane paths must be feasible i.e. they must be compliant with rigid body motion of the camera with respect to the object so as to avoid image jacobian singularities and local minima problems. In addition, the image planned trajectories must generate camera velocity screws which are smooth and within the allowed bounds of the robot. We will show that a scaled 3D motion planning algorithm can be devised in order to generate feasible image plane trajectories. Since the paths in the image are off-line generated it is also possible to tune the planning parameters so as to maintain the target inside the camera field of view even if, in some unfortunate cases, the feature target points would leave the camera images due to 3D robot motions. To test the validity of the proposed approach some both experiments and simulations results have been reported taking also into account the influence of noise in the path planning strategy. The experiments have been realized with a 6DOF anthropomorphic manipulator with a fire-wire camera installed on its end effector: the results demonstrate the good performances and the feasibility of the proposed approach.

This study achieved four door operations, namely push-opening, push-closing, pull-opening, and pull-closing movements, using a mobile manipulator consisting of a commercially available arm robot and a mobile robot. We assumed that the arm robot is controlled by position commands at intervals of a few milliseconds, and that the mobile robot is guided by a simple straightforward linear trajectory. Ott, Borst, Bäuml, and Hirzinger proposed a push-opening method using impedance control in a cylindrical coordinate system for the arm robot. With this control, when the mobile robot advances toward and through the door, the arm robot moves passively and properly pushes the door open. However, their method is unsuitable for the above type of robot. Thus, we propose a method with two modifications: the use of admittance control and the improvement of force relaxation by considering a force obtained through a novel force decomposition. Furthermore, the proposed method was demonstrated not only in the push-opening movement but also in the push-closing, pull-opening, and pull-closing movements.

We propose seven velocity-estimation methods for fast moving targets beyond the Nyquist-limit velocity for pulse Doppler radars. The Doppler-shift frequency beyond the Nyquist limit, which equals onehalf the pulse-repetition frequency, cannot be correctly measured as the frequency is folded back within the Nyquist interval. Our methods use pulse compression with divided reference signals of varying center frequencies to detect the difference among the Doppler-shift frequencies of the pulse-compressed signals and increase sensitivity through signal integration, all without modifying the transmitted signals. These methods are either grid-search-type or Newton-type methods, each possessing distinct characteristics. The Cramér-Rao lower bounds on the variance of estimated velocity are derived, and the upper bounds on the probability of correct ambiguity resolution are shown. The results of our numerical simulation indicate that our methods can estimate target velocity beyond the Nyquistlimit velocity and that the method of independently estimating the Doppler frequencies at both end parts of the signal bandwidth and estimating the target velocity from the difference exhibits a lower decrease than the other methods regarding the success probability of Doppler ambiguity resolution even for high target velocities. We also confirmed that the upper bounds closely match the probability of corresponding methods for when the target velocity is lower than several times the Nyquist velocity and the signal-tonoise ratio on the range-Doppler map is above approximately 20 dB.

Technology is developing more and more to facilitate human life. Technology enables automation in all areas of life, and robots are among the most frequently used machines in automation. Robots can help with human work in all fields, including agriculture. A mobile robot manipulator is a combination of a robot arm and a mobile robot so that this type of robot can combine the capabilities of the two robots. This paper discusses the design of a robot manipulator to be used in agriculture to replace farmers in the harvesting of agricultural products, such as tomatoes. This paper presents a mechanical, electrical design and uses the Fuzzy Logic Controller as artificial intelligence. The feasibility of the proposed method is demonstrated by simulation in Mobotsim.

At present, mobile robotic manipulators have been greatly developed. However, these further promotions are limited by a low load capacity and short operation time. The above problems can be improved by using a hydraulic drive mode and increasing the system energy efficiency. In this paper, a novel energy-efficient wobble plate hydraulic joint is presented, which has the characteristics of having a small size, lightweight, large load capacity, and high energy efficiency. Based on the efficiency analysis in traditional robotic manipulators, this paper presents a novel hydraulic joint with a multi-chamber drive structure. Kinematics model and dynamics model are both established for the analysis of the mechanical characteristics, and the functional relationship between the input and output is depicted by numerical simulation. Based on the structural characteristics and control principle, the load matching controller is designed and specific control processes are formulated. Combined with a strategy of load matching, the servo control system is established and the energy-saving effect is verified by simulation. The result shows that the wobble plate hydraulic joint can change connections between a high-pressure circuit and different working chambers, which realizes the match between the output torque and load torque. With the load matching controller, the energy consumption of the wobble plate joint is greatly reduced, which contributes to a considerably improved energy efficiency. The research in this paper not only lays a theoretical foundation for the development of a wobble plate hydraulic joint, but also provides guidance for the improvement of the hydraulic system energy efficiency in mobile robotic manipulators.