Constrained visual predictive control of tendon-driven continuum robots

<p>Visual servoing (VS) is a technique which utilizes information acquired from a visual sensor to control the motion of a robot. Despite the advantages of the classic VS techniques in controlling of the robots- specifically in unstructured environments- they suffer from some system’s shortcomings such as the lack of an effective mechanism for handling the existing constraints within the control procedure. To alleviate these issues, among the several approaches proposed in the literature, the receding horizon or model predictive control (MPC) technique is an appropriate candidate to be employed by the system. Any combination of the model predictive control and visual servoing is called visual predictive control (VPC) which is categorized as the classic and hybrid methods. The classic VPC schemes are developed based on the classic VS approaches and the hybrid ones employ the hybrid VS schemes to be integrated by MPC. On the other hand, to guarantee the convergence of the system in presence of the noises and uncertainties, the proposed VPC schemes are embedded in internal model control (IMC) framework.</p> <p>This work contributes by utilizing the available VPC scheme in control of different robots, such as 3 DOF catheter as a continuum robot and a 6 DOF manipulator from Denso robotics as a robot arm, and by developing two hybrid predictive controllers, conjugated VPC and depth-based VPC, with higher constraint handling capabilities. By utilizing the proposed VPC methods, some of the available limitations could be resolved in the single-robot control level. In addition, to enable the system to perform more sophisticated tasks under difficult conditions such as unexpected occlusions, a multi-agent system controlled by the mentioned VPC methods is employed. Under these conditions, by utilizing the data sharing ability among neighboring agents inside the network, the obstructed one could be provided with enough information to complete the task. At each stage, the performance of the proposed algorithm is verified through numerous simulations and experiments to show their capability in handling existing constraints while accounting for system uncertainties.</p>

This paper aims to develop a visual servo control of a robotic manipulator for cherry tomato harvesting. In the robotic manipulator, an RGB-depth camera was mounted to the end effector to acquire the poses of the target cherry tomatoes in space. The eye-in-hand-based visual servo controller guides the end effector to implement eye–hand coordination to harvest the target cherry tomatoes, in which a hybrid visual servo control method (HVSC) with the fuzzy dynamic control parameters was proposed by combining position-based visual servo (PBVS) control and image-based visual servo (IBVS) control for the tradeoff of both performances. In addition, a novel cutting and clipping integrated mechanism was designed to pick the target cherry tomatoes. The proposed tomato-harvesting robotic manipulator with HVSC was validated and evaluated in a laboratory testbed based on harvesting implementation. The results show that the developed robotic manipulator using HVSC has an average harvesting time of 9.40 s/per and an average harvesting success rate of 96.25% in picking cherry tomatoes.

With the growing interest of integrating robotics into everyday life and industry, the requirements towards the quality and quantity of applications grows equally hard. This trend is profoundly recognized in applications involving visual perception. Whereas visual sensing in home environments tend to be mainly used for recognition and localization, safety becomes the driving factor for developing intelligent visual control algorithms. More specifically, a robot operating in a human environment should not collide with obstacles and executed motion should be as smooth as possible. Furthermore, as the environment is not known on beforehand, a high demand on the robustness of visual processing is a necessity. On the other hand, in an industrial setting, the environment is known on beforehand and safety is mainly guaranteed by excluding a human operator. Despite these reasons, and the fact that visual servoing has gained much attention from industry to become a standard solution for robotic automation tasks, applications are highly simplified. For example, methods such as visual fault detection are already a mature technique in industrial manufacturing, where a fixed camera observes a product (e.g., on a conveyor belt) and checks whether it meets certain requirements. These operations can be executed at a fairly high rate due to the simplicity of the system (e.g., static camera) and the simplification of the processing task (e.g., binary images). For both areas the identified difficulties are similar. Foremost, this is the slow nature of (robust) visual processing, in respect to the ever growing demand of increasing speed and reducing delay. These two application areas with analogous limitations motivate the design of more direct approaches of vision in visual control systems. Therefore, in order to meet the requirements for next generation visual control systems, this thesis presents approaches which employ visual measurements as a direct feedback to design constrained motion. First, for industrial robotics, in order to obtain the required positioning accuracy, the measurement and fixation system have to be highly rigid and welldesigned, implying high cost and long design time. By measuring the position of objects directly with a camera, instead of indirectly by motor encoders, the requirements of the measurement and fixation system are less demanding. Moreover, this motivates the miniaturization of the complete control system. The approach is validated in experiments on a simplified 2D planar stage (i.e., considerable friction, poor fixation), which attains similar performance compared to encoder-based positioning systems. Secondly, in a human-centred environment, this direct sensing can improve traditional visual control systems, when subject to certain disturbances. More specifically, a method is proposed that uses an image-based feedforward controller on top of traditional position-based visual servo control to overcome disturbances such as friction or poorly designed local motor controllers. This visual feedforward control action is only active when an image-based error is present and vanishes when that error goes to zero. The method is validated on an anthropomorphic robotic manipulator with 7 degrees of freedom, intended for operation in the human care environment. Third, sensing the product directly gives rise to designing motion directly. Whereas in traditional approaches the motion trajectory is designed offline and can not be changed at runtime, direct trajectory generation computes the motion of the next step based on current state and events. This means that at any instance in time, the trajectory of a motion system can be altered with respect to certain desired kinematic or dynamic constraints. For industrial applications this makes manufacturing on near-repetitive or non-rigid structures (e.g. flexible displays) possible. When applied to a robotic manipulator, this enables obstacle avoidance to no longer be on path planning level, but on trajectory planning level, where kinematic or dynamic constraints can be taken into account. This results in a motion that is smoother than when obstacle avoidance with path planning is employed. For both application areas this direct trajectory generation method is implemented and shows high flexibility in constrained motion trajectory design.

This work aims at designing a visual predictive control (VPC) scheme for a mobile manipulator. It consists in combining image-based visual servoing with model predictive control to benefit from the advantages of both control structures. Two challenges are addressed in this paper: the choice of the visual features and the closed-loop stability. The first ones rely on image moments to improve the end effector positioning precision. The second one is tackled through a terminal constraint coupled with suitable input constraints to reduce the computational burden. Simulation results using ROS and Gazebo validate the proposed approach.

Tendon-driven continuum robots possess versatile application capabilities and have a robust design. The actuation of such robots with non-straight tendons that wrap around the backbone, described by a variable function, offers a lot of untapped potentials. While it has been shown that these continuum robots are able to take up complex shapes using only one actuated segment, the merits of non-straight tendon routing have not been quantified in terms of workspace and motion. In this paper, we show that one additional helically routed tendon can greatly benefit the robot's reachable workspace. For instance, the reachable workspace of a one-segment robot with 3 conventional straight tendons increases by 400 % by adding one helically routed tendon. Furthermore, the dexterity of such a continuum robot is improved, i.e. motion sequences to avoid obstacles or to twine an object for grasping. For the first time, the potential of tendon-driven continuum robots with two segments and helically routed tendons is investigated. The general findings on the merits of helical tendon routing are supported with both simulation and experimental results.

This paper considers the question of designing a fully image-based visual servo control for a class of dynamic systems. The work is motivated by the ongoing development of image-based visual servo control of small aerial robotic vehicles. The kinematics and dynamics of a rigid-body dynamical system (such as a vehicle airframe) maneuvering over a flat target plane with observable features are expressed in terms of an un-normalized spherical centroid and an optic flow measurement. The image-plane dynamics with respect to force input are dependent on the height of the camera above the target plane. This dependence is compensated by introducing virtual height dynamics and adaptive estimation in the proposed control. A fully nonlinear adaptive control design is provided that ensures asymptotic stability of the closed-loop system for all feasible initial conditions. The choice of control gains is based on an analysis of the asymptotic dynamics of the system. Results from a realistic simulation are presented that demonstrate the performance of the closed-loop system. To the author’s knowledge, this paper documents the first time that an image-based visual servo control has been proposed for a dynamic system using vision measurement for both position and velocity.

Tendon-driven continuum robots offer exceptional flexibility and slenderness, making them particularly suitable for inspection and maintenance in constrained and unstructured environments as well as robotic surgery. However, their control remains challenging due to nonlinear behavior, limited sensory feedback, and a significant kinematic dissimilarity from existing controllers. These challenges make continuum robots, similarly to endoscopes, their non-robotic counterparts, require a cost- and time-expensive training for users who operate them in such critical environments. This paper addresses this point by designing and implementing a custom, user-friendly control interface for a tendon-driven continuum robot, leveraging principles from both robotics and gaming to enhance intuitiveness and accessibility. After developing a custom graphical user interface, a prototype has been manufactured to evaluate system performance in a dedicated testing environment. Twelve participants have attempted a timed navigation task using the proposed interface, with data collected to assess usability and control accuracy. Results confirmed the feasibility and effectiveness of the system: users were able to navigate the robot through the course with minimal training. Participant feedback is then analyzed to identify meaningful directions for improvement.

Image moments provide an important class of image features used for image-based visual servo control. Spherical image moments have the additional desirable property that they are invariant under rotation of the camera frame. For these features one can study the local and global stability and performance of the position control independently of the rotation control. In this paper we study a range of control algorithms including the classical approximately linearising control, a recently proposed robust control based on Lyapunov function design methodology, and modifications of these designs to improve global and asymptotic performance and robustness of the schemes. The comparison of performance demonstrates that the choice of image feature and control design for image-based visual servo are each equally important and highly coupled. We finally propose a control law using a modified image feature and a Lyapunov control design that ensures global asymptotic stability and good performance equally in image space as in task space

Tendon-driven continuum robots are compliant and capable of assuming complex curves, making them ideal for minimally invasive surgery in confined spaces with obstacles. These robots can achieve an infinite family of curves, determined by both actuation and design considerations, such as the tendon routing paths of tendons along the robot. Design optimization of surgical continuum robots has been studied (see [1] for review), but the idea of variable and complex tendon routing is newer, and although it has been modeled [2], it has yet to be used to design a practical surgical device. For tendon-driven continuum robots in general, design optimization has thus far focused on multi-segment robots that use straight (i.e. parallel to the backbone) segments of different lengths, where tendons terminate at various arclengths along the robot [3], [4]. The ad- vantage of designing with nonlinearly routed tendons is the potential to enable a much more expressive family of shapes during actuation [2]. However, it has thus far been challenging to design nonlinear tendons because one is working in an infinite design space with mechanics- based models that require solving differential equations to compute robot kinematics. Toward solving this problem, this paper presents a new tendon routing parameterization for tendon-driven robots and leverages it to optimize the routings for a de- sign objective drawn from a practical neurosurgical ap- plication: choroid plexus cauterization for hydrocephalus [5]. This application was an early motivating example in the development of concentric tube robots, extended in [6], although physical size, achievable curvatures, and stiffness constraints have thus far limited practical fabrication of the robots designed in simulation. The endoscopic approach to choroid plexus cauterization requires maneuvering through tight spaces in the ven- tricles, which is challenging with traditional constant curvature endoscopes (see Fig.1). Further, even in “suc- cessful” procedures, surgeons often want to cauterize more of the choroid plexus than they can currently reach and simply settle for what they can reach. Thus, a robot with nonlinear tendon routing that can access more of the choroid plexus would be a valuable tool.

Generic and high-performance feedback control is still challenging for tendon-driven continuum robots. Conventional model-based controllers, based on the piecewise constant curvature (PCC) assumption, explicitly require the arc parameters (bending angle and direction angle) to link the task (in Cartesian coordinates) and actuation spaces. However, the approaches' effectiveness remains to be explored when robot shapes deviate from circular arcs. This paper proposed a hybrid scheme for novel kinematic control of continuum robots. The error led by the slack state has been avoided through tension supervision, while analytical differential kinematics is further developed to avoid the explicit call of arc parameters by importing Cylindrical coordinates into task space and applying accurate piecewise linear approximation. Comparison between a conventional PCC-based controller and the proposed controller has been done by implementing them in a twin-pivot joint-based continuum section. An overall tip positioning accuracy of ±0.35mm has been reached, and a result of root-mean-square-error (RMSE): 0.3mm and Max error: 0.97mm has been observed when running two predefined path tracking. Further, in order to evaluate the versatility of the proposed controller, a dual-revolute joint-based and a 3D-printed continuum section were used to test for path tracking to prove the effectiveness of the controller on a wide range of continuum robotic systems.

Tendon-driven continuum robots (TDCRs) have the potential to be used in minimally invasive surgery and industrial inspection, where the robot must navigate narrow and confined spaces. We propose a model predictive controller (MPC) to leverage the non-linear kinematics and redundancy of TDCRs for whole-body collision avoidance, with real-time capabilities for hanadling inputs at 30 Hz. Key to our method's effectiveness is the integration of a nominal piecewise constant curvature (PCC) model for efficient computation of feasible trajectories, with a local feedback controller to handle modeling uncertainties and disturbances. Our experiments in simulation show that our MPC outperforms conventional Jacobian-based controller in position tracking, particularly under disturbances and user-defined shape constraints, while also allowing the incorporation of control limits. We further validate our method on a hardware prototype, showcasing its potential for enhancing the safety of teleoperation.

This paper addresses the adaptive visual servoing control for robotic manipulator under uncalibrated eye-to-hand configuration with unknown actuator backlash. Most existing control approaches to visual servoing manipulator system ignore actuator backlash constraint, which is inevitable in mechanical robot and will significantly affect the dynamic performance. To alleviate the deleterious and undesirable effect caused by the unknown actuator backlash, a smooth adaptive inverse is proposed to compensate the backlash effect Based on the visual servoing system, a novel updating law is presented to estimate the backlash model parameters. Moreover, an adaptive image-based visual servoing(IBVS) controller is constructed to guarantee the global stability of closed-loop system and the boundary of parameters. The stability of dynamic system and convergence of the image error are proved by Lyapunov method. Lastly, simulations are carried out to validate the availability of the proposed controller.

Visual predictive control of fire monitor with time delay model of fire extinguishing jet

This work addresses the problem of simultaneously controlling two robotic arms to automatically pick up fruits in an orchard. When considering such a scenario, the design of the controller has to face several challenges: (i) the arms share the same workspace, (ii) the presence of obstacles such as branches and other fruits, and (iii) the system evolves in a dynamic environment, i.e., the positions of the fruits and branches change over time. In this paper, it is proposed to control the two robotic arms relying on a Visual Predictive Control (VPC) scheme. It allows to control the system in a reactive fashion while taking into account a large number of constraints. The global and local models of the VPC scheme as well as constraints dealing with collisions or joint boundaries are presented. Finally, the efficiency of the proposed approach is highlighted with numerous simulation results using the PR2 arms model.

At the beginning of its application, arm-robot manipulator worked blindly in completing the assigned task, and along with the improvement of camera technology as it is getting cheaper and smaller, the application of visual sensor lead to visual servoing control which utilized target detection as the input to control/move the end-effector of an arm-robot. The application of camera in arm-robot manipulator is divided into eye-in-hand where a camera is mounted to the robot, and eye-to-hand where the camera is fixed elsewhere. Visual control involves kinematics modeling and target position in image plane provided by image processing to ensure the motion accuracy in grasping and harvesting the fruit. The end-effector movement for harvesting relies heavily on the accuracy of target position detection that can be obtained by image processing method. This paper is a review study that discusses the design of visual servoing control of an arm-robot manipulator applied in agriculture starting from kinematics and inverse kinematics for controlling the motion and trajectory of the robot, and image processing for target detecting. An example of an image processing method is given by using Edge Detection method for detecting a dragon fruit simulated in SCILAB.