Robotic assembly of smartphone back shells with eye-in-hand visual servoing

Presents an approach to real time implementation of visual feedback control for a robotic manipulator. There are mainly two ways to put the visual feedback into practice. One is called look-and-move and the other is visual servoing. The former is the method which transforms the position and orientation of an object obtained by a visual sensor into those in the world frame fixed to an environment and guides the arm of the manipulator to a desired location in the world frame. Visual servoing uses the Jacobian matrix which relates the displacement of an image feature to the displacement of a camera motion and performs a closed-loop control regarding the feature as a scale of the state. Therefore, we can construct a servo system based only on the image and can have a robust control against the calibration error because there is no need to calculate the corresponding location in the world frame. In order to control the position and orientation of a robot with respect to an object, a new technique is proposed using binocular stereo vision. The stereo vision enables us to calculate an exact image Jacobian not only around a desired location but also at other locations. The suggested technique can guide a robot manipulator to the desired location without giving such priori knowledge as the relative distance to the desired location or the model of an object even if the initial positioning error is large. The performance of the proposed visual servoing system is illustrated by the experimental results and compared with the case of conventional method for an assembling robot with four joints.

Hand-eye relation in visual servoing involves eye-in-hand and eye-to-hand configuration. Both have its own merit and drawback regarding to its precision and sight range. This paper addresses this problem and introduces the camera utilization as eye-to-hand configuration for the second robot, while it retains as eye-in-hand for the first robot. Hence, the camera becomes mobile and provides more precision and more global sight of scene. Moreover, this paper also addresses the real-time constraints within real-time vision system and real-time data exchange due to different processing units for robot and vision system. Here, a pattern design and simplified image processing are considered. This paper considers a 3D visual servoing within dynamic look and move scheme based on object pose. The system performance is validated by the experimental result.

With the growing interest of integrating robotics into everyday life and industry, the requirements towards the quality and quantity of applications grows equally hard. This trend is profoundly recognized in applications involving visual perception. Whereas visual sensing in home environments tend to be mainly used for recognition and localization, safety becomes the driving factor for developing intelligent visual control algorithms. More specifically, a robot operating in a human environment should not collide with obstacles and executed motion should be as smooth as possible. Furthermore, as the environment is not known on beforehand, a high demand on the robustness of visual processing is a necessity. On the other hand, in an industrial setting, the environment is known on beforehand and safety is mainly guaranteed by excluding a human operator. Despite these reasons, and the fact that visual servoing has gained much attention from industry to become a standard solution for robotic automation tasks, applications are highly simplified. For example, methods such as visual fault detection are already a mature technique in industrial manufacturing, where a fixed camera observes a product (e.g., on a conveyor belt) and checks whether it meets certain requirements. These operations can be executed at a fairly high rate due to the simplicity of the system (e.g., static camera) and the simplification of the processing task (e.g., binary images). For both areas the identified difficulties are similar. Foremost, this is the slow nature of (robust) visual processing, in respect to the ever growing demand of increasing speed and reducing delay. These two application areas with analogous limitations motivate the design of more direct approaches of vision in visual control systems. Therefore, in order to meet the requirements for next generation visual control systems, this thesis presents approaches which employ visual measurements as a direct feedback to design constrained motion. First, for industrial robotics, in order to obtain the required positioning accuracy, the measurement and fixation system have to be highly rigid and welldesigned, implying high cost and long design time. By measuring the position of objects directly with a camera, instead of indirectly by motor encoders, the requirements of the measurement and fixation system are less demanding. Moreover, this motivates the miniaturization of the complete control system. The approach is validated in experiments on a simplified 2D planar stage (i.e., considerable friction, poor fixation), which attains similar performance compared to encoder-based positioning systems. Secondly, in a human-centred environment, this direct sensing can improve traditional visual control systems, when subject to certain disturbances. More specifically, a method is proposed that uses an image-based feedforward controller on top of traditional position-based visual servo control to overcome disturbances such as friction or poorly designed local motor controllers. This visual feedforward control action is only active when an image-based error is present and vanishes when that error goes to zero. The method is validated on an anthropomorphic robotic manipulator with 7 degrees of freedom, intended for operation in the human care environment. Third, sensing the product directly gives rise to designing motion directly. Whereas in traditional approaches the motion trajectory is designed offline and can not be changed at runtime, direct trajectory generation computes the motion of the next step based on current state and events. This means that at any instance in time, the trajectory of a motion system can be altered with respect to certain desired kinematic or dynamic constraints. For industrial applications this makes manufacturing on near-repetitive or non-rigid structures (e.g. flexible displays) possible. When applied to a robotic manipulator, this enables obstacle avoidance to no longer be on path planning level, but on trajectory planning level, where kinematic or dynamic constraints can be taken into account. This results in a motion that is smoother than when obstacle avoidance with path planning is employed. For both application areas this direct trajectory generation method is implemented and shows high flexibility in constrained motion trajectory design.

Development of a novel integrated automated assembly system for large volume components in outdoor environment

Vision is in fact the richest source of information for ourself and also for outdoors Robotics, and can be considered the most complex and challenging problem in signal processing for pattern recognition. The first results using Vision in the control loop have been obtained in indoors and structured environments, in which a line or known patterns are detected and followed by a robot (Feddema & Mitchell (1989), Masutani et al. (1994)). Successful works have demonstrated that visual information can be used in tasks such as servoing and guiding, in robot manipulators and mobile robots (Conticelli et al. (1999), Mariottini et al. (2007), Kragic & Christensen (2002).) Visual Servoing is an open issue with a long way for researching and for obtaining increasingly better and more relevant results in Robotics. It combines image processing and control techniques, in such a way that the visual information is used within the control loop. The bottleneck of Visual Servoing can be considered the fact of obtaining robust and on-line visual interpretation of the environment, which can be usefully treated by control structures and algorithms. The solutions provided in Visual Servoing are typically divided into Image Based Control Techniques and Pose Based Control Techniques, depending on the kind of information provided by the vision system that determine the kind of references that have to be sent to the control structure (Hutchinson et al. (1996), Chaumette & Hutchinson (2006) and Siciliano & Khatib (2008)). Another classical division of the Visual Servoing algorithms considers the physical disposition of the visual system, yielding to eye-in-hand systems and eye-to-hand systems, that in the case of Unmanned Aerial Vehicles (UAV) can be translated as on-board visual systems (Mejias (2006)) and ground visual systems (Martinez et al. (2009)). The challenge of Visual Servoing is to be useful in outdoors and non-structured environments. For this purpose the image processing algorithms have to provide visual information that has to be robust and works in real time. UAV can therefore be considered as a challenging testbed for visual servoing, that combines the difficulties of abrupt changes in the image sequence (i.e. vibrations), outdoors operation (non-structured environments) and 3D information changes (Mejias et al. (2006)). In this chapter we give special relevance to the fact of obtaining robust visual information for the visual servoing task. In section (2).we overview the main algorithms used for visual tracking and we discuss their robustness when they are applied to image sequences taken from the UAV. In sections (3). and (4). we analyze how vision systems can perform 3D pose estimation that can be used for

Robotics has been a paradigm for science in the last few decades. At first, scientists’ efforts were devoted to the solution of the problem of planning and control of themotion of robot manipulators. However, the motion control of robot manipulators in unstructured environments is today an attractive scientific problem. An interesting solution for motion control is the use of sensor information, such as computer vision, in the system’s feedback. Several works and tools have been developed in recent years in this field (Corke; 2005), (Chaumette and Hutchinson; 2006). The more typical approaches consider visual perception for servoing and for the so called look and move (Hutchinson et al.; 1996). Visual servoing (Kelly et al.; 2000) can be classified into two approaches: camera-in-hand or camera-to-hand (Flandin et al.; 2000). In camera-to-hand robotic systems, multiple cameras or a single camera fixed in the worldcoordinate frame capture images of both, the robot and its environment. The tracking of the object with visual feedback can be made in 2D or 3D. An interesting solution for the visual servoing of camera-in hand robot manipulators in 2D can be found in e. g. (Bonfe et al.; 2002) and (Hernandez et al.; 2008) where stability demonstration of a decoupled controller have been presented. For 3D tracking some solutions reported are look and move controller, with one camera (Sim et al.; 2002) or more than one camera (Xie et al.; 2005). A 3D visual servoing with stability analysis in continuous time is presented by (Hernandez et al.; 2008a) and (Kelly et al.; 2006) present a direct visual servoing with transpose Jacobian control technique for regulation of robot manipulators in the 3D Cartesian space. In a similar vein (Enescu et al.; 2006) present mobile robot navigation for person tracking using a stereo head-camera. In this chapter, we consider the control problem of camera-in-hand robot manipulators in 2D and 3D. In both cases only one camera is mounted on the robot’s arm, which supplies visual information of the environment, with the aim of moving the manipulator by maintaining the image of the tracked object (a sphere) in the centre of the image plane, despite the possible movements of the object. In the 3D control the constant radius is used as a feature too. In this work, the proposed control system considers two loops in cascade, an internal loop solving the robots’ joint control, and an external loop implementing a dynamic look and move visual controller. A stability analysis in discrete time is developed under the conditions that it is possible to approximate the dynamic effect of the internal loop as an external loop time delay (Corke; 1996) and (Bonfe et al.; 2002). The more classic presentations of servovisual control are velocity controllers, based in the term feature Jacobian,(Chaumette and Hutchinson; 2006) 2

The automatic assembly of photo products has been developed by Nikon Corp. over more than ten years. Automatic assembly methods are historically classified into three categories: dedicated machines, semi-flexible automatic assembly systems, and universal automatic assembly systems. This paper surveys these types of automatic assembly systems. First, automatic assembly systems for mechanical components of photo products are considered, and their background and design policies discussed. Next, the performances of these assembly systems and the results achieved in practise are described. Finally, the future trends of photo product automation are outlined.

This thesis deals with Visual Servoing and its strictly connected disciplines like projective geometry, image processing, robotics and non-linear control. More specifically the work addresses the problem to control a robotic manipulator through one of the largely used Visual Servoing techniques: the Image Based Visual Servoing (IBVS). In Image Based Visual Servoing the robot is driven by on-line performing a feedback control loop that is closed directly in the 2D space of the camera sensor. The work considers the case of a monocular system with the only camera mounted on the robot end effector (eye in hand configuration). Through IBVS the system can be positioned with respect to a 3D fixed target by minimizing the differences between its initial view and its goal view, corresponding respectively to the initial and the goal system configurations: the robot Cartesian Motion is thus generated only by means of visual informations. However, the execution of a positioning control task by IBVS is not straightforward because singularity problems may occur and local minima may be reached where the reached image is very close to the target one but the 3D positioning task is far from being fulfilled: this happens in particular for large camera displacements, when the the initial and the goal target views are noticeably different. To overcame singularity and local minima drawbacks, maintaining the good properties of IBVS robustness with respect to modeling and camera calibration errors, an opportune image path planning can be exploited. This work deals with the problem of generating opportune image plane trajectories for tracked points of the servoing control scheme (a trajectory is made of a path plus a time law). The generated image plane paths must be feasible i.e. they must be compliant with rigid body motion of the camera with respect to the object so as to avoid image jacobian singularities and local minima problems. In addition, the image planned trajectories must generate camera velocity screws which are smooth and within the allowed bounds of the robot. We will show that a scaled 3D motion planning algorithm can be devised in order to generate feasible image plane trajectories. Since the paths in the image are off-line generated it is also possible to tune the planning parameters so as to maintain the target inside the camera field of view even if, in some unfortunate cases, the feature target points would leave the camera images due to 3D robot motions. To test the validity of the proposed approach some both experiments and simulations results have been reported taking also into account the influence of noise in the path planning strategy. The experiments have been realized with a 6DOF anthropomorphic manipulator with a fire-wire camera installed on its end effector: the results demonstrate the good performances and the feasibility of the proposed approach.

This paper proposes a new controller for dynamic visual servoing of robotic manipulators. In this work the image is acquired from a CCD camera mounted on planar robotic manipulator end-effector, and processed in a standard PC based vision system in order to close the visual servoing loop with PD controllers. First, we propose an extension to dynamic visual servoing based on a pose estimation algorithm and then compare it with another Lyapunov stable controller. The proposed dynamic controller uses only information from the image and robot encoders. This approach is inspired on classical 3D visual servoing and the Jacobian transpose controller approach. An experimental setup developed for visual servoing experiments, including software and hardware is presented. Experimental results for a planar robotic manipulator are presented.

This paper aims to develop a visual servo control of a robotic manipulator for cherry tomato harvesting. In the robotic manipulator, an RGB-depth camera was mounted to the end effector to acquire the poses of the target cherry tomatoes in space. The eye-in-hand-based visual servo controller guides the end effector to implement eye–hand coordination to harvest the target cherry tomatoes, in which a hybrid visual servo control method (HVSC) with the fuzzy dynamic control parameters was proposed by combining position-based visual servo (PBVS) control and image-based visual servo (IBVS) control for the tradeoff of both performances. In addition, a novel cutting and clipping integrated mechanism was designed to pick the target cherry tomatoes. The proposed tomato-harvesting robotic manipulator with HVSC was validated and evaluated in a laboratory testbed based on harvesting implementation. The results show that the developed robotic manipulator using HVSC has an average harvesting time of 9.40 s/per and an average harvesting success rate of 96.25% in picking cherry tomatoes.

This work presents different proposals to deal with common problems in robot visual servoing based on the application of discontinuous control methods. The feasibility and effectiveness of the proposed approaches are substantiated by simulation results and real experiments using a 6R industrial manipulator. The main contributions are:\n\n- Geometric invariance using sliding mode control (Chapter 3): the defined higher-order invariance is used by the proposed approaches to tackle problems in visual servoing. Proofs of invariance condition are presented.\n\n- Fulfillment of constraints in visual servoing (Chapter 4): the proposal uses sliding mode methods to satisfy mechanical and visual constraints in visual servoing, while a secondary task is considered to properly track the target object. The main advantages of the proposed approach are: low computational cost, robustness and fully utilization of the allowed space for the constraints.\n\n- Robust auto tool change for industrial robots using visual servoing (Chapter 4): visual servoing and the proposed method for constraints fulfillment are applied to an automated solution for tool changing in industrial robots. The robustness of the proposed method is due to the control law of the visual servoing, which uses the information acquired by the vision system to close a feedback control loop. Furthermore, sliding mode control is simultaneously used in a prioritized level to satisfy the aforementioned constraints. Thus, the global control accurately places the tool in the warehouse, but satisfying the robot constraints.\n\n- Sliding mode controller for reference tracking (Chapter 5): an approach based on sliding mode control is proposed for reference tracking in robot visual servoing using industrial robot manipulators. The novelty of the proposal is the introduction of a sliding mode controller that uses a high-order discontinuous control signal, i.e., joint accelerations or joint jerks, in order to obtain a smoother behavior and ensure the robot system stability, which is demonstrated with a theoretical proof.\n\n- PWM and PFM for visual servoing in fully decoupled approaches (Chapter 6): discontinuous control based on pulse width and pulse frequency modulation is proposed for fully decoupled position based visual servoing approaches, in order to get the same convergence time for camera translation and rotation.\n\nMoreover, other results obtained in visual servoing applications are also described.

This paper introduces a visual servoing scheme for humanoid walking. Though most of the existing approaches follow a perception-decision-action scheme, we close the loop so that the control is robust to model error. Our approach is based on a new reactive pattern generator which modifies, at the control level, the footsteps, the center of mass and the center of pressure trajectories for the center of mass to track a reference velocity. And, in this paper, the reference velocity is directly given by a visual servoing control law. Since, the HRP-2 walk induces a sway motion that disturbs the regulation of the visual control law, we introduce a control law allowing convergence in the image space and taking into account this sway motion.

We use the theory of two dimensional discrete wavelet transforms to derive inversion formulas for the Radon transform of terahertz datasets. These inversion formulas with good localised properties are implemented for the reconstruction of terahertz imaging in the area of interest, with a significant reduction in the required measurements. As a form of optical coherent tomography, terahertz CT complements the current imaging techniques and offers a promising approach for achieving non-invasive inspection of solid materials, with potentially numerous applications in industrial manufacturing and biomedical engineering.

The analysis of dynamic deformation processes is an important task in civil and in mechanical engineering. One main emphasis is set on the derivation of the objects transmission function, represented by systems of ordinary or partial differential equations (“white box”-models). The parametric identification of the initial and boundary conditions and relevant material parameters is one of the new challenges in the application of least-square algorithms. Embedding the investigation of the deformation process in a combination of a theoretical and an experimental system analysis suitable methods are provided with adaptive KALMAN-filtering as central identification tool. In this paper the approach of a full system analysis is shown quantifying a dynamic “white-box”-model for the calculation of thermal deformations of bar-shaped machine elements. The task was motivied from mechanical engineering searching new methods for the precise prediction and computational compensation of thermal influences in the heating and cooling phases of machine tools (i.e. robot arms, etc.). The quantification of thermal deformations under variable dynamic loads requires the modelling of the non-stationary spatial temperature distribution inside the object. Based upon FOURIERS law of heat flow the high-grade non-linear temperature gradient is represented by a system of partial differential equations within the framework of a dynamic Finite Element topology. It is shown that adaptive KALMAN-filtering is suitable to quantify relevant disturbance influences and to identify thermal parameters (i.e. thermal diffusivity) with a deviation of only 0,2%. As result an identified (and verified) parametric model for the realistic prediction respectively simu-lation of dynamic temperature processes is presented. Classifying the thermal bend as the main deformation quantity of bar-shaped machine tools, the temperature model is extended to a temperature deformation model. In lab tests with an aluminium column control measurements show that the identified model can be used to predict the columns bend with a mean deviation smaller than 10 mgon. Consequently the deformation model is a precise predictor and suitable for realistic simulations of thermal deformations under variable dynamic loads. In future our activities will be primarily focussed on applications in industrial manufacturing. 1. Parametric identification of dynamic deformation processes The creation of a realistic parametric dynamic deformation model is directly associated with the task to combine systems of differential equations (theoretical physical model) with empirical measurements. Consequently basic principles of system theory can be adapted to the geodetic deformation analysis. The parametric identification of a dynamic “white box”-model 3rd IAG / 12th FIG Symposium, Baden, May 22-24, 2006

Quality control is extremely important for manufacturing compatible parts to supply products that meet production requirements. It provides to track and control the stages of the process and minimizes waste by supporting high levels of productivity. Most of the manufacturers prefer a video measurement system (VMS), which offers non-contact high accurate measurement devices, for evaluating machined parts and products. However, due to the advanced technology and low competition the cost of the non-contact measurement devices is high. Besides some facilities and some research laboratories couldn’t reach these high-cost devices. Today, with the help of evolving technology and open-source image processing libraries, it is possible to offer cost-effective and accurate non-contact measurement systems. This study aims to put forward a VMS to measure parts/products in two dimensions with swift and accurate results. The proposed system has an error below 1% and the linear regression coefficient (r2) was found over 0.95. It works in real-time and minimum frequency was found 10 Hz for repetitive measurements, real-time measurement applications. The proposed cost-effective device can be adapted into various quality control applications in industrial manufacturing