Insect-inspired high-speed motion vision system for robot control

Although a great deal of experimental evidence supports the notion of a Reichardt correlator as a mechanism for biological motion detection, the correlator does not signal true image velocity. This study examines the accuracy with which realistic Reichardt correlators can provide velocity estimates in an organism's natural visual environment. The predictable statistics of natural images imply a consistent correspondence between mean correlator response and velocity, allowing the otherwise ambiguous Reichardt correlator to act as a practical velocity estimator. Analysis and simulations suggest that processes commonly found in visual systems, such as prefiltering, response compression, integration, and adaptation, improve the reliability of velocity estimation and expand the range of velocities coded. Experimental recordings confirm our predictions of correlator response to broadband images.

A high-speed vision system can be applied to color-histogram-based tracking at 2000 fps by hardware-implementing an improved CamShift algorithm. In the improved CamShift algorithm, the size, position, and orientation of a color-patterned object to be tracked in an image can be simultaneously extracted using only the hardware implementation of a color-histogram circuit module for calculating moment features of binary images quantized by 16 hue-based color bins. By hardware-implementing color-histogram circuit modules on a high-speed vision platform, IDP Express, the improved CamShift algorithm enables color-histogram-based tracking at 2000 fps for 512×511 pixel images in real-time. By installing our tracking system on a two-axis mechanical active vision system, we demonstrate the effectiveness of 2000 fps color-histogram-based tracking by performing several experiments of color-patterned objects, which are always tracked in the camera views even when they move rapidly under complicated backgrounds.

We developed a super-low-latency telemanipulation system using a high-speed vision system and a high-speed robot hand with a high-speed tactile sensor. This system does not require users to wear sensors. Also, since it has latency lower than the sampling rate of visual recognition by humans, the latency is not recognizable by humans. Low-latency telemanipulation systems are needed to perform tasks that require high speed, such as catching falling objects or fast-moving objects. We evaluated the latency of our telemanipulation system and successfully demonstrated catching of falling objects using our system, in contrast to a conventional vision system operating at 30 fps, which failed the task.

In recent years, high-speed vision systems are increasingly used for robot control. However, robot systems developed to date have not sufficiently utilized rapid responses of high-speed vision due to poor performance of control methods. In order to realize high-speed robot operation, it is necessary to design a control method to use the maximum torque generated by the robot. In this paper, we propose a control method integrating high speed visual processing and real time optimum control. This control method is based on nonlinear model predictive control (NMPC), and optimizes the response of the position of the robot arm so as to accurately follow the target trajectory. In this paper, we describe system design to integrate high speed visual processing and NMPC, taking high-speed operation of air hockey robot as an example. Finally, we present experimental data and verify the effectiveness of the system.

In recent years, the need for computer vision systems is increasing in various fields, such as security and visual inspection. It is crucial there to realize simple and high-speed practical vision systems. The present paper addresses the author's theoretical research and its applications developed thus far in working toward this goal. First, the problem of the conventional approach is pointed out, and the general framework of pattern recognition, in particular the feature extraction theory, is referred to as the theoretical foundation. Next, a scheme of adaptive vision system with learning capability is presented, which comprises two stages of feature extraction, namely, Higher-order Local Auto-Correlation and multivariate data analysis. Several applications are demonstrated, showing the flexible and effective performance of the proposed scheme.

In order to achieve faster and more dexterous manipulations, we propose a strategy called holding. In the dynamic holding condition, the object is held in a stable condition while moving at high-speed. In this paper, we first explain the concept of dynamic holding, then we describe an experiment of dribbling by a robot as an example of dynamic holding using a high-speed multifingered hand and a high-speed vision system.

Angular Velocity(AV) estimation is necessary for controllers used for rotation motion monitoring. The velocity estimated is used for feedback and is very crucial in speed control for a variety of applications such as spacecraft tumbling detection, propeller speed control etc. Linear motion detection and linear velocity estimation has been exploited vastly. But, when it comes to angular velocity estimation of a randomly moving object with varying angular speeds, computation is very complex. Hence, computation of AV by non-contact based methods are in huge demand. The need for detection of the direction and value of rotation of a tumbling spacecraft in the Earth's orbit is a huge necessity. Accurate AV estimation has applications in spacecraft docking in space. In the proposed work we have developed an improved algorithm for real-time estimation of rotation using live video feed. The AV of the object can be measured by the camera feed of the rotating bodies as grabbed by the video camera. Angle is computed on every frame with respect to a reference and monitored for a certain number of frames to give the instantaneous AV. The number of frames processed by the camera is dynamic and is estimated prior to the angle calculation. Comparison of performance of AV estimation by the proposed algorithm with state-of-the-art Lucas Kanade(LK) object tracking method based AV estimation is also reported. The proposed approach shows significant improvement in estimating velocities whose motion is circular in nature. Additionally, this work advocates a novel approach towards an effective real-time angular velocity calculation by thresholding techniques, blob identification, and centroid detection.

There is a need for compact, high-speed, and low-power vision systems for enabling real-time mobile autonomous applications. The best approach to achieve this is to implement the bulk of the application in hardware. Reconfigurable hardware meet these requirements without the limitation of fixed functionality that accompanies application-specific circuits. Resource constraints of reconfigurable hardware calls for optimized implementations i terms of resource usage with maintained performance. The research group in Robotics at Malardalen University is moving toward the completion of a reconfigurable hardware-platform for stereo vision, coupled with a compact embedded computer. This system will incorporate hardware-based preprocessing components enabling visual perception for autonomous machines. This thesis covers the reconfigurable hardware section of the vision system concerning the realization of scene depth extraction. It shows the advantages of image preprocessing in hardware and propose a resource optimized approach to stereo matching. The work quantifies the impact of reduced resource utilization and a desire for increased accuracy in disparity estimation. The implemented stereo matching approach performs on par with recent similar implementations in terms of accuracy, but excels in terms of resource utilization and resource sharing, as the external memory requirement is removed for larger images. Future work aims to further include processes for navigation, and structure and object recognition. Furthermore, the system will be adapted to real world scenarios, both indoors and outdoors.

This paper introduced a high-speed vision platform, JDP Express, which can record image sequences of 512×512 pixel at 2000fps as well as real-time image processing. IDP Express was also applied to a high-speed active vision system for target tracking to capture image sequences at high resolution, which were magnified by tracking a target in an image. Several experiments were actually executed for automatic high-frame-rate video recording by using image-based trigger, and the videos captured at a high frame rate were also shown to verify its effectiveness.

Experimental analysis on relationship between roughness parameters and texture features of 6082T6 sandblasted components

In this paper, we propose a stereo vision system for observing continuously a quickly moving object. The system consists of two high-speed gaze controllers by rotational mirrors, and capture stereo images while gaze controllers track the object. Temporally continuous 3D shapes of the object are reconstructed from the stereo images, and the system measures high-speed motion and deformation of the object. We discuss settings of the system considering characteristics of the gaze controller by rotational mirrors. The experimental results show that the system can capture stereo images tracking a quickly and irregularly moving object, and can reconstruct temporally continuous high-resolution 3D shapes of the object.

In this study, we develop a real-time high-speed color vision system to track multi-colored pie-shaped markers in 512 × 512 images at 2000 fps. Our color vision system has six cell-based labeling circuits that can calculate the 0th and 1st moment features of color-labeled regions during a single scan of a 512 × 512 image divided into 4096 8 × 8 cells. Our system examines the geometric distribution of five color regions in each pie-shaped marker using their 0th and 1st moment features, and it can simultaneously extract the unique marker codes, locations, and orientations of multiple pie-shaped markers in a 512 × 512 image at 2000 fps. To evaluate the performance of our system, we performed several experiments to track multi-colored pie-shaped markers printed on high-speed rotating objects.

Flies achieve supreme flight maneuverability through a small set of miniscule steering muscles attached to the wing base. The fast flight maneuvers arise from precisely timed activation of the steering muscles and the resulting subtle modulation of the wing stroke. In addition, slower modulation of wing kinematics arises from changes in the activity of indirect flight muscles in the thorax. We investigated if these modulations can be described as a superposition of a limited number of elementary deformations of the wing stroke that are under independent physiological control. Using a high-speed computer vision system, we recorded the wing motion of tethered flying fruit flies for up to 12 000 consecutive wing strokes at a sampling rate of 6250 Hz. We then decomposed the joint motion pattern of both wings into components that had the minimal mutual information (a measure of statistical dependence). In 100 flight segments measured from 10 individual flies, we identified 7 distinct types of frequently occurring least-dependent components, each defining a kinematic pattern (a specific deformation of the wing stroke and the sequence of its activation from cycle to cycle). Two of these stroke deformations can be associated with the control of yaw torque and total flight force, respectively. A third deformation involves a change in the downstroke-to-upstroke duration ratio, which is expected to alter the pitch torque. A fourth kinematic pattern consists in the alteration of stroke amplitude with a period of 2 wingbeat cycles, extending for dozens of cycles. Our analysis indicates that these four elementary kinematic patterns can be activated mutually independently, and occur both in isolation and in linear superposition. The results strengthen the available evidence for independent control of yaw torque, pitch torque, and total flight force. Our computational method facilitates systematic identification of novel patterns in large kinematic datasets.

This paper deals with deflection sensing for robotic control via high speed vision system. Marker is attached to the end effector of the 3-DoF robot arm and the robot is moved to satisfy a line trajectory of the end effector. The motion of the marker attached to the end effector is recorded using the high speed vision system and actual position of the end effector is calculated via image processing of recorded high-frame rate images. We found maximum 1.8 mm position error from the image processing results. We verify that deflection and vibration can be measured by comparing the actual position obtained from image and the ideal position of trajectory planning.

This study proposes a real-time video shooting system using a view-expanded microscope. The proposed system composed of a microscope, a galvanometer mirror and a high-speed vision system is capable of obtaining a 16,000×2,000 pixel microscopic image in real time at 25 fps by combining multiple visual fields. The effectiveness of our system is demonstrated through video shooting experiments of moving sea urchin embryos in a microchannel.