Color-histogram-based tracking at 2000 fps

In this study, we develop a high-speed color-histogram-based tracking system that can be applied to 512x511 pixel images at 2000~fps using the hardware implementation of an improved CamShift algorithm. In the improved algorithm, the size, position, and orientation of an object to be tracked can be extracted using only the hardware implementation of color conversion and the moment feature calculation of 16 binary images quantized by color bins on a high-speed vision platform. We demonstrate its effectiveness by presenting several target tracking results at 2000~fps when color-patterned objects move rapidly under complicated backgrounds.

The mechanism for motion detection in a fly's vision system, known as the Reichardt correlator, suffers from a main shortcoming as a velocity estimator: low accuracy. To enable accurate velocity estimation, responses of the Reichardt correlator to image sequences are analyzed in this paper. An elaborated model with additional preprocessing modules is proposed. The relative error of velocity estimation is significantly reduced by establishing a real-time response-velocity lookup table based on the power spectrum analysis of the input signal. By exploiting the improved velocity estimation accuracy and the simple structure of the Reichardt correlator, a high-speed vision system of 1 kHz is designed and applied for robot yaw-angle control in real-time experiments. The experimental results demonstrate the potential and feasibility of applying insect-inspired motion detection to robot control.

This paper introduced a high-speed vision platform, JDP Express, which can record image sequences of 512×512 pixel at 2000fps as well as real-time image processing. IDP Express was also applied to a high-speed active vision system for target tracking to capture image sequences at high resolution, which were magnified by tracking a target in an image. Several experiments were actually executed for automatic high-frame-rate video recording by using image-based trigger, and the videos captured at a high frame rate were also shown to verify its effectiveness.

We developed a super-low-latency telemanipulation system using a high-speed vision system and a high-speed robot hand with a high-speed tactile sensor. This system does not require users to wear sensors. Also, since it has latency lower than the sampling rate of visual recognition by humans, the latency is not recognizable by humans. Low-latency telemanipulation systems are needed to perform tasks that require high speed, such as catching falling objects or fast-moving objects. We evaluated the latency of our telemanipulation system and successfully demonstrated catching of falling objects using our system, in contrast to a conventional vision system operating at 30 fps, which failed the task.

In recent years, high-speed vision systems are increasingly used for robot control. However, robot systems developed to date have not sufficiently utilized rapid responses of high-speed vision due to poor performance of control methods. In order to realize high-speed robot operation, it is necessary to design a control method to use the maximum torque generated by the robot. In this paper, we propose a control method integrating high speed visual processing and real time optimum control. This control method is based on nonlinear model predictive control (NMPC), and optimizes the response of the position of the robot arm so as to accurately follow the target trajectory. In this paper, we describe system design to integrate high speed visual processing and NMPC, taking high-speed operation of air hockey robot as an example. Finally, we present experimental data and verify the effectiveness of the system.

In order to achieve faster and more dexterous manipulations, we propose a strategy called holding. In the dynamic holding condition, the object is held in a stable condition while moving at high-speed. In this paper, we first explain the concept of dynamic holding, then we describe an experiment of dribbling by a robot as an example of dynamic holding using a high-speed multifingered hand and a high-speed vision system.

In recent years, the need for computer vision systems is increasing in various fields, such as security and visual inspection. It is crucial there to realize simple and high-speed practical vision systems. The present paper addresses the author's theoretical research and its applications developed thus far in working toward this goal. First, the problem of the conventional approach is pointed out, and the general framework of pattern recognition, in particular the feature extraction theory, is referred to as the theoretical foundation. Next, a scheme of adaptive vision system with learning capability is presented, which comprises two stages of feature extraction, namely, Higher-order Local Auto-Correlation and multivariate data analysis. Several applications are demonstrated, showing the flexible and effective performance of the proposed scheme.

There is a need for compact, high-speed, and low-power vision systems for enabling real-time mobile autonomous applications. The best approach to achieve this is to implement the bulk of the application in hardware. Reconfigurable hardware meet these requirements without the limitation of fixed functionality that accompanies application-specific circuits. Resource constraints of reconfigurable hardware calls for optimized implementations i terms of resource usage with maintained performance. The research group in Robotics at Malardalen University is moving toward the completion of a reconfigurable hardware-platform for stereo vision, coupled with a compact embedded computer. This system will incorporate hardware-based preprocessing components enabling visual perception for autonomous machines. This thesis covers the reconfigurable hardware section of the vision system concerning the realization of scene depth extraction. It shows the advantages of image preprocessing in hardware and propose a resource optimized approach to stereo matching. The work quantifies the impact of reduced resource utilization and a desire for increased accuracy in disparity estimation. The implemented stereo matching approach performs on par with recent similar implementations in terms of accuracy, but excels in terms of resource utilization and resource sharing, as the external memory requirement is removed for larger images. Future work aims to further include processes for navigation, and structure and object recognition. Furthermore, the system will be adapted to real world scenarios, both indoors and outdoors.

In this study, we develop a real-time high-speed color vision system to track multi-colored pie-shaped markers in 512 × 512 images at 2000 fps. Our color vision system has six cell-based labeling circuits that can calculate the 0th and 1st moment features of color-labeled regions during a single scan of a 512 × 512 image divided into 4096 8 × 8 cells. Our system examines the geometric distribution of five color regions in each pie-shaped marker using their 0th and 1st moment features, and it can simultaneously extract the unique marker codes, locations, and orientations of multiple pie-shaped markers in a 512 × 512 image at 2000 fps. To evaluate the performance of our system, we performed several experiments to track multi-colored pie-shaped markers printed on high-speed rotating objects.

This paper deals with deflection sensing for robotic control via high speed vision system. Marker is attached to the end effector of the 3-DoF robot arm and the robot is moved to satisfy a line trajectory of the end effector. The motion of the marker attached to the end effector is recorded using the high speed vision system and actual position of the end effector is calculated via image processing of recorded high-frame rate images. We found maximum 1.8 mm position error from the image processing results. We verify that deflection and vibration can be measured by comparing the actual position obtained from image and the ideal position of trajectory planning.

It is difficult to address the problem of occlusion and loss of target when a single image tracking algorithm is utilized in high reliability occasions, such as prison, coal mines and military area. This paper presents a novel tracking and monitoring system based on wireless location and video. A coordinate frame of wireless system of scene constructed by binding the targets with RSSI labels is used to extract the real-time location information of targets. A high-speed ball camera is used to set up the coordinate frame based on vision system. Double shaft angle encoder added on the high-speed ball camera and control system devised based on ARM are adopted to obtain real-time coordinates of field of visual system. Thus, it solves the problems of the high-speed ball camera without real-time angle feedback, and improves the reliability of the system. It makes the position of camera as the origin of coordinates, and fuses the two coordinates to find out the relationship with Pan & Tilt and the label position. Firstly, the system detects and captures targets by the help of label. Secondly, using the image processing algorithm tracks the targets on a larger scale. Finally, it uses the label information as a feedback to correct position when the targets lost or be covered. Results show that, the new system is robust in the large-scale scene and complicated background.

Reaching out for object is often described as consisting of two components that are based on different visual information. Information about the object's position and orientation guides the hand to the object, while information about the object's shape and size determines how the fingers move relative to the thumb to grasp it. We propose an alternative description, which consists of determining suitable positions on the object - on the basis of its shape, surface roughness, and so on - and then moving one's thumb and fingers more or less independently to these positions. We modeled this description using a minimum-jerk approach, whereby the finger and thumb approach their respective target positions approximately orthogonally to the surface. Our model predicts how experimental variable such as object size, movement speed, fragility, and required accuracy will influence the timing and size of the maximum aperture of the hand. An extensive review of experimental studies on grasping showed that the predicted influences correspond to human behavior.

Experimental analysis on relationship between roughness parameters and texture features of 6082T6 sandblasted components

In this paper, we propose a stereo vision system for observing continuously a quickly moving object. The system consists of two high-speed gaze controllers by rotational mirrors, and capture stereo images while gaze controllers track the object. Temporally continuous 3D shapes of the object are reconstructed from the stereo images, and the system measures high-speed motion and deformation of the object. We discuss settings of the system considering characteristics of the gaze controller by rotational mirrors. The experimental results show that the system can capture stereo images tracking a quickly and irregularly moving object, and can reconstruct temporally continuous high-resolution 3D shapes of the object.

Flies achieve supreme flight maneuverability through a small set of miniscule steering muscles attached to the wing base. The fast flight maneuvers arise from precisely timed activation of the steering muscles and the resulting subtle modulation of the wing stroke. In addition, slower modulation of wing kinematics arises from changes in the activity of indirect flight muscles in the thorax. We investigated if these modulations can be described as a superposition of a limited number of elementary deformations of the wing stroke that are under independent physiological control. Using a high-speed computer vision system, we recorded the wing motion of tethered flying fruit flies for up to 12 000 consecutive wing strokes at a sampling rate of 6250 Hz. We then decomposed the joint motion pattern of both wings into components that had the minimal mutual information (a measure of statistical dependence). In 100 flight segments measured from 10 individual flies, we identified 7 distinct types of frequently occurring least-dependent components, each defining a kinematic pattern (a specific deformation of the wing stroke and the sequence of its activation from cycle to cycle). Two of these stroke deformations can be associated with the control of yaw torque and total flight force, respectively. A third deformation involves a change in the downstroke-to-upstroke duration ratio, which is expected to alter the pitch torque. A fourth kinematic pattern consists in the alteration of stroke amplitude with a period of 2 wingbeat cycles, extending for dozens of cycles. Our analysis indicates that these four elementary kinematic patterns can be activated mutually independently, and occur both in isolation and in linear superposition. The results strengthen the available evidence for independent control of yaw torque, pitch torque, and total flight force. Our computational method facilitates systematic identification of novel patterns in large kinematic datasets.