1A1-C12 高フレームレート動画記録機能を有する高速アクティブビジョン

The first quarter of 2007 started as an active season for the real-time image processing community. The annual SPIE Conference on Real-Time Image Processing, which was held in San Jose, CA, highlighted recent developments in this growing branch of image processing. As one of the few regular events with its focus matching that of JRTIP, one witnessed the growth in applications of real-time image and video processing. The conference served as an effective instrument to foster the dissemination of recent results and to leverage the cohesion of the real time image processing community. The preparations for the next SPIE Conference on Real-Time Image Processing in 2008 are already progressing and its call-for-papers is included at the end of this issue. In 2007, JRTIP is going to seek involvement of more scientists in the field of real-time image and video processing. After 1 year of operation, the editors-in-chief have started to recruit additional associate editors to enlarge the expertise base of the editorial board, to solicit manuscript submissions from potential authors, and to make the reviewing process more efficient. Interested colleagues should contact the editors-in-chief expressing their Associate Editor candidature in support of one or more fields within the focus of JRTIP. In addition, they wish to encourage researchers to act as guest editors for special issues on any real-time image or video processing related topics of interest to them. It is possible to schedule two special issues in 2007, provided that a draft call-for-papers is sent to the editors-in-chief as soon as possible. The third issue of JRTIP presents four articles, two are survey papers and two are original research papers. These articles address different real-time image processing perspectives ranging from algorithmic to hardware issues. The first survey paper deals with the real-time filtering in 2D and 3D using ordered statistics that are supported by a rich set of experimental results derived from several algorithmic variations. Performance measurements are used to demonstrate the effect of noise suppression and preservation of fine image structures and to show the DSP implementation of denoising filters. The second survey paper introduces a non-invasive analysis of dynamic processes in living cells using microscopic computer vision techniques for understanding the interactions between cellular and molecular functional units and for providing novel insights into the cell biology. The third article is an original research paper covering a machine vision application involving vision-based position sensing for aerial refueling of unmanned aerial vehicles in real-time. A full image processing chain is investigated and tested in a MATLAB/Simulink soft real-time environment as well as in a Linux/RTAI hard real-time environment. The feasibility of the real-time implementation is shown using off-the-shelf commercial hardware. The fourth article is another original research paper describing a hardware architecture for real-time fractal coding for image compression. The introduced parallel hardware architecture contains units for domain M. F. Carlsohn (&) Ingenieurberatung fur Computer Vision und Bildkommunikation, Am Heiddamm 36g, 28355 Bremen, Germany e-mail: matthias.carlsohn@t-online.de

A high-speed vision system can be applied to color-histogram-based tracking at 2000 fps by hardware-implementing an improved CamShift algorithm. In the improved CamShift algorithm, the size, position, and orientation of a color-patterned object to be tracked in an image can be simultaneously extracted using only the hardware implementation of a color-histogram circuit module for calculating moment features of binary images quantized by 16 hue-based color bins. By hardware-implementing color-histogram circuit modules on a high-speed vision platform, IDP Express, the improved CamShift algorithm enables color-histogram-based tracking at 2000 fps for 512×511 pixel images in real-time. By installing our tracking system on a two-axis mechanical active vision system, we demonstrate the effectiveness of 2000 fps color-histogram-based tracking by performing several experiments of color-patterned objects, which are always tracked in the camera views even when they move rapidly under complicated backgrounds.

In this paper we introduce a new real-time parallel image processing system based on an IEEE1394-based PC-cluster. PC-clusters are becoming standard tools in the field of computer vision, especially for real-time multi-view image processing. To reduce the cost of system construction, we have employed IEEE1394 bus as a base network of the PC-cluster. IEEE1394 bus has the throughput of up to 400Mbps, which is large enough to transfer uncompressed video data, and supports not only asynchronous data transfer but isochronous one, which is inevitable for real-time data transfer. In addition, since we have several products of IEEE1394-based digital cameras, we can easily integrate real-time image processing system including camera systems. This paper presents an overview of our real-time parallel image processing system on an IEEE1394based PC-cluster, referring to basic features of IEEE1394. We also show some experimental results to evaluate the performance of the PC-cluster, comparing it with the performance of our previous Myrinet-based PC-cluster. Then we show a prototypical application of the system, which is real-time volume reconstruction from multi-view images by visual cone intersection method.

For real time target detection, identification and tracking in high frame rates, large field of view images, a real-time image processing system is designed. A TMS320C6678 DSP runs as the chief arithmetic processor of this system and FPGA as the secondary controller. C6678 is compared with the same series C6414 in image compression algorithm test. Experimental results show that the new system has a more effective construct, and higher reliability, and can provide a platform for the new high-speed image processing.

Charge coupled device(CCD) is one kind of photoelectric solid-state imaging device with excellent performance. It has the features of large array and good image quality. Working under very complex driving timing, it usually has low frame frequency and can not meet the demand of high imaging frame rate. The method of equivalent high frame rate imaging technique is introduced based on normal CCD in this paper. Some CCDs can be drived via electrodes separately. To meet the imaging demands of high frame rate, high resolution and high quality of two frame imaging technique for pulse light images, the right kind of CCDs should be adopted. The most important is that the CCD electrodes must be controlled directly and respectively. The charge transfer period is a key point while designing CCD driving timing for separate pulse light integration, charge package transfer and readout for two pulse light images. The validation experiments for the principle of equivalent high frame rate exposure of two pulse light images has been accomplished. It is shown that two pulse light images with interval of about 2 μs can be distinguished while all CCD’s inherent excellent performance are retained. A kind of two framing camera system with simple structure can be developed based on this principle.

The mechanism for motion detection in a fly's vision system, known as the Reichardt correlator, suffers from a main shortcoming as a velocity estimator: low accuracy. To enable accurate velocity estimation, responses of the Reichardt correlator to image sequences are analyzed in this paper. An elaborated model with additional preprocessing modules is proposed. The relative error of velocity estimation is significantly reduced by establishing a real-time response-velocity lookup table based on the power spectrum analysis of the input signal. By exploiting the improved velocity estimation accuracy and the simple structure of the Reichardt correlator, a high-speed vision system of 1 kHz is designed and applied for robot yaw-angle control in real-time experiments. The experimental results demonstrate the potential and feasibility of applying insect-inspired motion detection to robot control.

The unmanned aerial vehicle (UAV)-borne video synthetic aperture radar (SAR) possesses the characteristic of having high-continuous-frame-rate imaging, which is conducive to the real-time monitoring of ground-moving targets. The real-time imaging-processing system for UAV-borne video SAR (ViSAR) requires miniaturization, low power consumption, high frame rate, and high-resolution imaging. In order to achieve high-frame-rate real-time imaging on limited payload-carrying platforms, this study proposes a miniaturization design of a high-integration UAV-borne ViSAR real-time imaging-processing component (MRIPC). The proposed design integrates functions such as broadband signal generation, high-speed real-time sampling, and real-time SAR imaging processing on a single-chip FPGA. The parallel access mechanism using multiple sets of high-speed data buffers increases the data access throughput and solves the problem of data access bandwidth. The range-Doppler (RD) algorithm and map-drift (MD) algorithm are optimized using parallel multiplexing, achieving a balance between computing speed and hardware resources. The test results have verified that our proposed component is effective for the real-time processing of 2048 × 2048 single-precision floating-point data points to realize a 5 Hz imaging frame rate and 0.15 m imaging resolution, satisfying the requirements of real-time ViSAR-imaging processing.

The digital micromirror device DMD is widely used in visible light projection, special-purpose spatial light modulation, and infrared scene simulation, due to its high resolution, uniformity and energy concentration. In some applications that require high frame rate scene image display, it is necessary to ensure that the DMD displays high gray level images at a high frame rate. However, the display frame rate of the pulse width modulation (PWM) method is limited by the minimum time required for DMD loading data, unable to achieve high frame rate display. Although the DMD binary display mode can meet the requirements of high frame rate display, it cannot meet the requirements of high gray level. The paper proposes an image display method that uses light source and DMD to synchronize modulation. Decompose the high-order gray image into bit planes according to the gray threshold, and the DMD displays each bit plane in binary mode under the trigger of an sync pulse. The intensity of the illumination laser source is modulated by an acousto-optic modulator to match the bit plane and the radiated laser power. This method makes use of the function of high frame rate display in DMD binary mode cooperate with light intensity modulation of illumination laser source, and realizes high frame rate and high dynamic range image display. With this method, the maximum frame rate of 8-bit gray level image with 1920 ×1080 resolution can reach 2KHz. The experimental system has realized 200Hz frame rate display of 8-bit gray level image with 1920 ×1080 resolution.

In accordance with advances in camera technology, the requirement of high-quality video has greatly increased. Among the factors required for high-quality video are a high-resolution and a high frame rate. However, the limitation of the pixel transfer rate restricts compatibility of a high-resolution and a high frame rate in commercial camera. We propose a dual-sensor camera that consists of two cameras: one with a high-resolution and a low frame rate and the other with a low-resolution and a high frame rate. The system is capable of capturing two different image sequences: high-resolution images and high-frame-rate images. A sensor calibration for the dual-sensor camera is also proposed.

In recent years, high-speed vision systems are increasingly used for robot control. However, robot systems developed to date have not sufficiently utilized rapid responses of high-speed vision due to poor performance of control methods. In order to realize high-speed robot operation, it is necessary to design a control method to use the maximum torque generated by the robot. In this paper, we propose a control method integrating high speed visual processing and real time optimum control. This control method is based on nonlinear model predictive control (NMPC), and optimizes the response of the position of the robot arm so as to accurately follow the target trajectory. In this paper, we describe system design to integrate high speed visual processing and NMPC, taking high-speed operation of air hockey robot as an example. Finally, we present experimental data and verify the effectiveness of the system.

We developed a super-low-latency telemanipulation system using a high-speed vision system and a high-speed robot hand with a high-speed tactile sensor. This system does not require users to wear sensors. Also, since it has latency lower than the sampling rate of visual recognition by humans, the latency is not recognizable by humans. Low-latency telemanipulation systems are needed to perform tasks that require high speed, such as catching falling objects or fast-moving objects. We evaluated the latency of our telemanipulation system and successfully demonstrated catching of falling objects using our system, in contrast to a conventional vision system operating at 30 fps, which failed the task.

The purpose of this study was to assess whether or not high frame rate (HFR) videos recorded using high-speed digital imaging (HSDI) improve the visual recognition of the motions of the laryngopharyngeal structures during pharyngeal swallow in fiberoptic endoscopic evaluation of swallowing (FEES). Five healthy subjects were asked to swallow 0.5ml water under fiberoptic nasolaryngoscopy. The endoscope was connected to a high-speed camera, which recorded the laryngopharyngeal view throughout the swallowing process at 4000 frames/s (fps). Each HFR video was then copied and downsampled into a standard frame rate (SFR) video version (30 fps). Fifteen otorhinolaryngologists observed all of the HFR/SFR videos in random order and rated the four-point ordinal scale reflecting the degree of visual recognition of the rapid laryngopharyngeal structure motions just before the 'white-out' phenomenon. Significantly higher scores, reflecting better visibility, were seen for the HFR videos compared with the SFR videos for the following laryngopharyngeal structures: the posterior pharyngeal wall (p=0.001), left pharyngeal wall (p=0.015), right lateral pharyngeal wall (p=0.035), tongue base (p=0.005), and epiglottis tilting (p=0.005). However, when visualized with HFR and SFR, 'certainly clear observation' of the laryngeal structures was achieved in <50% of cases, because all the motions were not necessarily captured in each video. These results demonstrate the use of HSDI in FEES makes the motion perception of the laryngopharyngeal structures during pharyngeal swallow easier in comparison to SFR videos with equivalent image quality due to the ability of HSDI to depict the laryngopharyngeal motions in a continuous manner.

A new high-speed driver has been developed for the magneto-optic spatial light modulator (SLM). The driver allows the output of an image sensing device such as a CCD array to drive the SLM. Using digital circuitry, high frame rates are made possible. Furthermore, the digital nature of the interface allows real-time digital image processing to be performed on input imagery; the input image can be processed by a 3 X 3 kernel. For example, functions such as noise filtering, edge detection, or thresholding can be performed in real-time. Adaptive filters or multiple filter operations on the same data are also possible. Processed imagery can be displayed on the SLM at over 400 frames/second. In addition, the interface can be applied to other electrically-addressed SLMs. The potential for higher frame rates is possible with faster electronic components.

In real-time image processing, the demand for efficient solutions has surged with the proliferation of applications spanning from autonomous vehicles to medical diagnostics. This study addresses the imperative need for accelerated machine learning algorithms to enhance the processing speed of image-related tasks. The research focuses on leveraging Field-Programmable Gate Arrays (FPGAs) to implement hardware acceleration, exploiting their parallel computing capabilities. The advent of machine learning in image processing has revolutionized various industries, yet real-time applications encounter computational bottlenecks. This research delves into hardware acceleration using FPGAs to overcome these constraints, offering a novel approach to expedite machine learning algorithms. Traditional software implementations of machine learning algorithms often fall short in meeting real-time processing requirements. This research aims to bridge this gap by exploring FPGA-based hardware acceleration, addressing the performance limitations hindering the seamless integration of machine learning into real-time image processing systems. While existing literature acknowledges the potential of FPGA-based acceleration, a comprehensive exploration of its application for real-time image processing is lacking. This research fills the void by presenting a detailed method and empirical results, contributing to the limited body of knowledge on FPGA-accelerated machine learning in the of image processing. The study employs a systematic approach, integrating machine learning algorithms onto FPGAs through hardware description languages. The implementation is optimized to exploit parallelism inherent in FPGAs, resulting in a tailored hardware solution for real-time image processing. Comparative analyses against software implementations provide insights into the performance gains achieved. The experimental results demonstrate a significant enhancement in processing speed, validating the efficacy of FPGA-based hardware acceleration for machine learning algorithms in real-time image processing applications.

In recent years, there has been an increasing growth of using vision-based systems for tracking the players in team sports to evaluate and enhance their performance. Player tracking using vision systems is a very challenging task due to the nature of sports games, which includes severe and frequent interactions (e.g. occlusions) between the players. Additionally, these vision systems have high computational demands since they require processing of a huge amount of video data based on the utilization of multiple cameras with high resolution and high frame rate. As a result, most of the existing systems based on general-purpose computers are not able to perform online real-time player tracking, but track the players offline using pre-recorded video files, limiting, e.g., direct feedback on the player performance during the game. In this paper, we present a reconfigurable system to track the players in indoor sports automatically and without user interaction. The proposed system performs real-time processing of the incoming video streams from the cameras, achieving online player tracking. The teams are identified, and the players’ positions are detected based on the colors of their jerseys. FPGA technology is used to handle the compute-intensive vision processing tasks by implementing the video acquisition, video preprocessing, player segmentation, and team identification & player detection modules in hardware, realizing an online real-time system. While the pixel processing is performed in the FPGA, the less compute-intensive player tracking is performed on a general purpose computer. The maximum achieved frame rate for the FPGA implementation is 96.7 fps using a mature Xilinx Virtex-4 FPGA, and can be increased to 136.4 fps using a Xilinx Virtex-7 device. The Player tracking requires an average time of 2.5 ms per frame in the host-PC. As a result, the proposed reconfigurable system supports a maximum frame rate of 78.9 fps using two cameras with a resolution of 1392 × 1040 pixels each. Our results show that the achieved average precision and recall for player detection are up to 84.02% and 96.6%, respectively. Including player tracking, the achieved average precision and recall are up to 94.85% and 94.72%, respectively. Using the proposed FPGA implementation, a speedup by a factor of 15.2 is achieved compared to an OpenCV-based software implementation on a PC equipped with a 2.93 GHz Intel i7-870 CPU.