Depth from a Motion Algorithm and a Hardware Architecture for Smart Cameras

This paper describes a resource-efficient vision system for the mini-robot “Khepera”. It is implemented to perform a fundamental approach of robotic navigation, which is obstacle detection by using an optical flow algorithm. This is inspired by visual perception of insects. The optical flow field is evaluated by implementing the sum of absolute differences (SAD) operation of block matching while the mobile mini-robot moves on texture plane. The SAD is realized in an additional processing module based on a Field Programmable Gate Array (FPGA) solution which can perform various tasks in parallel; for example, image processing, camera control and symbolic protocol transmission to the mini-robot. As a result, the Khepera equipped with 2D camera and the additional module is able to autonomously move avoiding collisions with obstacles, called ego-motion. The 30 × 16 SAD operation is processed by computing 16 pairs of pixels in each operation, which accomplishes the real-time approach. The proposed implementation reduces processing time and power consumption.

Two-step phase extraction and random phase shift estimation in phase-shifting profilometry based on least-squared optical flow method

Symptoms of depression and comorbid anxiety are known risk factors for cognitive impairment in major depressive disorder (MDD). Understanding their relationships is crucial for developing targeted interventions to mitigate cognitive impairments in MDD patients. We expect that the severity of sleep disturbances and other depressive symptoms will be positively correlated with the degree of cognitive impairments. We also hypothesize that anxiety symptoms, especially psychic anxiety, is a key factor in predicting cognitive performance in MDD patients and may indirectly contribute to cognitive impairment by affecting sleep disturbances and other potential factors. To determine which dimension of the depressive and anxiety symptoms predicts cognitive impairment during a depressive episode. A comprehensive neurocognitive test battery assessed executive function, attention, processing speed, and memory in 162 medication-free MDD patients and 142 matched healthy controls. The 24-item Hamilton Depression Rating Scale was used to assess depressive symptoms, and the 14-item Hamilton Anxiety Scale was used to assess anxiety symptoms. Linear regression analyses and mediation analyses were conducted to evaluate the impact of depressive and anxiety symptoms, as well as their interactions, on cognitive impairments. Among the depressive symptoms, sleep disturbances were associated with poorer executive function (P = 0.004), lower processing speed (P = 0.047), and memory impairments (P < 0.001), and psychomotor retardation (PR) was associated with lower processing speed in patients with MDD (P = 0.019). Notably, PR was found to mediate the impact of sleep disturbances on the processing speed. Regarding anxiety symptoms, psychic anxiety, rather than somatic anxiety, was associated with cognitive impairments in all aspects. Sleep disturbances mediated the effect of psychic anxiety on executive function [β = -0.013, BC CI (-0.027, -0.001)] and memory [β = -0.149, BC CI (-0.237, -0.063)], while PR mediated its effect on processing speed (β = -0.023, BC CI (-0.045, -0.004)]. Sleep disturbances may be a key predictor of poorer executive function, lower processing speed, and memory loss, while PR is crucial for lower processing speed during a depressive episode. Psychic anxiety contributes to all aspects of cognitive impairments, mediated by sleep disturbances and PR.

Hyperspectral image contains various wavelength channels and the corresponding imagery processing requires a computation platform with high performance. Target and anomaly detection on hyperspectral image has been concerned because of its practicality in many real-time detection fields while wider applicability is limited by the computing condition and low processing speed. The field programmable gate arrays (FPGAs) offer the possibility of on-board hyperspectral data processing with high speed, low-power consumption, reconfigurability and radiation tolerance. In this paper, we develop a novel FPGA-based technique for efficient real-time target detection algorithm in hyperspectral images. The collaborative representation is an efficient target detection (CRD) algorithm in hyperspectral imagery, which is directly based on the concept that the target pixels can be approximately represented by its spectral signatures, while the other cannot. To achieve high processing speed on FPGAs platform, the CRD algorithm reduces the dimensionality of hyperspectral image first. The Sherman–Morrison formula is utilized to calculate the matrix inversion to reduce the complexity of overall CRD algorithm. The achieved results demonstrate that the proposed system may obtains shorter processing time of the CRD algorithm than that on 3.40 GHz CPU.

This paper proposes a novel particle image velocimetry (PIV) technique to generate an instantaneous two-dimensional velocity field for sediment-laden fluid based on the optical flow algorithm of ultrasound imaging. In this paper, an ultrasonic PIV (UIV) system is constructed by integrating a medical ultrasound instrument and an ultrasonic particle image velocimetry algorithm. The medical ultrasound instrument with a phased sensor array is used to acquire acoustic echo signals of particles in water and generate two-dimensional underwater ultrasound images. Based on the optical flow field, the instantaneous velocity of the particles in water corresponding to the pixels in the ultrasonic particle images is derived from the grayscale change between adjacent images under the L-K local constraint based on the optical flow field, and finally, the two-dimensional flow field is obtained. Through multiple sets of experiments, the proposed algorithm is verified. The experimental results are compared with those of the conventional cross-correlation algorithms. The results show that the L-K optical flow method can not only obtain the underwater velocity field accurately, but also has the advantages of good smoothness and extensive suitability, especially for the flow field measurement in sediment-laden fluid than conventional algorithms.

Stereo vision finds a wide range of applications in automotive, object detection, robot navigation, agriculture mapping, and others. Stereo matching is a stereo algorithm targeted to identify the corresponding pixels from two or more images. This study shows the implementation of stereo matching using the Sum of Absolute Difference (SAD) algorithm to extract the object's depth or disparity from stereo images. Our key objective revolves around the implementation of the stereo matching algorithm for a small field-programmable gate array (FPGA)—requiring relatively few resources while maintaining the processing speed as well as disparity map. For meeting this requirement, we used small window buffers to compute the stereo matching. The occluded pixels were reduced by introducing secondary consistency checking implementation. From the results of experiments performed using the Zynq UltraScale+ ZCU102 FPGA board with SDSoC compiler, the processing speed for computing the stereo matching algorithm with a 4×4 window buffer was 0.038 s for an image size of 486×720 pixels and 0.051 s for 375×1242 pixels resolution. The proposed design needed 1% each of BRAM and FF and 7% of LUT. An 18% reduction in the pixel errors has been observed when employing the secondary consistency matching on the post-processing.

Abstract. Eastern China is one of the most economically developed and densely populated areas in the world. Due to its special geographical location and climate, eastern China is affected by different weather systems, such as monsoons, shear lines, typhoons, and extratropical cyclones. In the near future, the rainfall rate becomes difficult to predict precisely due to these systems. Traditional physics-based methods such as numerical weather prediction (NWP) tend to perform poorly on nowcasting problems due to the spin-up issue. Moreover, various meteorological stations are distributed in this region, generating a large amount of observation data every day, which have great potential for application to data-driven methods. Thus, it is important to train a data-driven model from scratch that is suitable for the specific weather situation of eastern China. However, due to the high degrees of freedom and nonlinearity of machine learning algorithms, it is difficult to add physical constraints. Therefore, with the intention of using various kinds of data as a proxy for physical constraints, we collected three kinds of data (radar, satellite, and precipitation data) in the flood season from 2017 to 2018 in this area and preprocessed them into tensors (256×256) that cover eastern China with a domain of 12.8×12.8∘. The developed multisource data model (MSDM) combines the optical flow, random forest, and convolutional neural network (CNN) algorithms. It treats the precipitation nowcasting task as an image-to-image problem, which takes radar and satellite data with an interval of 30 min as inputs and predicts radar echo intensity with a lead time of 30 min. To reduce the smoothing caused by convolutions, we use the optical flow algorithm to predict satellite data in the following 120 min. The predicted radar echoes from the MSDM together with satellite data from the optical flow algorithm are recursively implemented in the MSDM to achieve a 120 min lead time. The MSDM predictions are comparable to those of other baseline models with a high temporal resolution of 6 min. To solve blurry image problems, we applied a modified structural similarity (SSIM) index as a loss function. Furthermore, we use the random forest algorithm with predicted radar and satellite data to estimate the rainfall rate, and the results outperform those of the traditional, nonlinear radar reflectivity factor and rainfall rate (Z–R) relationships that use logarithmic functions. The experiments confirm that machine learning with multisource data provides more reasonable predictions and reveals a better nonlinear relationship between radar echo and precipitation rate. Apart from developing complicated machine learning algorithms, exploiting the potential of multisource data will yield more improvements.

As a requirement of many modern image compression standards faced today, a computational complexity is observed due to the best mode selection in the intra-prediction stage. This computational complexity is tried to be reduced by various techniques without affecting the performance criteria of the image. In this study, a fast search algorithm, which simplifies the mode selection process of the intra-prediction algorithm and provides calculation with less number of modes is proposed. The hardware architecture of this proposed algorithm is implemented for realization. There are two main sections of the intra-prediction algorithm in image compression, namely the image prediction process and the mode selection process. In this study, main objective is to reduce the process time of the mode selection and the simplification of the hardware design. Sum of absolute difference (SAD) is a frequently used criterion to simplify hardware design. The algorithm searches for the most suitable mode in a single step, where the decision is based on the SAD criterion preferred for the simplicity. The proposed algorithm and related hardware architecture is tested by using various experiments. The number of the modes calculated is reduced effectively, while the process is kept within the acceptable limits in terms of peak signal to noise ratio (PSNR) and compression rate (CR) performance criteria. Therefore, the number of clock cycles observed is considerably reduced. The designed architecture is synthesized for the field programmable gate arrays (FPGA) board and the obtained results are given. In addition, these results are compared with the HM reference software where the corresponding results are in accordance with the reference software.

Wavefront aberrations caused by turbulent or rapidly changing media can considerably degrade the performance of an imaging system. To dynamically compensate these wavefront distortions adaptive optics is applied. We developed an affordable adaptive optic system which combines CMOS sensor and Liquid Crystal on Silicon (LCOS) display technology with the Field Programmable Gate Arrays (FPGA) devices parallel computing capabilities. A high-speed, accurate wavefront sensor is an elemental part of an adaptive optic system. In the paper, an efficient FPGA implementation of the Sum of Absolute Differences (SAD) algorithm, which accomplishes the correlation-based wavefront sensing, is introduced. This architecture was implemented on a Spartan-3 FPGA which is capable of real-time (>500 fps) measuring the incoming wavefront.

This letter presents a new efficient architecture for high-speed advanced encryption standard (AES) encryptor. This technique is implemented using composite field arithmetic byte substitution, where higher efficiency is achieved by merging and location rearrangement of different operations required in the steps of encryption. The proposed architecture is presented with multistage subpipelined architecture that allows having higher efficiency in terms of (throughput/area) than any previous field-programmable gate array (FPGA) implementations.

An enhanced constrained one-bit transform (C1BT)-based fast motion estimation (ME) method is proposed. Binary transform-based ME identifies the proper motion vector by transforming the 8-bit pixels in the original image into a low bit depth of the bit plane, substantially reducing the computational complexity and facilitating hardware implementation by using the number of non-matching points (NNMPs) instead of the sum of absolute difference (SAD). However, a motion block size is N × N , therefore the dynamic range of NNMP (0 ≤ NNMP ≤ N × N ) is decreased to 1/256 compared to that of the SAD with 8-bit images (0 ≤ SAD ≤ 256 × N × N ). The higher the NNMP between the current bit plane and the previous bit plane, the more probable that it will be the block most similar to each other. Therefore, the matching error criterion of NNMP is extended to improve ME performance. Experimental results show that the proposed algorithm improves the performance of ME accuracy by 0.27, 0.38 and 0.67 dB compared to the C1BT-based ME, two-bit transform (2BT)-based ME and 1BT-based ME, respectively.

The growing popularity of high resolution video and the continuously increasing demands for high quality video on mobile devices are producing stronger needs for more efficient video encoder. Concerning these desires, HEVC, a newest video coding standard, has been developed by a joint team formed by ISO/IEO MPEG and ITU/T VCEG. Its design goal is to achieve a 50% compression gain over its predecessor H.264 with an equal or even higher perceptual video quality. Motion Estimation (ME) being as one of the most critical module in video coding contributes almost 50%-70% of computational complexity in the video encoder. This high consumption of the computational resources puts a limit on the performance of encoders, especially for full HD or ultra HD videos, in terms of coding speed, bit-rate and video quality. Thus the major part of this work concentrates on the computational complexity reduction and improvement of timing performance of motion estimation algorithms for HEVC standard. First, a new strategy to calculate the SAD (Sum of Absolute Difference) for motion estimation is designed based on the statistics on property of pixel data of video sequences. This statistics demonstrates the size relationship between the sum of two sets of pixels has a determined connection with the distribution of the size relationship between individual pixels from the two sets. Taking the advantage of this observation, only a small proportion of pixels is necessary to be involved in the SAD calculation. Simulations show that the amount of computations required in the full search algorithm is reduced by about 58% on average and up to 70% in the best case. Secondly, from the scope of parallelization an enhanced TZ search for HEVC is proposed using novel schemes of multiple MVPs (motion vector predictor) and shared MVP. Specifically, resorting to multiple MVPs the initial search process is performed in parallel at multiple search centers, and the ME processing engine for PUs within one CU are parallelized based on the MVP sharing scheme on CU (coding unit) level. Moreover, the SAD module for ME engine is also parallelly implemented for PU size of 32×32. Experiments indicate it achieves an appreciable improvement on the throughput and coding efficiency of the HEVC video encoder. In addition, the other part of this thesis is contributed to the VLSI architecture design for finding the first W maximum/minimum values targeting towards high speed and low hardware cost. The architecture based on the novel bit-wise AND scheme has only half of the area of the best reference solution and its critical path delay is comparable with other implementations. While the FPCG (full parallel comparison grid) architecture, which utilizes the optimized comparator-based structure, achieves 3.6 times faster on average on the speed and even 5.2 times faster at best comparing with the reference architectures. Finally the architecture using the partial sorting strategy reaches a good balance on the timing performance and area, which has a slightly lower or comparable speed with FPCG architecture and a acceptable hardware cost

Optical flow can greatly improve the robustness of visual tracking algorithms. While dense optical flow algorithms have various applications, they can not be used for real-time solutions without resorting to GPU calculations. Furthermore, most optical flow algorithms fail in challenging lighting environments due to the violation of the brightness constraint. We propose a simple but effective iterative regularisation scheme for real-time, sparse optical flow algorithms, that is shown to be robust to sudden illumination changes and can handle large displacements. The algorithm proves to outperform well known techniques in real life video sequences, while being much faster to calculate. Our solution increases the robustness of a real-time particle filter based tracking application, consuming only a fraction of the available CPU power. Furthermore, a new and realistic optical flow dataset with annotated ground truth is created and made freely available for research purposes.

Particle image velocimetry (PIV) has matured as a flow measurement technique. It enables the description of the instantaneous velocity field of the flow by analyzing the particle motion obtained from digitally recorded images. Correlation based PIV evaluation technique is widely used because of its good accuracy and robustness. Although very successful, correlation PIV technique has some weakness which can be avoided by optical flow based PIV algorithms. At present, most of the optical flow methods applied to PIV are based on brightness constancy assumption. However, some factors of flow imaging technology and the nature property of the fluids make the brightness constancy assumption less appropriate in real PIV cases. In this paper, an implementation of a 2D optical flow algorithm (GCOF) based on gradient constancy assumption is introduced. The proposed GCOF assumes the edges of the illuminated PIV particles are constant during motion. It comprises two terms: a combined local-global gradient data term and a first-order divergence and vorticity smooth term. The approach can provide accurate dense motion fields. The approach are tested on synthetic images and on two experimental flows. The comparison of GCOF with other optical flow algorithms indicates the proposed method is more accurate especially in conditions of illumination variation. The comparison of GCOF with correlation PIV technique shows that the proposed GCOF has advantages on preserving small divergence and vorticity structures of the motion field and getting less outliers. As a consequence, the GCOF acquire a more accurate and better topological description of the turbulent flow.

It has been a challenging task to estimate optical flow for videos in which either foreground or background exhibits remarkable motion information (i.e., large displacement), or those with insufficient resolution due to artifacts like motion blur or noise. We present a novel optical flow algorithm, which approaches the above problem as solving the task of energy minimization, which exploits image data and smoothness terms at the superpixel level. Our proposed method can be considered as an extended mean-shift algorithm, which advances color and gradient information of superpixels across consecutive frames with smoothness guarantees. Since we do not require assumptions of linearlization during optimization (as standard optical flow approaches do), we are able to alleviate local minimum problems and thus produce improved estimation results. Empirical results on the MPI-Sintel video dataset verify the effectiveness of our proposed method.