FPGA based real-time object detection approach with validation of precision and performance

This thesis work presents the implementation and validation of image processing problems in hardware to estimate the performance and precision gain. It compares the implementation for the addressed problem on a Field Programmable Gate Array (FPGA) with a software implementation for a General Purpose Processor (GPP) architecture. For both solutions the implementation costs for their development is an important aspect in the validation. The analysis of the flexibility and extendability that can be achieved by a modular implementation for the FPGA design was another major aspect. This work is based upon approaches from previous work, which included the detection of Binary Large OBjects (BLOBs) in static images and continuous video streams [13, 15]. One addressed problem of this work is the tracking of the detected BLOBs in continuous image material. This has been implemented for the FPGA platform and the GPP architecture. Both approaches have been compared with respect to performance and precision. This research project is motivated by the MI6 project of the Computer Vision research group, which is located at the Bonn-Rhein-Sieg University of Applied Sciences. The intent of the MI6 project is the tracking of a user in an immersive environment. The proposed solution is to attach a light emitting device to the user for tracking the created light dots on the projection surface of the immersive environment. Having the center points of those light dots would allow the estimation of the user’s position and orientation. One major issue that makes Computer Vision problems computationally expensive is the high amount of data that has to be processed in real-time. Therefore, one major target for the implementation was to get a processing speed of more than 30 frames per second. This would allow the system to realize feedback to the user in a response time which is faster than the human visual perception. One problem that comes with the idea of using a light emitting device to represent the user, is the precision error. Dependent on the resolution of the tracked projection surface of the immersive environment, a pixel might have a size in cm2. Having a precision error of only a few pixels, might lead to an offset in the estimated user’s position of several cm. In this research work the development and validation of a detection and tracking system for BLOBs on a Cyclone II FPGA from Altera has been realized. The system supports different input devices for the image acquisition and can perform detection and tracking for five to eight BLOBs. A further extension of the design has been evaluated and is possible with some constraints. Additional modules for compressing the image data based on run-length encoding and sub-pixel precision for the computed BLOB center-points have been designed. For the comparison of the FPGA approach for BLOB tracking a similar implementation in software using a multi-threaded approach has been realized. The system can transmit the detection or tracking results on two available communication interfaces, USB and RS232. The analysis of the hardware solution showed a similar precision for the BLOB detection and tracking as the software approach. One problem is the strong increase of the allocated resources when extending the system to process more BLOBs. With one of the applied target platforms, the DE2-70 board from Altera, the BLOB detection could be extended to process up to thirty BLOBs. The implementation of the tracking approach in hardware required much more effort than the software solution. The design of high level problems in hardware for this case are more expensive than the software implementation. The search and match steps in the tracking approach could be realized more efficiently and reliably in software. The additional pre-processing modules for sub-pixel precision and run-length-encoding helped to increase the system’s performance and precision.

Online fault detection in industrial machinery, such as induction motors or their components (e.g., bearings), continues to be a priority. Most commercial equipment provides general measurements and not a diagnosis. On the other hand, commonly, research works that focus on fault detection are tested offline or over processors that do not comply with an online diagnosis. In this sense, the present work proposes a system based on a proprietary field programmable gate array (FPGA) platform with several developed intellectual property cores (IPcores) and tools. The FPGA platform together with a stray magnetic flux sensor are used for the online detection of faults in the outer race of bearings in induction motors. The integrated parts comprising the monitoring system are the stray magnetic flux triaxial sensor, several developed IPcores, an embedded processor for data processing, and a user interface where the diagnosis is visualized. The system performs the fault diagnosis through a statistical analysis as follows: First, a triaxial sensor measures the stray magnetic flux in the motor’s surroundings (this flux will vary as symptoms of the fault). Second, an embedded processor in an FPGA-based proprietary board drives the developed IPcores in calculating the statistical features. Third, a set of ranges is defined for the statistical features values, and it is used to indicate the condition of the bearing in the motor. Therefore, if the value of a statistical feature belongs to a specific range, the system will return a diagnosis of whether a fault is present and, if so, the severity of the damage in the outer race. The results demonstrate that the values of the root mean square (RMS) and kurtosis, extracted from the stray magnetic field from the motor, provide a reliable diagnostic of the analyzed bearing. The results are provided online and displayed for the user through interfaces developed on the FPGA platform, such as in a liquid crystal display or through serial communication by a Bluetooth module. The platform is based on an FPGA XC6SLX45 Spartan 6 of Xilinx, and the architecture of the modules used are described through hardware description language. This system aims to be an online tool that can help users of induction motors in maintenance tasks and for the early detection of faults related to bearings.

Some algorithms benefit from a hardware digital logic implantation to achieve higher speed or to meet specific timing requirements, such as in digital signal processing, digital communication, and also when investigating hardware-accelerated machine learning algorithms. We present an extensible, miniature, battery-operated Field Programmable Gate Array (FPGA) platform for wearable computing and IoT research, based on an Intel MAX10 FPGA. The platform is 30x30mm in size and can be used as a standalone device, or as an extension to a similarly sized microcontroller board, for example to pre-process high-speed data streams in hardware prior to relaying the data to a conventional processor. We present the FPGA board and characterise power consumption, resource usage, and processing speed for the implementation of elementary DSP operations, notably FIR filters. We also carry out a direct comparison of these metrics for the FIR algorithm running on an ARM Cortex M4 processor as well as a soft-core processor synthesized on the FPGA board. The results show that this miniature FPGA platform has sufficient logic gates and computing power for a wide class of digital communication algorithms. The platform hardware and firmware is available on GitHub.

Compared with the four-phase optical structure, the grating encoder based on two-phase optical structure reduces the number of optical devices used in the system and makes the system more compact. Due to the high requirements for realtime and parallel processing of algorithm solution, the powerful parallel computing ability of Field Programmable Gate Array (FPGA) and customized hardware acceleration algorithm are needed to improve the real-time performance. In the previous research, the displacement signal generated by the grating encoder can be input into the FPGA through analog to digital converter (ADC) sampling, and then complete self-designed filter filtering, phase correction and displacement solution. In this paper, further, the ADC sampling rate adjustable interface is added to the FPGA, the global signal and the dc offset remove algorithm is added, and the displacement solution results in the form of fixed-point number are output to the host computer through the MicroBlaze (MB) soft core. MB core can realize process control and interface conversion on FPGA, and use a small amount of logic resources to replace the functions of MCU and DSP of traditional embedded measurement system, so as to further improve the integration of the instrument. A series of experiments are carried out on the two-phase FPGA platform. ADC sampling rate is 200ksps, 8-Channel synchronous parallel sampling, FPGA system clock frequency is 200MHz. The linear displacement table is set to drive the measurement grating at different displacement speeds, and the total stroke is set to 10mm. The FPGA real-time displacement solution platform is tested. The experimental results show that FPGA obtains accurate displacement solution results under different speed tests. In the test of 2 mm/s, the maximum cumulative displacement measurement error is 5um, which shows the real-time performance and accurate displacement solution performance of FPGA platform.

Hyperspectral image contains various wavelength channels and the corresponding imagery processing requires a computation platform with high performance. Target and anomaly detection on hyperspectral image has been concerned because of its practicality in many real-time detection fields while wider applicability is limited by the computing condition and low processing speed. The field programmable gate arrays (FPGAs) offer the possibility of on-board hyperspectral data processing with high speed, low-power consumption, reconfigurability and radiation tolerance. In this paper, we develop a novel FPGA-based technique for efficient real-time target detection algorithm in hyperspectral images. The collaborative representation is an efficient target detection (CRD) algorithm in hyperspectral imagery, which is directly based on the concept that the target pixels can be approximately represented by its spectral signatures, while the other cannot. To achieve high processing speed on FPGAs platform, the CRD algorithm reduces the dimensionality of hyperspectral image first. The Sherman–Morrison formula is utilized to calculate the matrix inversion to reduce the complexity of overall CRD algorithm. The achieved results demonstrate that the proposed system may obtains shorter processing time of the CRD algorithm than that on 3.40 GHz CPU.

Non-linear solar photovoltaic (SPV) system characteristics and its dependency on meteorological variables of irradiance and temperature; renders this energy source rather involved to visualize. In this paper a low power realtime SPV module simulator is realized in field programmable gate array (FPGA) XCS100E platform for classroom teaching. FPGA platform is used due to its parallel computing nature and lucid hardware description languages (HDL) available. Output of FPGA being high frequency PWM (pulse width modulated) signals, can be used to upgrade the realized simulator at actual power levels through power electronics interfacing. SPV module with MPPT enabling feature is included, which can be used for testing and deployment of various MPPT techniques.

In the presented work the FPGA based run-time reconfigurable platform with temporal partitioning of hardware resources is proposed. This platform is based on the Field Programmable Gate Array (FPGA) device that can be reconfigured "on-fly" to provide the optimal adaptation of a processing architecture to the algorithm and data structure by utilization of developed mechanisms of temporal partitioning of computational / logic resources. It was shown that the proposed approach allows reaching very high cost-effectiveness of the computing platform oriented on processing of framed data-streams. On the other hand, the hardware programming and compilation processes could be simplified by utilization of library of precompiled Virtual Hardware Components stored in the on-board FLASH memory. Paper presents theoretical proof of the proposed approach by analytical comparison of the performance that could be reached on the conventional processors and FPGA platform with Temporal Partitioning Mechanism (TPM) of hardware resources. The implementation of the proposed TPM on the basis of Xilinx Spartan-3 and Xilinx Virtex II FPGA devices is described. Experimental results gained on the prototype of the FPGA based platform with TPM are discussed and analyzed. Keywords: reconfigurable computing, data-stream processing, FPGA, run-time reconfiguration, temporal partitioning.

With the rapid development of in-depth learning, neural network and deep learning algorithms have been widely used in various fields, e.g., image, video and voice processing. However, the neural network model is getting larger and larger, which is expressed in the calculation of model parameters. Although a wealth of existing efforts on GPU platforms currently used by researchers for improving computing performance, dedicated hardware solutions are essential and emerging to provide advantages over pure software solutions. In this paper, we systematically investigate the neural network accelerator based on Field-Programmable Gate Array(FPGA). Specifically, we respectively review the accelerators designed for specific problems, specific algorithms, algorithm features, and general templates. We also compared the design and implementation of the accelerator based on FPGA under different devices and network models and compared it with the versions of CPU and GPU. Finally, we present to discuss the advantages and disadvantages of accelerators on FPGA platforms and to further explore the opportunities for future research.

Neural networks (NNs) are now being extensively utilized in various artificial intelligence platforms specifically in the area of image classification and real-time object tracking. We propose a novel design to address the problem of real-time unmanned aerial vehicle (UAV) monitoring and detection using a Zynq UltraScale FPGA-based convolutional neural network (CNN). The biggest challenge while implementing real-time algorithms on FPGAs is the limited DSP hardware resources available on FPGA platforms. Our proposed design overcomes the challenge of autonomous real-time UAV detection and tracking using a Xilinx’s Zynq UltraScale XCZU9EG system on a chip (SoC) platform. Our proposed design explores and provides a solution for overcoming the challenge of limited floating-point resources while maintaining real-time performance. The solution consists of two modules: UAV tracking module and neural network–based UAV detection module. The tracking module uses our novel background-differencing algorithm, while the UAV detection is based on a modified CNN algorithm, designed to give the maximum field-programmable gate array (FPGA) performance. These two modules are designed to complement each other and enabled simultaneously to provide an enhanced real-time UAV detection for any given video input. The proposed system has been tested on real-life flying UAVs, achieving an accuracy of 82%, running at the full frame rate of the input camera for both tracking and neural network (NN) detection, achieving similar performance than an equivalent deep learning processor unit (DPU) with UltraScale FPGA-based HD video and tracking implementation but with lower resource utilization as shown by our results.

Multiple user satellites monitor is an important technology which can improve both the monitoring network coverage and the capacity of multi-target signal processing. Multi user detection (MUD) is one of the key technology for multiple user satellites communication. In this paper, a new algorithm for MUD is implemented on field-programmable gate array (FPGA). This FPGA design is based on the minimum mean square error (MMSE) algorithm which consists of two main components: Matched filter and minimum mean square error matrix. Additionally, in order to conduct Monte Carlo simulation, we adopt a way which based on Mat lab, UART and FPGA platform.

This paper describes the development of several software-based games using a high-level programming language (C in our case), designed so that they can be ported to a Field-Programmable Gate Array (FPGA). It also outlines the mathematical foundations underlying these games. Making executables portable in this way can simplify running applications on FPGA platforms. Porting a game to an FPGA serves as evidence that arbitrary executables can be migrated to such hardware. The complete workflow for creating the game, along with the final game outcomes, is detailed in this paper. In addition, statistical analyses of these games were conducted. The proposed approach relies on graphics and character-handling libraries typically available in advanced programming languages. The background of this work is that a microcontroller architecture which can easily be run on a Spartan-6 FPGA was needed. The innovative point of this paper is that it created the cross-compilation toolchain on an uncommon microcontroller architecture, like the OpenRISC.

The detection of gravitational waves (GWs) from binary neutron stars (BNSs) with possible telescope follow-ups opens a window to ground-breaking discoveries in the field of multi-messenger astronomy. With the improved sensitivity of current and future GW detectors, more BNS detections are expected in the future. Therefore, enhancing low-latency GW search algorithms to achieve rapid speed, high accuracy, and low computational cost is essential. One innovative solution to reduce latency is the use of machine learning (ML) methods embedded in field-programmable gate arrays (FPGAs). In this work, we present a novel WaveNet-based method, leveraging the state-of-the-art ML model, to produce early-warning alerts for BNS systems. Using simulated GW signals embedded in Gaussian noise from the Advanced LIGO and Advanced Virgo detectors’ third observing run (O3) as a proof-of-concept dataset, we demonstrate significant performance improvements. Compared to the current leading ML-based early-warning system, our approach enhances detection accuracy from 66.81% to 76.22% at a 1% false alarm probability. Furthermore, we evaluate the time, energy, and economical cost of our model across CPU, GPU, and FPGA platforms, showcasing its potential for deployment in real-time GW detection pipelines.

Deep neural networks (DNNs) are widely used in human activity recognition (HAR). Compared to traditional HAR, embedded real-time HAR offers portability, low latency, and low power consumption. U-Net-based models enhance feature fusion and temporal resolution but face deployment challenges due to high complexity. This study proposes a hardware-accelerated U-Net++ inference model for embedded HAR. The U-Net++ enables network pruning for resourcelimited devices, and quantization converts FLOAT32 parameters to INT8 format for efficient execution. Implemented on a field programmable gate array (FPGA) device, the design optimizes digital signal processor (DSP) utilization via multiplexing and enhances data bandwidth using on-chip block RAM and FIFO-based weight caching. Validated on a Xilinx FPGA, the design achieves 99.0% and 96.6% accuracy on the WISDM and PAMAP2 datasets, respectively, using 39590 lookup tables, 35707 flip-flops, and 400 DSPs. Working at 267 MHz, it achieves <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$330 \mu$</tex> s inference latency and <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$25.78 \text{GOPS} / \mathrm{W}$</tex> energy efficiency, supporting configurable input channels and network parameters across various FPGA platforms.

Software intellectual property (SWIP) is a critical component of increasingly complex field programmable gate arrays (FPGA)-based system-on-chip (SOC) designs. As a result, developers want to ensure that their Software Intellectual Property (SWIP) is protected from being exposed to or tampered with by unauthorized parties. By restricting the execution of SWIP to a single trusted FPGA platform, SWIP binding addresses developers' concerns about maintaining control of their intellectual property and the market position it affords. This work proposes a novel design flow for SWIP binding on a commodity FPGA platform lacking specialized hardcore security facilities. We accomplish this by leveraging the qualities of a Physical Unclonable Function (PUF) and a tight integration of hardware and software security features. A prototype implementation demonstrates our design flow's ability to successfully protect software by encryption using a 128 bit FPGA-unique key extracted from a PUF. Based on this proof of concept, a solution to perform secure remote software updates, a common challenge in embedded systems, is proposed to showcase the practicality and flexibility of the design flow.

Energy efficient low density parity check scheme for body channel communication using FPGA