An Overview of FPGA Based Deep Learning Accelerators: Challenges and Opportunities

Online fault detection in industrial machinery, such as induction motors or their components (e.g., bearings), continues to be a priority. Most commercial equipment provides general measurements and not a diagnosis. On the other hand, commonly, research works that focus on fault detection are tested offline or over processors that do not comply with an online diagnosis. In this sense, the present work proposes a system based on a proprietary field programmable gate array (FPGA) platform with several developed intellectual property cores (IPcores) and tools. The FPGA platform together with a stray magnetic flux sensor are used for the online detection of faults in the outer race of bearings in induction motors. The integrated parts comprising the monitoring system are the stray magnetic flux triaxial sensor, several developed IPcores, an embedded processor for data processing, and a user interface where the diagnosis is visualized. The system performs the fault diagnosis through a statistical analysis as follows: First, a triaxial sensor measures the stray magnetic flux in the motor’s surroundings (this flux will vary as symptoms of the fault). Second, an embedded processor in an FPGA-based proprietary board drives the developed IPcores in calculating the statistical features. Third, a set of ranges is defined for the statistical features values, and it is used to indicate the condition of the bearing in the motor. Therefore, if the value of a statistical feature belongs to a specific range, the system will return a diagnosis of whether a fault is present and, if so, the severity of the damage in the outer race. The results demonstrate that the values of the root mean square (RMS) and kurtosis, extracted from the stray magnetic field from the motor, provide a reliable diagnostic of the analyzed bearing. The results are provided online and displayed for the user through interfaces developed on the FPGA platform, such as in a liquid crystal display or through serial communication by a Bluetooth module. The platform is based on an FPGA XC6SLX45 Spartan 6 of Xilinx, and the architecture of the modules used are described through hardware description language. This system aims to be an online tool that can help users of induction motors in maintenance tasks and for the early detection of faults related to bearings.

Some algorithms benefit from a hardware digital logic implantation to achieve higher speed or to meet specific timing requirements, such as in digital signal processing, digital communication, and also when investigating hardware-accelerated machine learning algorithms. We present an extensible, miniature, battery-operated Field Programmable Gate Array (FPGA) platform for wearable computing and IoT research, based on an Intel MAX10 FPGA. The platform is 30x30mm in size and can be used as a standalone device, or as an extension to a similarly sized microcontroller board, for example to pre-process high-speed data streams in hardware prior to relaying the data to a conventional processor. We present the FPGA board and characterise power consumption, resource usage, and processing speed for the implementation of elementary DSP operations, notably FIR filters. We also carry out a direct comparison of these metrics for the FIR algorithm running on an ARM Cortex M4 processor as well as a soft-core processor synthesized on the FPGA board. The results show that this miniature FPGA platform has sufficient logic gates and computing power for a wide class of digital communication algorithms. The platform hardware and firmware is available on GitHub.

Building the machines that advance scientific knowledge is one of the most significant challenges of our time. So-called Big Science instruments are unique; the expertise required to build them is multidisciplinary, and few countries can build and maintain them. Since resources and budgets are not unlimited, these require efficient solutions. Big Science machines take a decade to plan, a decade to build, and at least a decade to operate. One such machine is the largest fusion reactor project, ITER, which promises to be the next step in fusion research and has a big challenge in system integration. To mitigate the integration effort, ITER has invested in standardization, both in software and hardware, introducing standard instrumentation software named Nominal Device Support (NDSv3); Micro Telecommunication Computing Architecture (MTCA) as the long-term hardware standard choice, and EPICS as a general-purpose distributed control system. These control systems are essential in the management of the facility. They provide functions such as communication protocols, process management, monitoring, alarms, and human-machine interfaces. But also, more advanced access to databases or data acquisition systems. The latter is typically managed by a Field Programmable Gate Array (FPGA), of which facilities can easily have thousands of these FPGAs managing millions of signals within the control system environment. Nevertheless, these FPGA data acquisition (DAQ) systems continue to be labor-intensive. Standardized, modular, and open-source frameworks have been proposed to solve this problem, however, the FPGA programming expertise requirement remained. This thesis proposes a complete technology stack that considers these DAQ systems from the control system to the hardware. It provides a methodology using the Open Computing Language (OpenCL) to reduce the FPGA programming effort. The FPGA devices are programmed using C/C++-like environments, which are much simpler than traditional Hardware Description Languages (HDLs). The software, named IRIO-OpenCL, is written in C++ following ITER's NDSv3 model, which abstracts and generalizes the development of software device drivers and simplifies the interface to EPICS. IRIO-OpenCL is specifically designed to simplify the integration of intelligent instrumentation systems, allowing the integration of algorithms into the FPGA without modifying large parts of the supporting library. The MTCA standard was chosen to maintain modularity. The chassis backplane connects to the devices via Ethernet and PCIe, enabling standard communications. The Advanced Mezzanine Card is the typical MTCA FPGA-based DAQ carrier board. It connects the FPGA to the chassis backplane and typically provides FPGA Mezzanine Connectors (FMCs) for plugging in analog-to-digital (ADC) carrier boards. An ADC connected via the FMC interface samples the analog signals passed to the OpenCL processing kernels. Using OpenCL, the FPGA kernels communicate with the host machine in a standardized way. To demonstrate the applicability of this methodology, reference problems were selected from the literature and implemented, both with traditional and deep learning (DL) solutions. The methodology elaborates on connecting the OpenCL kernels for any use case, and the results show examples of kernel sets. The neutron flux measurement problem is solved at 1GS/s in real time using pulse counting and Campbell techniques, using 40% of the target Arria10 FPGA. The DL application example is based on a neutron/gamma discrimination neural network (NN) that achieves a classification rate of up to 1.3 MEvents/s using less than 25% of the target resource. RESUMEN Construir máquinas que avancen el conocimiento científico es uno de los desafíos más significativos de nuestro tiempo. Los instrumentos de la llamada Big Science son únicos; las competencias requeridas para su construcción son multidisciplinares, y pocos países pueden construirlos y mantenerlos. Dado que los recursos y presupuestos no son ilimitados, estos requieren soluciones eficientes. Las máquinas de Gran Ciencia tardan una década en planificarse, una década en construirse y al menos una década en operarse. Una de estas máquinas es el proyecto del reactor de fusión más grande, ITER, que promete ser el siguiente paso en la investigación de fusión y presenta un gran desafío en la integración de sistemas. Para mitigar el esfuerzo de integración, ITER ha invertido en estandarización, tanto en software como en hardware, introduciendo un software de instrumentación estándar llamado Nominal Device Support v3 (NDSv3); Micro Telecommunication Computing Architecture (MTCA) como la elección estándar de hardware a largo plazo, y EPICS como un sistema de control distribuido de propósito general. Estos sistemas de control son esenciales en la gestión de la instalación ya que proporcionan funciones como protocolos de comunicación, gestión de procesos, monitorización, alarmas e interfaces de usuario, además de acceso avanzado a bases de datos o sistemas de adquisición de datos. Este último es típicamente gestionado por un Field Programmable Gate Array (FPGA), de los cuales las instalaciones pueden tener fácilmente miles de unidades gestionando millones de señales dentro del entorno del sistema de control. Estos sistemas de adquisición de datos (DAQ) con FPGA siguen requiriendo programación experta y laboriosa. Se han propuesto marcos estandarizados, modulares y de código abierto para resolver este problema, sin embargo, la necesidad de experiencia en programación FPGA no se ha podido eludir. Esta tesis propone una pila de tecnologías que considera estos sistemas DAQ desde el sistema de control hasta el hardware. Proporciona una metodología utilizando el Open Computing Language (OpenCL) para reducir el esfuerzo de programación FPGA. Los dispositivos FPGA se programan utilizando entornos similares a C/C++, que son mucho más simples que los Lenguajes de Descripción de Hardware (HDL) tradicionales. El software, llamado IRIO-OpenCL, está escrito en C++ siguiendo el modelo de NDSv3 de ITER, que abstrae y generaliza el desarrollo de controladores de dispositivos de software y simplifica la interfaz con EPICS. IRIO-OpenCL está específicamente diseñado para simplificar la integración de sistemas de instrumentación inteligentes, permitiendo la integración de algoritmos en el FPGA sin modificar grandes partes de la biblioteca de soporte. El estándar MTCA fue elegido para mantener la modularidad. El backplane del chasis se conecta a los dispositivos a través de Ethernet y PCIe, habilitando comunicaciones estándar. La Advanced Mezzanine Card es la placa portadora DAQ FPGA típica de MTCA, que conecta el FPGA al backplane del chasis y típicamente proporciona FPGA Mezzanine Connectors (FMC) para conectar placas con conversores analógico-digitales (ADC). Un ADC conectado a través de la interfaz FMC muestrea las señales analógicas pasadas a los núcleos de procesamiento OpenCL. Utilizando OpenCL, los núcleos FPGA se comunican con la máquina anfitriona de una manera estandarizada. Para demostrar la aplicabilidad de esta metodología, se seleccionaron e implementaron problemas de referencia de la literatura, tanto con soluciones tradicionales como de Deep Learning (DL). La metodología elabora ejemplos de conexión de núcleos OpenCL para cualquier caso de uso, y los resultados muestran ejemplos de conjuntos de núcleos. El problema de medición del flujo de neutrones se resuelve a 1GS/s en tiempo real utilizando técnicas de conteo de pulsos y Campbell, utilizando el 40% del objetivo FPGA Arria10. El ejemplo de aplicación DL se basa en una red neuronal para la discriminación neutrones/gammas que logra una tasa de clasificación de hasta 1.3 MEventos/s utilizando menos del 25% de la FPGA objetivo. LABURPENA Ezagutza zientifikoa aurreratzen duten makinak eraikitzea gure garaiko erronka garrantzitsuenetako bat da. Big Science deritzonaren makinak bakarrak dira; beharrezko esperientzia diziplina anitzekoa da, eta herrialde gutxik eraiki eta mantendu ditzakete. Baliabideak eta aurrekontuak mugagabeak ez direnez, konponbide eraginkorrak behar dira. Zientzia Handi makinak hamarkada bat behar dute diseinatuak izateko, hamarkada bat eraikitzeko eta gutxienez hamarkada bat jarduteko. Makina horietako bat fusio-erreaktore handienaren proiektua da, ITER, fusio-ikerketan hurrengo urratsa izango dena eta sistemak integratzeko erronka handia aurkezten duena. Integrazio-ahalegina arintzeko, ITERek estandarizazioan inbertitu du, bai softwarean bai - eta EPICS helburu orokorreko kontrol-sistema banatua. Kontrol-sistema horiek funtsezkoak dira instalazioa kudeatzeko. Komunikazio-protokoloak, prozesuen kudeaketa, monitorizazioa, alarmak eta pertsona-makina interfazeak bezalako funtzioak ematen dituzte, datubaseetarako edo datuak eskuratzeko sistemetarako sarbide aurreratua izateaz gain. Azken hori "Field Programmable Gate Array" (FPGA) batek kudeatzen du, milaka FPGA konektatuz makina bakoitzean eta kontrol-sistemaz milioika seinale kudeatuz. FPGA duten datuak eskuratzeko sistemek (DAQ) programazio aditua eta neketsua eskatzen jarraitzen dute. Problema hori ebazteko marko estandarizatuak, modularrak eta kode irekikoak proposatu dira, hala ere, FPGA hardware programazio espezializatuaren beharra ezin izan da ukatu. Tesi honek DAQ sistema hauek kontrol sistematik hardwareraino kontsideratuz teknologia bilduma bat programazio-ahalegina murrizten du. FPGA gailuak C/C++ antzeko inguruneak erabiliz programatzen dira, - OpenCL izenekoa, C++n idatzita dago, ITERren "Nominal Device Support v3" (NDSv3) eredua jarraituz. Honek software-gailuen garapena diseinu abstraktu eta orokorrez EPICSekiko interfazea sinplifikatzen du. IRIOOpenCL berariaz diseinatuta dago instrumentazio-sistema adimendunen integrazioa sinplifikatzeko, eta algoritmoak FPGAn integratzea ahalbidetzen du, software zati handiak aldatu gabe. Modularitateari eusteko, MTCA estandarra aukeratu da. Txasisaren backplanea Ethernet eta PCIe bidez konektatzen da gailuetara, komunikazio estandarrak gaituz. "Advanced Mezzanine Card" MTCAren ohiko DAQ FPGA plaka eramailea da. FPGA txasisaren backplanera konektatzen du eta, normalean, "FPGA Mezzanine Connectors"ak (FMC) ditu plaka bihurgailu analogiko-digitalak (ADC) konektatzeko. ADCak, FMC interfazearen bidez konektatuta, OpenCL prozesatze-nukleoetara igarotako seinale analogikoak bihurtzen ditu. OpenCL erabiliz, FPGA nukleoak ordenagailuarekin modu estandarizatuan komunikatzen dira. Metodologia horren aplikagarritasuna egiaztatzeko, erreferentziazko adibideak literaturatik hautatu eta nukleoen konexio-adibideak egiten ditu, edozein kasutan erabiltzeko, eta emaitzek nukleo multzoen adibideak erakusten dituzte. Neutroi-fluxuaren neurketak 1GS/s-ko ebazten dira real time errendimenduaz, pultsuak eta Campbell zenbatzeko teknikak erabiliz, Arria10 FPGAren % 40 erabiliz. Azkenik, DL aplikazioaren adibidea, neutroi/gamma sailkapena, sare neuronal batean oinarritzen da, eta FPGAren baliabideen % 25 baino gutxiago erabiliz 1.3 MSailkapen/s sailkapen-tasa lortzen du.

At present, deep learning is becoming one of the most popular approach to creation of the artificial intelligences systems such as speech recognition, natural language processing, computer vision and so on. The paper presents a historical overview of deep learning in neural networks. The model of the artificial neural network is described as well as the learning algorithms for neural networks including the error backpropagation algorithm, which is used to train deep neural networks. The development of neural networks architectures is presented including neocognitron, autoencoders, convolutional neural networks, restricted Boltzmann machine, deep belief networks, long short-term memory, gated recurrent neural networks, and residual networks. Training deep neural networks with many hidden layers is impeded by the vanishing gradient problem. The paper describes the approaches to solve this problem that provide the ability to train neural networks with more than hundred layers. An overview of popular deep learning libraries is presented. Nowadays, for computer vision tasks convolutional neural networks are utilized, while for sequence processing, including natural language processing, recurrent networks are preferred solution, primarily long short-term memory networks and gated recurrent neural networks.

The deep learning has become the key for artificial intelligence applications development. It was successfully used to solve computer vision tasks. But the deep learning algorithms are based on Deep Neural Networks (DNN) with many hidden layers which need a huge computation effort and a big storage space. Thus, the general-purpose graphical processing units (GPGPU) are the best candidate for DNN development and inference because of their high number of processing core and the big integrated memory. In the other side, the disadvantage of the GPGPU is high-power consumption. In a real-world application, the processing unit is an embedded system based on limited power and computation resources. In recent years, Field Programmable Gate Array (FPGA) becomes a serious solution that can outperform GPGPU because of their flexible architecture and low power consumption. The FPGA is equipped with a very small integrated memory and a low bandwidth. To make DNNs fit into FPGA we need a lot of optimization techniques at different levels such as the network level, the hardware level, and the implementation tools level. In this paper, we will cite the existing optimization techniques and evaluate them to provide a complete overview of FPGA based DNN accelerators.

In this paper, a speech-to-text translation model has been developed for Malaysian speakers based on 41 classes of Phonemes. A simple data acquisition algorithm has been used to develop a MATLAB graphical user interface (GUI) for recording the isolated word speech signals from 35 non-native Malaysian speakers. The collected database consists of 86 words with 41 classes of phoneme based on Affricatives, Diphthongs, Fricatives, Liquid, Nasals, Semivowels and Glides, Stop and Vowels. The speech samples are preprocessed to eliminate the undesirable artifacts and the fuzzy voice classifier has been employed to classify the samples into voiced sequence and unvoiced sequence. The voiced sequences are divided into frame segments and for each frame, the Linear Predictive co-efficients features are obtained from the voiced sequence. Then the feature sets are formed by deriving the LPC features from all the extracted voiced sequences, and used for classification. The isolated words chosen based on the phonemes are associated with the extracted features to establish classification system input-output mapping. The data are then normalized and randomized to rearrange the values into definite range. The Multilayer Neural Network (MLNN) model has been developed with four combinations of input and hidden activation functions. The neural network models are trained with 60%, 70% and 80% of the total data samples. The neural network architecture was aimed at creating a robust model with 60%, 70%, and 80% of the feature set with 25 trials. The trained network model is validated by simulating the network with the remaining 40%, 30%, and 20% of the set. The reliability of trained network models were compared by measuring true-positive, false-negative, and network classification accuracy. The LPC features show better discrimination and the MLNN neural network models trained using the LPC spectral band features gives better recognition.

Edge AI systems are increasingly being adopted in a wide range of application domains. These systems typically deploy Convolutional Neural Network (CNN) models on edge devices to perform inference, while relying on the cloud for model training. This is due to the high computational and memory demands of conventional model training, which exceeds the capabilities of resource-constrained edge devices running on tight power budgets. The dependency on the cloud for training is not suitable in many applications, where new objects or environmental conditions different from the ones present during training, are frequently encountered. In such applications, continual learning of new knowledge on the edge device becomes a necessity to avoid performance bottlenecks due to round-trip communication delays, network connectivity, and the available bandwidth. In this thesis, we propose Field-Programmable Gate Array (FPGA) based accelerator architecture and optimization strategies for a new paradigm of machine learning algorithms that is capable of continual learning. The proposed methods will enable edge FPGA systems to perform on-device deep continual learning for object classification. Specifically, the proposed methods aim to achieve real-time learning on-device, while providing for a high degree of scalability to learn a large number of classes. We first propose a FPGA accelerator for a Self-Organizing Neural Network (SONN), that can perform class-incremental continual learning in a streaming manner when combined with a CNN. The SONN model performs unsupervised learning from embedding features extracted from the CNN model by dynamically growing neurons and connections. We introduce design optimization strategies and runtime scheduling techniques to optimize resource usage, latency, and energy consumption. Experimental results based on Core50 dataset for continuous object recognition from video sequences, demonstrated that the proposed FPGA architecture outperforms CPU and GPU based counterparts in terms of latency and power. However, the SONN model grows proportionally to the classes learnt, which limits its scalability to learn a large number of classes efficiently. Next, we propose a FPGA accelerator for a Streaming Linear Discriminant Analysis (SLDA) model to overcome the scalability limitations of SONN. Similar to the SONN, SLDA performs continual learning from embedding features extracted from a CNN in a streaming manner. SLDA is highly scalable for learning a large number of classes as the network does not grow dynamically to accommodate new knowledge. We propose several design and runtime optimizations to minimize resource usage, latency, and energy consumption. Additionally, we introduce a new variant of SLDA and discuss the accuracy-efficiency trade-offs using popular datasets for continual learning, CoRE50 and CUB200. The results demonstrate that the proposed SLDA accelerator outperforms CPU and GPU counterparts, in terms of latency and energy efficiency. Finally, we demonstrate a full on-chip deep continual learning pipeline on FPGA, by integrating the proposed SLDA accelerator with Xilinx DPU, a programmable CNN accelerator IP. The design is implemented on a Xilinx Zynq Ultrascale+ MPSoC. In order to overcome the large performance bottleneck due to the communication overhead between the ARM processing system (PS) and programmable logic (PL), we implemented a Linux device driver that facilitates efficient memory mapping between PS-PL. The experimental results based on CoRE50 dataset show that the proposed pipeline is capable of performing continual learning at nearly the same latency as the inference pipeline, with only a marginal increase in energy consumption. Our results clearly demonstrate the viability of deploying real-time deep continual learning on edge AI systems that are equipped with FPGA accelerators.

This paper presents the development of several efficient learning algorithms for neural networks (ELEANNE). The ELEANNE 1 and ELEANNE 2 are two recursive least-squares learning algorithms, proposed for training single-layered neural networks with analog output. This paper also proposes a new optimization strategy for training single-layered neural networks, which provides the basis for the development of a variety of efficient learning algorithms. This optimization strategy is the source of the ELEANNE 3, a second-order learning algorithm for training single-layered neural networks with binary output. A simplified version of this algorithm, called ELEANNE 4, is also derived on the basis of some simplifying but reasonable assumptions. The two algorithms developed for single-layered neural networks provide the basis for the derivation of ELEANNE 5 and ELEANNE 6, which are proposed for training multilayered neural networks with binary output. The ELEANNE 7 is an efficient algorithm developed for training multilayered neural networks with either binary or analog output. >

Compared with the four-phase optical structure, the grating encoder based on two-phase optical structure reduces the number of optical devices used in the system and makes the system more compact. Due to the high requirements for realtime and parallel processing of algorithm solution, the powerful parallel computing ability of Field Programmable Gate Array (FPGA) and customized hardware acceleration algorithm are needed to improve the real-time performance. In the previous research, the displacement signal generated by the grating encoder can be input into the FPGA through analog to digital converter (ADC) sampling, and then complete self-designed filter filtering, phase correction and displacement solution. In this paper, further, the ADC sampling rate adjustable interface is added to the FPGA, the global signal and the dc offset remove algorithm is added, and the displacement solution results in the form of fixed-point number are output to the host computer through the MicroBlaze (MB) soft core. MB core can realize process control and interface conversion on FPGA, and use a small amount of logic resources to replace the functions of MCU and DSP of traditional embedded measurement system, so as to further improve the integration of the instrument. A series of experiments are carried out on the two-phase FPGA platform. ADC sampling rate is 200ksps, 8-Channel synchronous parallel sampling, FPGA system clock frequency is 200MHz. The linear displacement table is set to drive the measurement grating at different displacement speeds, and the total stroke is set to 10mm. The FPGA real-time displacement solution platform is tested. The experimental results show that FPGA obtains accurate displacement solution results under different speed tests. In the test of 2 mm/s, the maximum cumulative displacement measurement error is 5um, which shows the real-time performance and accurate displacement solution performance of FPGA platform.

Convolution neural networks (CNNs) as one of today's main flavor of deep learning techniques dominate in various image recognition tasks. As the model size of modern CNNs continues to grow, neural network compression techniques have been proposed to prune the redundant neurons and synapses. However, prior techniques disconnect the software neural networks compression and hardware acceleration, which fail to balance multiple design parameters, including sparsity, performance, hardware area cost, and efficiency. More concretely, prior unstructured pruning techniques achieve high sparsity at the expense of extra performance overhead, while prior structured pruning techniques relying on strict sparse patterns lead to low sparsity and extra hardware cost. In this article, we propose OMNI, a framework for accelerating sparse CNNs on hardware accelerators. The innovation of OMNI stems from that it uses hardware amenable on-chip memory partition patterns to seamlessly engage the software CNN model compression and hardware CNN acceleration. To accelerate the compute-intensive convolution kernel, a promising hardware optimization approach is memory partition, which divides the original weight kernels into several groups so that the different hardware processing elements can simultaneously access the weight. We exploit the memory partition patterns including block, cyclic, or hybrid as a means of CNN compression patterns. Our software CNN model compression balances the sparsity across different groups and our hardware accelerator employs hardware parallelization coordinately with the sparse patterns, leading to a desirable compromise between sparsity and performance. We further develop performance models to help the designers to quickly identify the pattern factors subject to an area constraint. Last, we evaluate our design on application specific integrated circuit (ASIC) and field-programmable gate array (FPGA) platform. Experiments demonstrate that OMNI achieves 3.4×- 6.2× speedup for the modern CNNs, over a comparably ideal dense CNN accelerator. OMNI shows 114.7× energy efficiency improvement compared with GPU platform. OMNI is also evaluated on Xilinx ZC706 and ZCU102 FPGA platforms, achieving 41.5 GOP/s and 125.3 GOP/s, respectively.

Neural networks (NNs) are now being extensively utilized in various artificial intelligence platforms specifically in the area of image classification and real-time object tracking. We propose a novel design to address the problem of real-time unmanned aerial vehicle (UAV) monitoring and detection using a Zynq UltraScale FPGA-based convolutional neural network (CNN). The biggest challenge while implementing real-time algorithms on FPGAs is the limited DSP hardware resources available on FPGA platforms. Our proposed design overcomes the challenge of autonomous real-time UAV detection and tracking using a Xilinx’s Zynq UltraScale XCZU9EG system on a chip (SoC) platform. Our proposed design explores and provides a solution for overcoming the challenge of limited floating-point resources while maintaining real-time performance. The solution consists of two modules: UAV tracking module and neural network–based UAV detection module. The tracking module uses our novel background-differencing algorithm, while the UAV detection is based on a modified CNN algorithm, designed to give the maximum field-programmable gate array (FPGA) performance. These two modules are designed to complement each other and enabled simultaneously to provide an enhanced real-time UAV detection for any given video input. The proposed system has been tested on real-life flying UAVs, achieving an accuracy of 82%, running at the full frame rate of the input camera for both tracking and neural network (NN) detection, achieving similar performance than an equivalent deep learning processor unit (DPU) with UltraScale FPGA-based HD video and tracking implementation but with lower resource utilization as shown by our results.

This paper presents the implementation and evaluation of a computer vision problem on a Field Programmable Gate Array (FPGA). This work is based upon previous work where the feasibility of application specific image processing algorithms on a FPGA platform have been evaluated by experimental approaches. This work coveres the development of a BLOB detection system on an Altera Development and Education II (DE2) board with a Cyclone II FPGA in Verilog. It detects binary spatially extended objects in image material and computes their center points. Bounding Box and Center-of-Mass have been applied for estimating center points of the BLOBs. The results are transmitted via a serial interface to the PC for validation of their ground truth and further processing. The evaluation compares precision and performance gains dependent on the applied computation methods.

Recent advances in Neural Network (NN) models for the Radio Frequency (RF) domain have made them a dominant force in realizing robust architectures that generalize well to novel operating conditions. While the performance of NN models when running on a Graphics Processing Unit (GPU) are generally very good, many applications require lower latency and higher throughput to be edge deployable. We have recently developed physics-driven NN models to perform Digital Signal Processing (DSP) functions for a Long Term Evolution (LTE) receiver application, demonstrating equal or better performance than their DSP equivalents. This paper discusses moving some of these NN models to Field Programmable Gate Array (FPGA) to tackle the latency and throughput goals and evaluate the performance at different quantization levels. We compare the FPGA performance results at different quantization levels with their GPU performance counterpart and discuss the path forward towards an RF edge solution.

This thesis work presents the implementation and validation of image processing problems in hardware to estimate the performance and precision gain. It compares the implementation for the addressed problem on a Field Programmable Gate Array (FPGA) with a software implementation for a General Purpose Processor (GPP) architecture. For both solutions the implementation costs for their development is an important aspect in the validation. The analysis of the flexibility and extendability that can be achieved by a modular implementation for the FPGA design was another major aspect. This work is based upon approaches from previous work, which included the detection of Binary Large OBjects (BLOBs) in static images and continuous video streams [13, 15]. One addressed problem of this work is the tracking of the detected BLOBs in continuous image material. This has been implemented for the FPGA platform and the GPP architecture. Both approaches have been compared with respect to performance and precision. This research project is motivated by the MI6 project of the Computer Vision research group, which is located at the Bonn-Rhein-Sieg University of Applied Sciences. The intent of the MI6 project is the tracking of a user in an immersive environment. The proposed solution is to attach a light emitting device to the user for tracking the created light dots on the projection surface of the immersive environment. Having the center points of those light dots would allow the estimation of the user’s position and orientation. One major issue that makes Computer Vision problems computationally expensive is the high amount of data that has to be processed in real-time. Therefore, one major target for the implementation was to get a processing speed of more than 30 frames per second. This would allow the system to realize feedback to the user in a response time which is faster than the human visual perception. One problem that comes with the idea of using a light emitting device to represent the user, is the precision error. Dependent on the resolution of the tracked projection surface of the immersive environment, a pixel might have a size in cm2. Having a precision error of only a few pixels, might lead to an offset in the estimated user’s position of several cm. In this research work the development and validation of a detection and tracking system for BLOBs on a Cyclone II FPGA from Altera has been realized. The system supports different input devices for the image acquisition and can perform detection and tracking for five to eight BLOBs. A further extension of the design has been evaluated and is possible with some constraints. Additional modules for compressing the image data based on run-length encoding and sub-pixel precision for the computed BLOB center-points have been designed. For the comparison of the FPGA approach for BLOB tracking a similar implementation in software using a multi-threaded approach has been realized. The system can transmit the detection or tracking results on two available communication interfaces, USB and RS232. The analysis of the hardware solution showed a similar precision for the BLOB detection and tracking as the software approach. One problem is the strong increase of the allocated resources when extending the system to process more BLOBs. With one of the applied target platforms, the DE2-70 board from Altera, the BLOB detection could be extended to process up to thirty BLOBs. The implementation of the tracking approach in hardware required much more effort than the software solution. The design of high level problems in hardware for this case are more expensive than the software implementation. The search and match steps in the tracking approach could be realized more efficiently and reliably in software. The additional pre-processing modules for sub-pixel precision and run-length-encoding helped to increase the system’s performance and precision.

Artificial neural network and neuron models have made significant contributions to the area of neurodynamics. Investigating the dynamics of artificial neurons and neural networks is vital in developing brain-like systems and understanding how the brain functions. Neural network models and memristive neurons are currently demonstrating a lot of promise in the study of neurodynamics. In order to model the dynamics of biological synapses, this study explores the complex dynamical behavior of a discrete fractional Hopfield-type neural network using a flux-controlled memristive element with periodic memductance. Hyperbolic tangent and sine are the heterogeneous activation functions that are implemented in the proposed system to improve nonlinearity and replicate various forms of brain activity. Stability and bifurcation analyses are used to illustrate the nonlinear dynamical nature of the constructed network model. We examine how the fractional order (ν) and periodical memductance aspects influence the dynamics of the system to emphasize the emerging complex phenomena like multi-layered dynamics and the presence of several distinct dynamical states throughout the system variables. Randomness and complexity of the time series data for the proposed system are illustrated with the help of approximate entropy analysis. These findings could help researchers better understand brain-like memory networks, neuromorphic computers, and the theoretical study of neurological and mental abilities. The study of multi-layer attractors can be useful in advanced sensory devices, neuromorphic devices, and secure communication.