Fast Convolutional Neural Networks in Low Density FPGAs Using Zero-Skipping and Weight Pruning

A fast and scalable architecture to run convolutional neural networks in low density FPGAs

Edge AI systems are increasingly being adopted in a wide range of application domains. These systems typically deploy Convolutional Neural Network (CNN) models on edge devices to perform inference, while relying on the cloud for model training. This is due to the high computational and memory demands of conventional model training, which exceeds the capabilities of resource-constrained edge devices running on tight power budgets. The dependency on the cloud for training is not suitable in many applications, where new objects or environmental conditions different from the ones present during training, are frequently encountered. In such applications, continual learning of new knowledge on the edge device becomes a necessity to avoid performance bottlenecks due to round-trip communication delays, network connectivity, and the available bandwidth. In this thesis, we propose Field-Programmable Gate Array (FPGA) based accelerator architecture and optimization strategies for a new paradigm of machine learning algorithms that is capable of continual learning. The proposed methods will enable edge FPGA systems to perform on-device deep continual learning for object classification. Specifically, the proposed methods aim to achieve real-time learning on-device, while providing for a high degree of scalability to learn a large number of classes. We first propose a FPGA accelerator for a Self-Organizing Neural Network (SONN), that can perform class-incremental continual learning in a streaming manner when combined with a CNN. The SONN model performs unsupervised learning from embedding features extracted from the CNN model by dynamically growing neurons and connections. We introduce design optimization strategies and runtime scheduling techniques to optimize resource usage, latency, and energy consumption. Experimental results based on Core50 dataset for continuous object recognition from video sequences, demonstrated that the proposed FPGA architecture outperforms CPU and GPU based counterparts in terms of latency and power. However, the SONN model grows proportionally to the classes learnt, which limits its scalability to learn a large number of classes efficiently. Next, we propose a FPGA accelerator for a Streaming Linear Discriminant Analysis (SLDA) model to overcome the scalability limitations of SONN. Similar to the SONN, SLDA performs continual learning from embedding features extracted from a CNN in a streaming manner. SLDA is highly scalable for learning a large number of classes as the network does not grow dynamically to accommodate new knowledge. We propose several design and runtime optimizations to minimize resource usage, latency, and energy consumption. Additionally, we introduce a new variant of SLDA and discuss the accuracy-efficiency trade-offs using popular datasets for continual learning, CoRE50 and CUB200. The results demonstrate that the proposed SLDA accelerator outperforms CPU and GPU counterparts, in terms of latency and energy efficiency. Finally, we demonstrate a full on-chip deep continual learning pipeline on FPGA, by integrating the proposed SLDA accelerator with Xilinx DPU, a programmable CNN accelerator IP. The design is implemented on a Xilinx Zynq Ultrascale+ MPSoC. In order to overcome the large performance bottleneck due to the communication overhead between the ARM processing system (PS) and programmable logic (PL), we implemented a Linux device driver that facilitates efficient memory mapping between PS-PL. The experimental results based on CoRE50 dataset show that the proposed pipeline is capable of performing continual learning at nearly the same latency as the inference pipeline, with only a marginal increase in energy consumption. Our results clearly demonstrate the viability of deploying real-time deep continual learning on edge AI systems that are equipped with FPGA accelerators.

Convolutional neural networks (CNNs) have gained a huge attention for real-world artificial intelligence (AI) applications such as image classification and object detection. On the other hand, for better accuracy, the size of the CNNs’ parameters (weights) has been increasing, which in turn makes it difficult to enable on-device CNN inferences in resource-constrained edge devices. Though weight pruning and 5-bit quantization methods have shown promising results, it is still challenging to deploy large CNN models in edge devices. In this paper, we propose an encoding and hardware-based decoding technique which can be applied to 5-bit quantized weight data for on-device CNN inferences in resource-constrained edge devices. Given 5-bit quantized weight data, we employ arithmetic coding with range scaling for lossless weight compression, which is performed offline. When executing on-device inferences with underlying CNN accelerators, our hardware decoder enables a fast in-situ weight decompression with small latency overhead. According to our evaluation results with five widely used CNN models, our arithmetic coding-based encoding method applied to 5-bit quantized weights shows a better compression ratio by 9.6× while also reducing the memory data transfer energy consumption by 89.2%, on average, as compared to the case of uncompressed 32-bit floating-point weights. When applying our technique to pruned weights, we obtain better compression ratios by 57.5×–112.2× while reducing energy consumption by 98.3%–99.1% as compared to the case of 32-bit floating-point weights. In addition, by pipelining the weight decoding and transfer with the CNN execution, the latency overhead of our weight decoding with 16 decoding unit (DU) hardware is only 0.16%–5.48% and 0.16%–0.91% for non-pruned and pruned weights, respectively. Moreover, our proposed technique with 4-DU decoder hardware reduces system-level energy consumption by 1.1%–9.3%.

Building the machines that advance scientific knowledge is one of the most significant challenges of our time. So-called Big Science instruments are unique; the expertise required to build them is multidisciplinary, and few countries can build and maintain them. Since resources and budgets are not unlimited, these require efficient solutions. Big Science machines take a decade to plan, a decade to build, and at least a decade to operate. One such machine is the largest fusion reactor project, ITER, which promises to be the next step in fusion research and has a big challenge in system integration. To mitigate the integration effort, ITER has invested in standardization, both in software and hardware, introducing standard instrumentation software named Nominal Device Support (NDSv3); Micro Telecommunication Computing Architecture (MTCA) as the long-term hardware standard choice, and EPICS as a general-purpose distributed control system. These control systems are essential in the management of the facility. They provide functions such as communication protocols, process management, monitoring, alarms, and human-machine interfaces. But also, more advanced access to databases or data acquisition systems. The latter is typically managed by a Field Programmable Gate Array (FPGA), of which facilities can easily have thousands of these FPGAs managing millions of signals within the control system environment. Nevertheless, these FPGA data acquisition (DAQ) systems continue to be labor-intensive. Standardized, modular, and open-source frameworks have been proposed to solve this problem, however, the FPGA programming expertise requirement remained. This thesis proposes a complete technology stack that considers these DAQ systems from the control system to the hardware. It provides a methodology using the Open Computing Language (OpenCL) to reduce the FPGA programming effort. The FPGA devices are programmed using C/C++-like environments, which are much simpler than traditional Hardware Description Languages (HDLs). The software, named IRIO-OpenCL, is written in C++ following ITER's NDSv3 model, which abstracts and generalizes the development of software device drivers and simplifies the interface to EPICS. IRIO-OpenCL is specifically designed to simplify the integration of intelligent instrumentation systems, allowing the integration of algorithms into the FPGA without modifying large parts of the supporting library. The MTCA standard was chosen to maintain modularity. The chassis backplane connects to the devices via Ethernet and PCIe, enabling standard communications. The Advanced Mezzanine Card is the typical MTCA FPGA-based DAQ carrier board. It connects the FPGA to the chassis backplane and typically provides FPGA Mezzanine Connectors (FMCs) for plugging in analog-to-digital (ADC) carrier boards. An ADC connected via the FMC interface samples the analog signals passed to the OpenCL processing kernels. Using OpenCL, the FPGA kernels communicate with the host machine in a standardized way. To demonstrate the applicability of this methodology, reference problems were selected from the literature and implemented, both with traditional and deep learning (DL) solutions. The methodology elaborates on connecting the OpenCL kernels for any use case, and the results show examples of kernel sets. The neutron flux measurement problem is solved at 1GS/s in real time using pulse counting and Campbell techniques, using 40% of the target Arria10 FPGA. The DL application example is based on a neutron/gamma discrimination neural network (NN) that achieves a classification rate of up to 1.3 MEvents/s using less than 25% of the target resource. RESUMEN Construir máquinas que avancen el conocimiento científico es uno de los desafíos más significativos de nuestro tiempo. Los instrumentos de la llamada Big Science son únicos; las competencias requeridas para su construcción son multidisciplinares, y pocos países pueden construirlos y mantenerlos. Dado que los recursos y presupuestos no son ilimitados, estos requieren soluciones eficientes. Las máquinas de Gran Ciencia tardan una década en planificarse, una década en construirse y al menos una década en operarse. Una de estas máquinas es el proyecto del reactor de fusión más grande, ITER, que promete ser el siguiente paso en la investigación de fusión y presenta un gran desafío en la integración de sistemas. Para mitigar el esfuerzo de integración, ITER ha invertido en estandarización, tanto en software como en hardware, introduciendo un software de instrumentación estándar llamado Nominal Device Support v3 (NDSv3); Micro Telecommunication Computing Architecture (MTCA) como la elección estándar de hardware a largo plazo, y EPICS como un sistema de control distribuido de propósito general. Estos sistemas de control son esenciales en la gestión de la instalación ya que proporcionan funciones como protocolos de comunicación, gestión de procesos, monitorización, alarmas e interfaces de usuario, además de acceso avanzado a bases de datos o sistemas de adquisición de datos. Este último es típicamente gestionado por un Field Programmable Gate Array (FPGA), de los cuales las instalaciones pueden tener fácilmente miles de unidades gestionando millones de señales dentro del entorno del sistema de control. Estos sistemas de adquisición de datos (DAQ) con FPGA siguen requiriendo programación experta y laboriosa. Se han propuesto marcos estandarizados, modulares y de código abierto para resolver este problema, sin embargo, la necesidad de experiencia en programación FPGA no se ha podido eludir. Esta tesis propone una pila de tecnologías que considera estos sistemas DAQ desde el sistema de control hasta el hardware. Proporciona una metodología utilizando el Open Computing Language (OpenCL) para reducir el esfuerzo de programación FPGA. Los dispositivos FPGA se programan utilizando entornos similares a C/C++, que son mucho más simples que los Lenguajes de Descripción de Hardware (HDL) tradicionales. El software, llamado IRIO-OpenCL, está escrito en C++ siguiendo el modelo de NDSv3 de ITER, que abstrae y generaliza el desarrollo de controladores de dispositivos de software y simplifica la interfaz con EPICS. IRIO-OpenCL está específicamente diseñado para simplificar la integración de sistemas de instrumentación inteligentes, permitiendo la integración de algoritmos en el FPGA sin modificar grandes partes de la biblioteca de soporte. El estándar MTCA fue elegido para mantener la modularidad. El backplane del chasis se conecta a los dispositivos a través de Ethernet y PCIe, habilitando comunicaciones estándar. La Advanced Mezzanine Card es la placa portadora DAQ FPGA típica de MTCA, que conecta el FPGA al backplane del chasis y típicamente proporciona FPGA Mezzanine Connectors (FMC) para conectar placas con conversores analógico-digitales (ADC). Un ADC conectado a través de la interfaz FMC muestrea las señales analógicas pasadas a los núcleos de procesamiento OpenCL. Utilizando OpenCL, los núcleos FPGA se comunican con la máquina anfitriona de una manera estandarizada. Para demostrar la aplicabilidad de esta metodología, se seleccionaron e implementaron problemas de referencia de la literatura, tanto con soluciones tradicionales como de Deep Learning (DL). La metodología elabora ejemplos de conexión de núcleos OpenCL para cualquier caso de uso, y los resultados muestran ejemplos de conjuntos de núcleos. El problema de medición del flujo de neutrones se resuelve a 1GS/s en tiempo real utilizando técnicas de conteo de pulsos y Campbell, utilizando el 40% del objetivo FPGA Arria10. El ejemplo de aplicación DL se basa en una red neuronal para la discriminación neutrones/gammas que logra una tasa de clasificación de hasta 1.3 MEventos/s utilizando menos del 25% de la FPGA objetivo. LABURPENA Ezagutza zientifikoa aurreratzen duten makinak eraikitzea gure garaiko erronka garrantzitsuenetako bat da. Big Science deritzonaren makinak bakarrak dira; beharrezko esperientzia diziplina anitzekoa da, eta herrialde gutxik eraiki eta mantendu ditzakete. Baliabideak eta aurrekontuak mugagabeak ez direnez, konponbide eraginkorrak behar dira. Zientzia Handi makinak hamarkada bat behar dute diseinatuak izateko, hamarkada bat eraikitzeko eta gutxienez hamarkada bat jarduteko. Makina horietako bat fusio-erreaktore handienaren proiektua da, ITER, fusio-ikerketan hurrengo urratsa izango dena eta sistemak integratzeko erronka handia aurkezten duena. Integrazio-ahalegina arintzeko, ITERek estandarizazioan inbertitu du, bai softwarean bai - eta EPICS helburu orokorreko kontrol-sistema banatua. Kontrol-sistema horiek funtsezkoak dira instalazioa kudeatzeko. Komunikazio-protokoloak, prozesuen kudeaketa, monitorizazioa, alarmak eta pertsona-makina interfazeak bezalako funtzioak ematen dituzte, datubaseetarako edo datuak eskuratzeko sistemetarako sarbide aurreratua izateaz gain. Azken hori "Field Programmable Gate Array" (FPGA) batek kudeatzen du, milaka FPGA konektatuz makina bakoitzean eta kontrol-sistemaz milioika seinale kudeatuz. FPGA duten datuak eskuratzeko sistemek (DAQ) programazio aditua eta neketsua eskatzen jarraitzen dute. Problema hori ebazteko marko estandarizatuak, modularrak eta kode irekikoak proposatu dira, hala ere, FPGA hardware programazio espezializatuaren beharra ezin izan da ukatu. Tesi honek DAQ sistema hauek kontrol sistematik hardwareraino kontsideratuz teknologia bilduma bat programazio-ahalegina murrizten du. FPGA gailuak C/C++ antzeko inguruneak erabiliz programatzen dira, - OpenCL izenekoa, C++n idatzita dago, ITERren "Nominal Device Support v3" (NDSv3) eredua jarraituz. Honek software-gailuen garapena diseinu abstraktu eta orokorrez EPICSekiko interfazea sinplifikatzen du. IRIOOpenCL berariaz diseinatuta dago instrumentazio-sistema adimendunen integrazioa sinplifikatzeko, eta algoritmoak FPGAn integratzea ahalbidetzen du, software zati handiak aldatu gabe. Modularitateari eusteko, MTCA estandarra aukeratu da. Txasisaren backplanea Ethernet eta PCIe bidez konektatzen da gailuetara, komunikazio estandarrak gaituz. "Advanced Mezzanine Card" MTCAren ohiko DAQ FPGA plaka eramailea da. FPGA txasisaren backplanera konektatzen du eta, normalean, "FPGA Mezzanine Connectors"ak (FMC) ditu plaka bihurgailu analogiko-digitalak (ADC) konektatzeko. ADCak, FMC interfazearen bidez konektatuta, OpenCL prozesatze-nukleoetara igarotako seinale analogikoak bihurtzen ditu. OpenCL erabiliz, FPGA nukleoak ordenagailuarekin modu estandarizatuan komunikatzen dira. Metodologia horren aplikagarritasuna egiaztatzeko, erreferentziazko adibideak literaturatik hautatu eta nukleoen konexio-adibideak egiten ditu, edozein kasutan erabiltzeko, eta emaitzek nukleo multzoen adibideak erakusten dituzte. Neutroi-fluxuaren neurketak 1GS/s-ko ebazten dira real time errendimenduaz, pultsuak eta Campbell zenbatzeko teknikak erabiliz, Arria10 FPGAren % 40 erabiliz. Azkenik, DL aplikazioaren adibidea, neutroi/gamma sailkapena, sare neuronal batean oinarritzen da, eta FPGAren baliabideen % 25 baino gutxiago erabiliz 1.3 MSailkapen/s sailkapen-tasa lortzen du.

Field Programmable Gate Array (FPGA) devices are coming very strongly in the digital hardware systems due to the availability of ready to use resources, parallel logic operations and reconfigurable designs. The usage of FPGA systems in real time domain is also a very fruitful proposition as the FPGA devices are coming with processing cores for Real Time data processing. In a complex system scenario involving a large amount of processing tasks, there is a requirement of building the system using multiple FPGA devices. To make this possible we have to establish a real time data communication between the FPGA devices and to make it even better we have to apply data encryption techniques for making this communication secured.In this paper we demonstrate the design and implementation of a 32-bit RSA algorithms by developing suitable Hardware and Software design on Xilinx Spartan- 3E (XC3S500E-FG320) device, the implementation has been tested successfully for real time serial data communication between multiple FPGA devices using the RS232 serial interface. This development work is also useful for the embedded applications, which requires on board execution of security algorithms. The system is optimized in terms of execution speed and also has been verified using real time debugging tools.

Deep learning (DL) has shown explosive growth in its application to bioinformatics and has demonstrated thrillingly promising power to mine the complex relationship hidden in large-scale biological and biomedical data. A number of comprehensive reviews have been published on such applications, ranging from high-level reviews with future perspectives to those mainly serving as tutorials. These reviews have provided an excellent introduction to and guideline for applications of DL in bioinformatics, covering multiple types of machine learning (ML) problems, different DL architectures, and ranges of biological/biomedical problems. However, most of these reviews have focused on previous research, whereas current trends in the principled DL field and perspectives on their future developments and potential new applications to biology and biomedicine are still scarce. We will focus on modern DL, the ongoing trends and future directions of the principled DL field, and postulate new and major applications in bioinformatics.

Deep learning (also known as deep structured learning, hierarchical learning, or deep machine learning) is a branch of machine learning based on a set of algorithms that attempt to model high-level abstractions in data by using a deep graph with multiple processing layers, composed of multiple linear and nonlinear transformations. Deep learning has been characterized as a class of machine learning algorithms with the following characteristics [257]: • They use a cascade of many layers of nonlinear processing units for feature extraction and transformation. Each successive layer uses the output from the previous layer as input. The algorithms may be supervised or unsupervised and applications include pattern analysis (unsupervised) and classification (supervised). • They are based on the (unsupervised) learning of multiple levels of features or representations of the data. Higher-level features are derived from lower-level features to form a hierarchical representation. • They are part of the broader machine learning field of learning representations of data. • They learn multiple levels of representations that correspond to different levels of abstraction; the levels form a hierarchy of concepts. These definitions have in common: multiple layers of nonlinear processing units and the supervised or unsupervised learning of feature representations in each layer, with the layers forming a hierarchy from low-level to high-level features. Various deep learning architectures such as deep neural networks, convolutional deep neural networks, deep belief networks (DBN), and recurrent neural networks have been applied to fields like computer vision, automatic speech recognition, natural language processing, audio recognition, and bioinformatics where they have been shown to produce state-of-the-art results on various tasks. In this chapter we start in Section 7.1 with an introduction, giving a brief history of this field, the relevant literature, and its applications. Then we study some basic concepts of deep learning such as convolutional neural networks, recurrent neural networks, backpropagation algorithm, restricted Boltzmann machines, and deep learning networks in Section 7.2. Then we illustrate three examples for Apache Spark implementation for mobile big data (MBD), user moving pattern extraction, and combination with nonparametric Bayesian learning, respectively, in Sections 7.3 through 7.5. Finally, we have summary in Section 7.6.

Convolutional Neural Network (CNN) has achieved great success in various fields of machine vision, such as image classification and recognition, image segmentation, and video analysis. In specific applications, it is often necessary to customize the network structure by altering image size, convolution kernel size, pooling size, network architecture, and the number of network layers. These customizations pose significant challenges to the architecture of CNN accelerator especially in real-time systems, and as the number of these network variables increases, the internal convolution calculations become extremely demanding. The inference speed and energy consumption of CNN are becoming more and more important, which requires the design of the accelerator to adapt to different network architecture and run efficiently. Field-programmable gate arrays (FPGAs) is an ideal choice for a CNN accelerator due to its high programmability and low power consumption. To address the aforementioned challenges, we present NRUCA, a novel network reconfigurable universal CNN accelerator based on FPGA. On one hand, we have designed a flexible architecture that can dynamically configure CNN’s network structure parameters. These parameters can be sent to the FPGA via a configuration file, enabling the same design to run different CNN networks and allowing to modify on the fly. This forms the basis of the architecture’s adaptability presented in this paper. On the other hand, we employ a multi-channel parallel and efficient pipeline matrix multiplication architecture to implement the convolution and fully-connected layers, which constitute the majority of the computational load in CNNs. By utilizing an innovative inverted matrix multiplication algorithm and an interleaving cache data method, we reduce the internal cache required for multi-channel data. Furthermore, most of the intermediate calculation data does not need to be output to DDR, significantly improving the operation efficiency of the accelerator. We also fully leverage the FPGA chip architecture to compile multiple calculation kernels, which can be flexibly scheduled and combined to cater to various application scenarios. Based on the flexible and efficient overall design architecture, the layout of various resources of FPGA is balanced, which enables the designed FPGA project to compile four acceleration kernels with the running clock up to 300MHz in Xilinx Vitis and test them on Xilinx Alveo U250. We verified the adaptability and acceleration capabilities of our FPGA project using LeNet, AlexNet, VGG11, VGG13, VGG16, and VGG19 networks. Experiments demonstrated that our FPGA architecture achieves 33X and 35X energy savings compared to the Intel Xeon 5220R CPU, and 1.08X and 1.05X energy savings compared to the Nvidia Tesla P100 GPU, when accelerating AlexNet and VGG16, respectively.

Mobile devices are becoming an important carrier for deep learning tasks, as they are being equipped with powerful, high-end mobile CPUs and GPUs. However, it is still a challenging task to execute 3D Convolutional Neural Networks (CNNs) targeting for real-time performance, besides high inference accuracy. The reason is more complex model structure and higher model dimensionality overwhelm the available computation/storage resources on mobile devices. A natural way may be turning to deep learning weight pruning techniques. However, the direct generalization of existing 2D CNN weight pruning methods to 3D CNNs is not ideal for fully exploiting mobile parallelism while achieving high inference accuracy. This paper proposes RT3D, a model compression and mobile acceleration framework for 3D CNNs, seamlessly integrating neural network weight pruning and compiler code generation techniques. We propose and investigate two structured sparsity schemes i.e., the vanilla structured sparsity and kernel group structured (KGS) sparsity that are mobile acceleration friendly. The vanilla sparsity removes whole kernel groups, while KGS sparsity is a more fine-grained structured sparsity that enjoys higher flexibility while exploiting full on-device parallelism. We propose a reweighted regularization pruning algorithm to achieve the proposed sparsity schemes. The inference time speedup due to sparsity is approaching the pruning rate of the whole model FLOPs (floating point operations). RT3D demonstrates up to 29.1x speedup in end-to-end inference time comparing with current mobile frameworks supporting 3D CNNs, with moderate 1%~1.5% accuracy loss. The end-to-end inference time for 16 video frames could be within 150 ms, when executing representative C3D and R(2+1)D models on a cellphone. For the first time, real-time execution of 3D CNNs is achieved on off-the-shelf mobiles.

This article presents a critical literature review on the security aspects of field-programmable gate array (FPGA) devices. FPGA devices present unique challenges to cybersecurity through their reconfigurable nature. The article also pays special attention to emerging system-on-chip (SoC) FPGA devices that incorporate a hard processing system (HPS) on the same die as the FPGA logic. While this incorporation reduces the need for vulnerable external signals, the HPS in SoC FPGA devices adds a level of complexity that is not present for stand-alone FPGA devices. This added complexity necessarily hands over the task of securing the device to developers. Even with standard security features in place, the HPS might still have unhindered access to the FPGA logic. A single software flaw could open up a breach that might allow an attacker to extract the FPGA’s configuration data. A robust cybersecurity strategy is thus required for developers. As such, this work aims to provide the groundwork to build a solid threat-based cybersecurity design strategy that is specially adapted to SoC FPGA devices.

Field-programmable Gate Array (FPGA) is a high-performance computing platform for Convolution Neural Networks (CNNs) inference. Winograd algorithm, weight pruning, and quantization are widely adopted to reduce the storage and arithmetic overhead of CNNs on FPGAs. Recent studies strive to prune the weights in the Winograd domain, however, resulting in irregular sparse patterns and leading to low parallelism and reduced utilization of resources. Besides, there are few works to discuss a suitable quantization scheme for Winograd. In this article, we propose a regular sparse pruning pattern in the Winograd-based CNN, namely, Sub-row-balanced Sparsity (SRBS) pattern, to overcome the challenge of the irregular sparse pattern. Then, we develop a two-step hardware co-optimization approach to improve the model accuracy using the SRBS pattern. Based on the pruned model, we implement a mixed precision quantization to further reduce the computational complexity of bit operations. Finally, we design an FPGA accelerator that takes both the advantage of the SRBS pattern to eliminate low-parallelism computation and the irregular memory accesses, as well as the mixed precision quantization to get a layer-wise bit width. Experimental results on VGG16/VGG-nagadomi with CIFAR-10 and ResNet-18/34/50 with ImageNet show up to 11.8×/8.67× and 8.17×/8.31×/10.6× speedup, 12.74×/9.19× and 8.75×/8.81×/11.1× energy efficiency improvement, respectively, compared with the state-of-the-art dense Winograd accelerator [20] with negligible loss of model accuracy. We also show that our design has 4.11× speedup compared with the state-of-the-art sparse Winograd accelerator [19] on VGG16.

With the leaps and bounds of computer performance and the advent of the era of big data, deep learning has drawn more and more attention from all walks of life. It can combine low-level features to form more abstract high-level features and describe the data more essentially. Therefore, it is widely used in various fields such as computer vision. Image segmentation is one of the most basic research topics in the field of computer vision. The main purpose is to extract regions of interest from images for later image processing research. In 3D reconstruction based on sequence images, the segmentation accuracy and speed of sequence images determine the quality and efficiency of target reconstruction. Therefore, when facing large-scale sequence images, the biggest problem is how to improve the segmentation speed while ensuring accuracy. Based on the above background, the research content of this article is an indoor scene segmentation algorithm based on full convolutional neural network. According to the characteristics of indoor application scenes, this paper proposes a fast convolutional neural network image segmentation method to segment the indoor scene image and construct the fast fully convolutional networks (FFCN) for indoor scene image segmentation uses inter-layer fusion to reduce the amount of network calculation parameters and avoid the loss of picture feature information by continuous convolution. In order to verify the effectiveness of the network, in this paper, a basic living object data set (XAUT data set) in an indoor environment is created. The XAUT data set is used to train the FFCN network under the Caffe framework to obtain an indoor scene segmentation model. In order to compare the effectiveness of the model, the structure of the worn FCN8s, FCN16s, and FCN32s models was fine-tuned, and the corresponding algorithm model for indoor scene segmentation was obtained by training with the XAUT data set. The experimental results show that the pixel recognition accuracy of all types of networks has reached 86%, and the mean IU ratio has reached more than 63%. The mean IU of the FCN8s network is the highest at 70.38%, but its segmentation speed is only 1/5 of FFCN. On the premise that other types of indicators are not much different, the average segmentation speed on FFCN fast segmentation convolutional neural network reaches 40 fps. It can be seen that the scale fusion technology can well avoid the loss of image feature information in the network convolution and reddening process. Compared with other FCN networks, it has a faster speed and is conducive to real-time image preprocessing.

Recent years have seen a meteoric rise in the usage of enormous image databases due to advancements in multimedia technologies. One of the most critical technologies for image processing nowadays is image retrieval. This study uses convolutional neural networks (CNNs) for content-based image retrieval (CBIR). With the ever-growing number of digital photos, practical methods for retrieving these images are crucial. CNNs are incredibly efficient in many computer vision applications. Improving the efficacy and precision of image retrieval systems is the primary goal of our research into using deep learning. The paper starts with a thorough analysis of the current state of CBIR methods and the difficulties they face. Afterwards, it explores CNN’s design and operation, focusing on CNN’s capacity to learn hierarchical features from images autonomously. This paper also looks at how the model performs when it alters its hyperparameters, transfer learning techniques, and CNN topologies. The insights obtained from these experiments enhance the comprehension of the elements impacting CNN effectiveness in CBIR. Finally, our study shows that CNNs can change the game for image search by transforming CBIR systems. This research adds to the expanding body of information about using cutting-edge deep learning algorithms to make image retrieval more efficient and accurate.

Convolutional neural networks (CNNs) are classical models for computer vision and machine learning applications such as video surveillance, pattern recognition, weather forecasting, traffic, and safety. CNNs involve computationally intensive operations and require huge off-chip memory bandwidth, which makes it a challenging task to deploy on real-time embedded systems. Compared to central processing units and graphic processing units, field programmable gate arrays (FPGA)-based CNNs are gaining popularity owing to their flexibility and efficiency. In this work, we present an efficient CNN accelerator based on blocked Winograd-GEMM architecture with high performance. We implement ResNet-18 CNN model on XC7VX690T FPGA using proposed architecture. This implementation operates at a clock frequency of 200 MHz and gives average throughput of 383 GOPS which is comparable to other state-of-art implementations. This manuscript is an extended version of [S. Kala, J. Mathew, B. R. Jose, and S. Nalesh, “UniWiG: Unified Winograd-GEMM Architecture for Accelerating CNN on FPGAs,” in 2019 32nd International Conference on VLSI Design and 2019 18th International Conference on Embedded Systems (VLSID), Delhi, NCR, India, 2019, pp. 209–214. DOI: 10.1109/VLSID.2019.00055.].

Rolling bearings are often exposed to high speeds and pressures, leading to the symmetry in their rotating structure being disrupted, which can lead to serious failures. Intelligent rolling bearing fault diagnosis is a critical part of ensuring operation of machinery, and it has been facilitated by the growing popularity of convolutional neural networks (CNNs). The outstanding performance of fault diagnosis CNNs results from complex and redundant network structures and parameters, resulting in huge storage and computational requirements, which makes it challenging to implement these models in resource-limited industrial devices. This study aims to address this problem by proposing a comprehensive compression method for CNNs that is applied to intelligent fault diagnosis. It involves several different compression methods, including tensor train decomposition, parameter quantization, and knowledge distillation for deep network compression. This results in a significant decrease in redundancy and speeding up the training of CNN models. Firstly, tensor train decomposition is applied to reduce redundant connections in both convolutional and fully connected layers. The next step is to perform parameter quantization to minimize the bits needed for parameter representation and storage. Finally, knowledge distillation is used to restore accuracy to the compressed model. The effectiveness of the proposed approach is confirmed by an experiment and ablation study with different models on several datasets. The results show that it can significantly reduce redundant information and floating-point operations with little degradation in accuracy. Notably, on the CWRU dataset, with about 60% parameter reduction, there is no degradation in our model’s accuracy. The proposed approach is a new attempt at the intelligent fault diagnosis of rolling bearings in industrial equipment.