Arithmetic Coding-Based 5-Bit Weight Encoding and Hardware Decoder for CNN Inference in Edge Devices

Advances in deep learning, especially Convolutional Neural Networks (CNNs) have revolutionized intelligent frame-works such as Human Activity Recognition (HAR) systems by effectively and efficiently inferring human activity from various modalities of data. However, the training and inference of CNNs are often resource-intensive. Recent research developments are focused on bringing the effectiveness of CNNs in resource con-strained edge devices through Tiny Machine Learning (TinyML). However, this is extremely hard to achieve due to the limitations in memory, compute power, and energy of resource constrained edge devices. This paper provides a benchmark to understand these trade-offs among variations of CNN network architectures, different training methodologies, and different modalities of data in the context of HAR, TinyML, and edge devices. We tested and reported the performance of CNN and Depthwise Separable CNN (DSCNN) models as well as two training methodologies: Quantization Aware Training (QAT) and Post-training Quantization (PTQ) on five commonly used benchmark datasets containing image and time-series data: UP-Fall, Fall Detection Dataset (FDD), PAMAP2, UCI-HAR, and WISDM. We also deployed and tested the performance of the model-based standalone applications on multiple commonly available resource constrained edge devices in terms of inference time and power consumption. The experimental results demonstrate the effectiveness and feasibility of Tiny ML for HAR in edge devices.

For model inference of convolutional neural networks (CNNs), we nowadays witness a shift from the Cloud to the Edge. Unfortunately, deploying and inferring large, compute- and memory-intensive CNNs on Internet of Things devices at the Edge is challenging as they typically have limited resources. One approach to address this challenge is to leverage all available resources across multiple edge devices to execute a large CNN by properly partitioning it and running each CNN partition on a separate edge device. However, there currently does not exist a design and programming framework that takes a trained CNN model as input and subsequently allows for efficiently exploring and automatically implementing a range of different CNN partitions on multiple edge devices to facilitate distributed CNN inference. Therefore, in this article, we propose a novel framework that automates the splitting of a CNN model into a set of submodels as well as the code generation needed for the distributed and collaborative execution of these submodels on multiple, possibly heterogeneous, edge devices, while supporting the exploitation of parallelism among and within the edge devices. In addition, since the number of different CNN mapping possibilities on multiple edge devices is vast, our framework also features a multistage and hierarchical design space exploration methodology to efficiently search for (near-)optimal distributed CNN inference implementations. Our experimental results demonstrate that our work allows for rapidly finding and realizing distributed CNN inference implementations with reduced energy consumption and memory usage per edge device, and under certain conditions, with improved system throughput as well.

Edge AI systems are increasingly being adopted in a wide range of application domains. These systems typically deploy Convolutional Neural Network (CNN) models on edge devices to perform inference, while relying on the cloud for model training. This is due to the high computational and memory demands of conventional model training, which exceeds the capabilities of resource-constrained edge devices running on tight power budgets. The dependency on the cloud for training is not suitable in many applications, where new objects or environmental conditions different from the ones present during training, are frequently encountered. In such applications, continual learning of new knowledge on the edge device becomes a necessity to avoid performance bottlenecks due to round-trip communication delays, network connectivity, and the available bandwidth. In this thesis, we propose Field-Programmable Gate Array (FPGA) based accelerator architecture and optimization strategies for a new paradigm of machine learning algorithms that is capable of continual learning. The proposed methods will enable edge FPGA systems to perform on-device deep continual learning for object classification. Specifically, the proposed methods aim to achieve real-time learning on-device, while providing for a high degree of scalability to learn a large number of classes. We first propose a FPGA accelerator for a Self-Organizing Neural Network (SONN), that can perform class-incremental continual learning in a streaming manner when combined with a CNN. The SONN model performs unsupervised learning from embedding features extracted from the CNN model by dynamically growing neurons and connections. We introduce design optimization strategies and runtime scheduling techniques to optimize resource usage, latency, and energy consumption. Experimental results based on Core50 dataset for continuous object recognition from video sequences, demonstrated that the proposed FPGA architecture outperforms CPU and GPU based counterparts in terms of latency and power. However, the SONN model grows proportionally to the classes learnt, which limits its scalability to learn a large number of classes efficiently. Next, we propose a FPGA accelerator for a Streaming Linear Discriminant Analysis (SLDA) model to overcome the scalability limitations of SONN. Similar to the SONN, SLDA performs continual learning from embedding features extracted from a CNN in a streaming manner. SLDA is highly scalable for learning a large number of classes as the network does not grow dynamically to accommodate new knowledge. We propose several design and runtime optimizations to minimize resource usage, latency, and energy consumption. Additionally, we introduce a new variant of SLDA and discuss the accuracy-efficiency trade-offs using popular datasets for continual learning, CoRE50 and CUB200. The results demonstrate that the proposed SLDA accelerator outperforms CPU and GPU counterparts, in terms of latency and energy efficiency. Finally, we demonstrate a full on-chip deep continual learning pipeline on FPGA, by integrating the proposed SLDA accelerator with Xilinx DPU, a programmable CNN accelerator IP. The design is implemented on a Xilinx Zynq Ultrascale+ MPSoC. In order to overcome the large performance bottleneck due to the communication overhead between the ARM processing system (PS) and programmable logic (PL), we implemented a Linux device driver that facilitates efficient memory mapping between PS-PL. The experimental results based on CoRE50 dataset show that the proposed pipeline is capable of performing continual learning at nearly the same latency as the inference pipeline, with only a marginal increase in energy consumption. Our results clearly demonstrate the viability of deploying real-time deep continual learning on edge AI systems that are equipped with FPGA accelerators.

The performance of convolutional neural networks (CNNs) in image classification and object detection has been remarkable, even though they contain millions and billions of parameters. This over‐parameterization of CNN makes them both memory‐intensive and computationally complex and exhaustive. This greatly hinders the application of CNNs in resource‐constrained environments such as Internet of things (IoT) and edge devices. This poses a critical challenge for CNNs in deploying these powerful computer vision tools to mobile devices, which needs immediate attention. In this study, we have proposed a novel technique based on non‐convex optimization, max‐norm regularization. The max‐norm will structurally prune the number of parameters without compromising the model's performance. The proximal gradient descent algorithm is used for network optimization while using this non‐convex regularizer. The max‐norm is combined with the channel pruning to achieve more sparse CNN networks. Later, the pruned network can be easily deployed in the resource‐constrained application environment. The proposed technique is tested on several benchmark datasets for validation. In addition, in this study, the sparsified CNNs are used for biomedical image analysis using the BRAIN MRI dataset. This sparsely trained CNN model can later serve as the best lightweight model applicable in the IoT healthcare sector for detecting and classifying three types of brain tumours, one of the most life‐threatening diseases whose early detection can save the costly lives of human beings. This is the first paper to propose the novel max‐norm regularizer to enforce sparse learning through CNNs. The paper provides a detailed analysis of convex and non‐convex regularizers before presenting the proposed novel max‐norm regularizer. Finally, the paper compares the proposed max‐norm regularizer with existing regularization methods using state‐of‐the‐art CNN models.

The past several years have seen both an explosion in the use of Convolutional Neural Networks (CNNs) and accelerators to make CNN inference practical. In the architecture community, the lion share of effort has targeted CNN inference for image recognition. The closely related problem of video recognition has received far less attention as an accelerator target. This is surprising, as video recognition is more computationally intensive than image recognition, and video traffic is predicted to be the majority of internet traffic in the coming years. This paper fills the gap between algorithmic and hardware advances for video recognition by providing a design space exploration and flexible architecture for accelerating 3D Convolutional Neural Networks (3D CNNs)—the core kernel in modern video understanding. When compared to (2D) CNNs used for image recognition, efficiently accelerating 3D CNNs poses a significant engineering challenge due to their large (and variable over time) memory footprint and higher dimensionality. To address these challenges, we design a novel accelerator called Morph, that can adaptively support different spatial and temporal tiling strategies depending on the needs of each layer of each target 3D CNN. We codesign a software infrastructure alongside the Morph hardware to find good-fit parameters to control the hardware. Evaluated on state-of-the-art 3D CNNs, Morph achieves up to 2.7× (1.9× average) reduction in energy consumption and improves performance/watt up to 4.4× (3× average) compared to a baseline 3D CNN accelerator, with an area overhead of 2%. Morph further achieves a 11.6× average energy reduction on 3D CNNs when compared to Eyeriss, a popular 2D CNN accelerator, while reducing efficiency compared to Eyeriss on a 2D CNN by 71%.

Edge devices are becoming smarter with the integration of machine learning methods, such as deep learning, and are therefore used in many application domains where decisions have to be made without human intervention. Deep learning and, in particular, convolutional neural networks (CNN) are more efficient than previous algorithms for several computer vision applications such as security and surveillance, where image and video analysis are required. This better efficiency comes with a cost of high computation and memory requirements. Hence, running CNNs in embedded computing devices is a challenge for both algorithm and hardware designers. New processing devices, dedicated system architectures and optimization of the networks have been researched to deal with these computation requirements. In this paper, we improve the inference execution times of CNNs in low density FPGAs (Field-Programmable Gate Arrays) using fixed-point arithmetic, zero-skipping and weight pruning. The developed architecture supports the execution of large CNNs in FPGA devices with reduced on-chip memory and computing resources. With the proposed architecture, it is possible to infer an image in AlexNet in 2.9 ms in a ZYNQ7020 and 1.0 ms in a ZYNQ7045 with less than 1% accuracy degradation. These results improve previous state-of-the-art architectures for CNN inference.

Among the many deep learning methods, the convolutional neural network (CNN) model has an excellent performance in image recognition. Research on identifying and classifying image datasets using CNN is ongoing. Animal species recognition and classification with CNN is expected to be helpful for various applications. However, sophisticated feature recognition is essential to classify quasi-species with similar features, such as the quasi-species of parrots that have a high color similarity. The purpose of this study is to develop a vision-based mobile application to classify endangered parrot species using an advanced CNN model based on transfer learning (some parrots have quite similar colors and shapes). We acquired the images in two ways: collecting them directly from the Seoul Grand Park Zoo and crawling them using the Google search. Subsequently, we have built advanced CNN models with transfer learning and trained them using the data. Next, we converted one of the fully trained models into a file for execution on mobile devices and created the Android package files. The accuracy was measured for each of the eight CNN models. The overall accuracy for the camera of the mobile device was 94.125%. For certain species, the accuracy of recognition was 100%, with the required time of only 455 ms. Our approach helps to recognize the species in real time using the camera of the mobile device. Applications will be helpful for the prevention of smuggling of endangered species in the customs clearance area.

This paper compares three different optimization approaches for accelerating the inference of convolutional neural networks (CNNs). We compare the techniques of separable convolution, weight pruning, and binarization. Each method is implemented and empirically compared in three aspects: preservation of accuracy, storage requirements, and achieved speed-up. Experiments are performed both on a desktop computer and on a mobile platform using a CNN model for vehicle type classification. Our experiments show that the largest speed-up is achieved by binarization, whereas pruning achieves the largest reduction in storage requirements. Both of these approaches largely preserve the accuracy of the original network.

Identification of crops is an important topic in the agricultural domain. Hyperspectral remote sensing data are very useful for crop feature extraction and classification. Remote sensing data is an unstructured data and Convolutional Neural Network (CNN) can work on unstructured data efficiently. This paper presents an evaluation of CNN for crop classification using the Indian Pines standard dataset obtained from the AVIRIS sensor and the study area dataset obtained from the EO-1hyperion sensor. Optimized CNN has been tuned by training the model on different parameters. It has been compared with two classification algorithms: Deep Neural Network (DNN) and Convolutional Autoencoder. According to the test results, the proposed optimized CNN model provided better results as compared to the other two methods. CNN has given 97 ± 0.58% overall accuracy for the Indian Pines standard dataset and 78 ± 2.43% for our study area dataset.

Understanding the learning mechanism of convolutional neural networks in spectral analysis

Accurately identifying epileptogenic zone (EZ) using high-frequency oscillations (HFOs) is a challenge that must be mastered to transfer HFOs into clinical use. We analyzed the ability of a convolutional neural network (CNN) model to distinguish EZ and non-EZ HFOs. Nineteen medically intractable epilepsy patients with good surgical outcomes 2 years after surgery were studied. Five-minute interictal intracranial electroencephalogram epochs of slow-wave sleep were selected randomly. Then 5 s segments of ripples (80–200 Hz) and fast ripples (FRs, 200–500 Hz) were detected automatically. The EZs and non-EZs were identified using the surgery resection range. We innovatively converted all epochs into four types of images using two scales: original waveforms, filtered waveforms, wavelet spectrum images, and smoothed pseudo Wigner–Ville distribution (SPWVD) spectrum images. Two scales were fixed and fitted scales. We then used a CNN model to classify the HFOs into EZ and non-EZ categories. As a result, 7,000 epochs of ripples and 2,000 epochs of FRs were randomly selected from the EZ and non-EZ data for analysis. Our CNN model can distinguish EZ and non-EZ HFOs successfully. Except for original ripple waveforms, the results from CNN models that are trained using fixed-scale images are significantly better than those from models trained using fitted-scale images (p < 0.05). Of the four fixed-scale transformations, the CNN based on the adjusted SPWVD (ASPWVD) produced the best accuracies (80.89 ± 1.43% and 77.85 ± 1.61% for ripples and FRs, respectively, p < 0.05). The CNN using ASPWVD transformation images is an effective deep learning method that can be used to classify EZ and non-EZ HFOs.

Machine learning (ML) within the edge internet of things (IoT) is instrumental in making significant shifts in various industrial domains, including smart farming. To increase the efficiency of farming operations and ensure ML accessibility for both small and large-scale farming, the need for a low-cost ML-enabled framework is more pressing. In this paper, we present an end-to-end solution that utilizes tiny ML (TinyML) for the low-cost adoption of ML in classification tasks with a focus on the post-harvest process of olive fruits. We performed dataset collection to build a dataset that consists of several varieties of olive fruits, with the aim of automating the classification and sorting of these fruits. We employed simple image segmentation techniques by means of morphological segmentation to create a dataset that consists of more than 16,500 individually labeled fruits. Then, a convolutional neural network (CNN) was trained on this dataset to classify the quality and category of the fruits, thereby enhancing the efficiency of the olive post-harvesting process. The goal of this study is to show the feasibility of compressing ML models into low-cost edge devices with computationally constrained settings for tasks like olive fruit classification. The trained CNN was efficiently compressed to fit into a low-cost edge controller, maintaining a small model size suitable for edge computing. The performance of this CNN model on the edge device, focusing on metrics like inference time and memory requirements, demonstrated its feasibility with an accuracy of classification of more than 97.0% and minimal edge inference delays ranging from 6 to 55 inferences per second. In summary, the results of this study present a framework that is feasible and efficient for compressing CNN models on edge devices, which can be utilized and expanded in many agricultural applications and also show the practical insights for implementing the used CNN architectures into edge IoT devices and show the trade-offs for employing them using TinyML.

Convolutional Neural Networks (CNNs) have demonstrated impressive performance in complex machine learning tasks such as classification and regression problems. A reliable neural network structure plays a decisive role in CNN studies. Through comparing and analyzing the structure of neural networks, a model structure for better visualization performance has been discovered, and such a method supports the development of deep learning research. These studies are of particular importance in end-to-end systems for autonomous driving to imitate human driving, where the interpretability of the system is limited. Because of the uncertainty of the ground truth, for the determination of human steering in an image, it is difficult to accurately compare the visualization performance of different CNN models or different visualization methods. For practical applications, however, an objective and quantitative measure for assessing visualization performance is necessary. Therefore, a method to evaluate the visualization performance of CNN models using a driver model instead of human drivers is proposed, to generate a data set which can be used to determine the decisional point (ground truth) in the input image. Then, an exclusive method is also put forth, to quantitatively calculate the relationship between the decisional point (ground truth) and the visualization results produced by CNN models. In this paper, five CNN models as an autonomous steering controller are designed based on PilotNet, and the visualization abilities of each CNN models is compared by three evaluation indicators. By comparing the visualization performance of five different CNN models, it is shown that the proposed method can successfully assess the visualization level of the CNN model.

Model optimization is a problem of great concern and challenge for developing an image classification model. In image classification, selecting the appropriate hyperparameters can substantially boost the model’s ability to learn intricate patterns and features from complex image data. Hyperparameter optimization helps to prevent overfitting by finding the right balance between complexity and generalization of a model. The ensemble genetic algorithm and convolutional neural network (EGACNN) are proposed to enhance image classification by fine-tuning hyperparameters. The convolutional neural network (CNN) model is combined with a genetic algorithm GA) using stacking based on the Modified National Institute of Standards and Technology (MNIST) dataset to enhance efficiency and prediction rate on image classification. The GA optimizes the number of layers, kernel size, learning rates, dropout rates, and batch sizes of the CNN model to improve the accuracy and performance of the model. The objective of this research is to improve the CNN-based image classification system by utilizing the advantages of ensemble learning and GA. The highest accuracy is obtained using the proposed EGACNN model which is 99.91% and the ensemble CNN and spiking neural network (CSNN) model shows an accuracy of 99.68%. The ensemble approaches like EGACNN and CSNN tends to be more effective as compared to CNN, RNN, AlexNet, ResNet, and VGG models. The hyperparameter optimization of deep learning classification models reduces human efforts and produces better prediction results. Performance comparison with existing approaches also shows the superior performance of the proposed model.

Convolutional Neural Networks (CNNs) have achieved significant success in image classification and object detection. CNN models generally consist of a single-stream and process single image data at once. In addition, multi-stream (or multi-modal) models have recently begun to be proposed that allow the processing of more than one input at the same time. The data can be an image, video, voice, or any other sensor data. Multi-modality may help us extract some hidden features of the same object. Furthermore, several new studies examine sharing feature maps between different streams of the same CNN. However, systematic studies that can adequately demonstrate the contribution of multi-modality and feature map sharing features to performance have not yet been conducted. Processing power and lack of available datasets are among the important factors that negatively affect progress. In this study, the contributions of multi-modality and feature map sharing (FMS) to increase the performance in object recognition are examined in detail. For this purpose, a new dataset and a new multi-modal multi-feature map sharing CNN model, which we call FMSNet, are developed. The proposed model achieved a 3.06% higher accuracy rate than its non-FMS counterpart, DenseNet-201, exceeding most of the state-of-the-art single-stream CNN models.