Lightweight Compression Of Neural Network Feature Tensors For Collaborative Intelligence

Classification of Bisyllabic Lexical Stress Patterns Using Deep Neural Networks

Due to the advent of modern embedded systems and mobile devices with constrained resources, there is a great demand for incredibly efficient deep neural networks for machine learning purposes. There is also a growing concern of privacy and confidentiality of user data within the general public when their data is processed and stored in an external server which has further fueled the need for developing such efficient neural networks for real-time inference on local embedded systems. The scope of our work presented in this paper is limited to image classification using a convolutional neural network. A Convolutional Neural Network (CNN) is a class of Deep Neural Network (DNN) widely used in the analysis of visual images captured by an image sensor, designed to extract information and convert it into meaningful representations for real-time inference of the input data. In this paper, we propose a neoteric variant of deep convolutional neural network architecture to ameliorate the performance of existing CNN architectures for real-time inference on embedded systems. We show that this architecture, dubbed CondenseNeXt, is remarkably efficient in comparison to the baseline neural network architecture, CondenseNet, by reducing trainable parameters and FLOPs required to train the network whilst maintaining a balance between the trained model size of less than 3.0 MB and accuracy trade-off resulting in an unprecedented computational efficiency.

The cloud-only and edge-computing approaches have recently been proposed to satisfy the requirements of complex neural networks. However, the cloud-only approach generates a latency challenge because of the high data volumes that must be sent to a centralized location in the cloud. Less-powerful edge computing resources require a compression model for computation reduction, which degrades the model trading accuracy. To address this challenge, deep neural network (DNN) partitioning has become a recent trend, with DNN models being sliced into head and tail portions executed at the mobile edge devices and cloud server, respectively. We propose Edgepipe, a novel partitioning method based on pipeline inference with an image stream to automatically partition DNN computation between the edge device and cloud server, thereby reducing the global latency and enhancing the system-wide real-time performance. This method adapts to various DNN architectures, hardware platforms, and networks. Here, when evaluated on a suite of five DNN applications, Edgepipe achieves average latency speedups of 1.241× and 1.154× over the cloud-only approach and the state-of-the-art approach known as “Neurosurgeon”, respectively.

Heartbeat classification is a crucial step for arrhythmia diagnosis during electrocardiographic (ECG) analysis. The new scenario of wireless body sensor network- (WBSN-) enabled ECG monitoring puts forward a higher-level demand for this traditional ECG analysis task. Previously reported methods mainly addressed this requirement with the applications of a shallow structured classifier and expert-designed features. In this study, modified frequency slice wavelet transform (MFSWT) was firstly employed to produce the time-frequency image for heartbeat signal. Then the deep learning (DL) method was performed for the heartbeat classification. Here, we proposed a novel model incorporating automatic feature abstraction and a deep neural network (DNN) classifier. Features were automatically abstracted by the stacked denoising auto-encoder (SDA) from the transferred time-frequency image. DNN classifier was constructed by an encoder layer of SDA and a softmax layer. In addition, a deterministic patient-specific heartbeat classifier was achieved by fine-tuning on heartbeat samples, which included a small subset of individual samples. The performance of the proposed model was evaluated on the MIT-BIH arrhythmia database. Results showed that an overall accuracy of 97.5% was achieved using the proposed model, confirming that the proposed DNN model is a powerful tool for heartbeat pattern recognition.

Deep Neural Networks (DNNs) have been successfully applied to various real-world machine learning applications. However, performing large DNN inference tasks in real-time remains a challenge due to its substantial computational costs. Recently, Spiking Neural Networks (SNNs) have emerged as an alternative way of processing DNN'fs task. Due to its eventbased, data-driven computation, SNN reduces both inference latency and complexity. With efficient conversion methods from traditional DNN, SNN exhibits similar accuracy, while leveraging many state-of-the-art network models and training methods. In this work, an efficient neuromorphic hardware architecture for image recognition task is presented. To preserve accuracy, the analog-to-spiking conversion algorithm is adopted. The system aims to minimize hardware area cost and power consumption, enabling neuromorphic hardware processing in edge devices. Simulation results have shown that, with the MNIST digit recognition task, the system has achieved $\times 20$ reduction in terms of core area cost compared to the state-of-the-art works, with an accuracy of 94.4%, core area of 15 $\mu m^{2}$ at a maximum frequency of 250 MHz.

Machine learning has surged in popularity in recent years, with deep neural networks (DNNs) becoming a cornerstone of applications such as autonomous driving, augmented reality, and natural language processing, owing to their remarkable accuracy and scalability. As the demand for greater portability and flexibility grows, there is a strong push to optimize these applications for resource-constrained edge devices like smartphones and IoT devices.However, deploying DNN models on such devices presents significant challenges, primarily due to the large size of these models, which is a key contributor to their high performance. For instance, the DeepSpeech 2 model outperforms its predecessor, DeepSpeech 1, but its size is eight times larger, encompassing approximately 68 million parameters. Similarly, other networks designed for various tasks, including convolutional neural networks like VGG and transformer-based networks like BERT and GPT, also exhibit substantial sizes. For example, the GPT-3 model, a predecessor to ChatGPT, consists of an astonishing 175 billion parameters. Consequently, fitting DNN models onto resource-limited edge devices is a formidable challenge. My thesis addresses these challenges with a three-part approach: * High-Performance DNN Accelerators: I will develop DNN accelerators that excel in high performance, low latency, and energy efficiency through an algorithm and hardware co-design across diverse FPGA architectures (Chapter 2). * Real-Time DNN Execution: I aim to achieve exceptional accuracy and real-time execution for various large-scale DNNs on off-the-shelf mobile devices, IoT devices, and other platforms (Chapter 3). * CompVQC Framework: I will explore a systematic framework, CompVQC, to reduce the circuit length of quantum neural networks (QNNs) through model compression, which can achieve up to a 20% improvement in accuracy. This reveals that compilation-aware compression can enhance the robustness of QNNs on near-term noisy quantum devices (Chapter 4). The field of AI has recently undergone a significant transformation, ushering in the era of generative AI, where models like ChatGPT and Stable Diffusion create realistic and creative content for a wide range of applications. As generative AI becomes more prevalent, it is crucial to adapt algorithm-hardware co-design strategies to meet the specific needs of these models, particularly on edge devices. In the final chapter (Chapter 5), I will introduce my ongoing and future work focused on activating generative AI on the edge.--Author's abstract

Automated image classification is an essential task of the computer vision field. The tagging of images into a set of predefined groups is referred to as image classification. The implementation of computer vision to automate image classification would be beneficial because manual image evaluation and identification can be time-consuming, particularly when there are many images of different classes. Deep learning approaches are proven to overperform existing machine learning techniques in many fields in recent years, and computer vision is one of the most notable examples. The very deep neural network (VDCNN) is a powerful deep learning model for image classification, and this paper examines it briefly using MNIST handwritten digit dataset. This dataset is used to prove the efficacy of very deep neural networks over other deep learning models. The proposed study aims to comprehend the very deep neural network architecture used to accomplish a handwritten digit recognition task. The feasibility of the proposed model is evaluated using mean accuracy, validation accuracy, and standard deviation. The study results of the very deep neural network model are compared to a convolutional neural network and convolutional neural network with batch normalization. According to the results of the comparison study, very deep neural networks achieve high accuracy of 99.1% for a handwritten dataset. The outcome of the proposed work is used to interpret how well a very deep neural network performs when compared to the other two models of deep neural networks. This proposed architecture may be used to automate the classification of handwritten digits dataset.

Deep neural networks for texture classification—A theoretical analysis

We investigate the use of Deep Neural Networks for the classification of image datasets where texture features are important for generating class-conditional discriminative representations. To this end, we first derive the size of the feature space for some standard textural features extracted from the input dataset and then use the theory of Vapnik-Chervonenkis dimension to show that hand-crafted feature extraction creates low-dimensional representations which help in reducing the overall excess error rate. As a corollary to this analysis, we derive for the first time upper bounds on the VC dimension of Convolutional Neural Network as well as Dropout and Dropconnect networks and the relation between excess error rate of Dropout and Dropconnect networks. The concept of intrinsic dimension is used to validate the intuition that texture-based datasets are inherently higher dimensional as compared to handwritten digits or other object recognition datasets and hence more difficult to be shattered by neural networks. We then derive the mean distance from the centroid to the nearest and farthest sampling points in an n-dimensional manifold and show that the Relative Contrast of the sample data vanishes as dimensionality of the underlying vector space tends to infinity.

Video hyperlinking represents a classical example of multimodal problems. Common approaches to such problems are early fusion of the initial modalities and crossmodal translation from one modality to the other. Recently, deep neural networks, especially deep autoencoders, have proven promising both for crossmodal translation and for early fusion via multimodal embedding. A particular architecture, bidirectional symmetrical deep neural networks, have been proven to yield improved multimodal embeddings over classical autoencoders, while also being able to perform crossmodal translation. In this work, we focus firstly at evaluating good single-modal continuous representations both for textual and for visual information. Word2Vec and paragraph vectors are evaluated for representing collections of words, such as parts of automatic transcripts and multiple visual concepts, while different deep convolutional neural networks are evaluated for directly embedding visual information, avoiding the creation of visual concepts. Secondly, we evaluate methods for multimodal fusion and crossmodal translation, with different single-modal pairs, in the task of video hyperlinking. Bidirectional (symmetrical) deep neural networks were shown to successfully tackle downsides of multimodal autoencoders and yield a superior multimodal representation. In this work, we extensively tests them in different settings, with different single-modal representations, within the context of video-hyperlinking. Our novel bidirectional symmetrical deep neural networks are compared to classical autoencoders and are shown to yield significantly improved multimodal embeddings that significantly (alpha=0.0001) outperform multimodal embeddings obtained by deep autoencoders with an absolute improvement in precision at 10 of 14.1% when embedding visual concepts and automatic transcripts and an absolute improvement of 4.3% when embedding automatic transcripts with features obtained with very deep convolutional neural networks, yielding 80% of precision at 10.

CICIDS2017 is a well-known benchmark dataset for intrusion detection system (IDS) model creation. However, several studies revealed that the CICIDS2017 dataset still has several weaknesses. This research aims to improve R-CICIDS2017 by removing unrealistic features. After removing unrealistic features, we use a deep neural network (DNN) to predict the R-CICIDS-2017 dataset. We focus our predictions on attacks labeled denial of service (DoS) and distributed DoS (DDoS) in the dataset, as these attacks are the most frequent in the Internet of Things (IoT) world. Because implementation in IoT must be lightweight, we only use 0.1 of the entire dataset. We develop quantization for several data types to determine which model is best suited for running on IoT edge devices. The test results show that the Shannon entropy of the sampled and original datasets for each label is identical. This answer indicates that the DNN model for IDS on IoT with R-CICIDS2017 uses a low-complexity algorithm. The DNN test on CICIDS2017 and R-CICIDS2017 reveals that DNNs using unrealistic datasets exhibit better performance; however, this is not feasible for implementation. DNN with R-CICIDS2017 is optimal because its performance surpasses that of GNB and uses features that can be implemented in the IoT environment. Finally, the test shows that dynamic quantization has better accuracy loss than static int 8 quantization, and its model size is better than static float 16 quantization. We obtain a DNN model that achieves optimum efficiency for IDS deployment on edge devices within IoT networks, utilizing R-CICIDS2017.

Deep Learning Radial Basis Function Neural Networks Based Automatic Detection of Diabetic Retinopathy

Deep neural networks (DNNs) have seen resurgent attraction to be implemented in edge applications. However, such implementations are not easy to achieve because execution of DNNs often require more resources than those provided by individual edge devices. On the other hand, relying on model-level distribution methods to implement a DNN on connected edge devices leads to costly communication overheads. To utilize available in-the-edge resources with less communication overhead, we propose using edge-tailored models comprised of nearly-independent narrow DNNs, the inference of which are accelerated using small cost-efficient RISC-based engines. We implement these engines on PYNQ boards as a platform that mimics the limited resources of edge devices. We create the narrow DNNs based on the available resources of PYNQ boards, and allocate each narrow DNN to one engine, implemented in an FPGA. We compare the communication overhead of our implantation against the state-of-the-art model-level distribution methods.

Recent breakthroughs in deep neural networks (DNNs) have fueled a tremendously growing demand for bringing DNN-powered intelligence into mobile platforms. While the potential of deploying DNNs on resource-constrained platforms has been demonstrated by DNN compression techniques, the current practice suffers from two limitations: 1) merely stand-alone compression schemes are investigated even though each compression technique only suit for certain types of DNN layers; and 2) mostly compression techniques are optimized for DNNs’ inference accuracy, without explicitly considering other application-driven system performance (e.g., latency and energy cost) and the varying resource availability across platforms (e.g., storage and processing capability). To this end, we propose AdaDeep, a usage-driven, automated DNN compression framework for systematically exploring the desired trade-off between performance and resource constraints, from a holistic system level. Specifically, in a layer-wise manner, AdaDeep automatically selects the most suitable combination of compression techniques and the corresponding compression hyperparameters for a given DNN. Thorough evaluations on six datasets and across twelve devices demonstrate that <inline-formula><tex-math notation="LaTeX">${\sf AdaDeep}$</tex-math></inline-formula> can achieve up to <inline-formula><tex-math notation="LaTeX">$18.6\times$</tex-math></inline-formula> latency reduction, <inline-formula><tex-math notation="LaTeX">$9.8\times$</tex-math></inline-formula> energy-efficiency improvement, and <inline-formula><tex-math notation="LaTeX">$37.3\times$</tex-math></inline-formula> storage reduction in DNNs while incurring negligible accuracy loss. Furthermore, <inline-formula><tex-math notation="LaTeX">${\sf AdaDeep}$</tex-math></inline-formula> also uncovers multiple novel combinations of compression techniques.

Interconnected devices are becoming attractive solutions to integrate physical parameters and making them more accessible for further analysis. Edge devices, located at the end of the physical world, measure and transfer data to the remote server using either wired or wireless communication. The exploding number of sensors, being used in the Internet of Things (IoT), medical fields, or industry, are demanding huge bandwidth and computational capabilities in the cloud, to be processed by Artificial Neural Networks (ANNs) – especially, processing audio, video and images from hundreds of edge devices. Additionally, continuous transmission of information to the remote server not only hampers privacy but also increases latency and takes more power. Deep Neural Network (DNN) is proving to be very effective for cognitive tasks, such as speech recognition, object detection, etc., and attracting researchers to apply it in edge devices. Microcontrollers and single-board computers are the most commonly used types of edge devices. These have gone through significant advancements over the years and capable of performing more sophisticated computations, making it a reasonable choice to implement DNN. In this thesis, a DNN model is trained and implemented for Keyword Spotting (KWS) on two types of edge devices: a bare-metal embedded device (microcontroller) and a robot car. The unnecessary components and noise of audio samples are removed, and speech features are extracted using Mel-Frequency Cepstral Co-efficient (MFCC). In the bare-metal microcontroller platform, these features are efficiently extracted using Digital Signal Processing (DSP) library, which makes the calculation much faster. A Depth wise Separable Convolutional Neural Network (DSCNN) based model is proposed and trained with an accuracy of about 91% with only 721 thousand trainable parameters. After implementing the DNN on the microcontroller, the converted model takes only 11.52 Kbyte (2.16%) RAM and 169.63 Kbyte (8.48%) Flash of the test device. It needs to perform 287,673 Multiply-and-Accumulate (MACC) operations and takes about 7ms to execute the model. This trained model is also implemented on the robot car, Jetbot, and designed a voice-controlled robotic vehicle. This robot accepts few selected voice commands-such as “go”, “stop”, etc. and executes accordingly with reasonable accuracy. The Jetbot takes about 15ms to execute the KWS. Thus, this study demonstrates the implementation of Neural Network based KWS on two different types of edge devices: a bare-metal embedded device without any Operating System (OS) and a robot car running on embedded Linux OS. It also shows the feasibility of bare-metal offline KWS implementation for autonomous systems, particularly autonomous vehicles.