Unified AI Models for Network Security on Edge Devices

Deep Neural Network (DNN) models have achieved excellent performance in many inference tasks and have been widely used in many intelligent applications. However, DNN models often require a lot of computational resources to complete the inference tasks, which hinders the deployment of such models to resource‐constrained edge devices. In order to extend the application scenarios of DNN models, the edge‐cloud collaborative inference methods, represented by model partition, have attracted much research attention in recent years. In scenarios that have multiple edge devices deployed, the edge‐cloud collaborative inference method requires partial migration of tasks, but traditional scheduling methods only migrate tasks at the task level. In this paper, we propose two task scheduling methods, which can solve the problem of partial migration of tasks in multiedge scenarios. The first scheduling method is based on the optimal cutting of a single DNN. The cutting positions of all the models are the same, regardless of the influence of external factors. This method is suitable for chain and directed acyclic graph‐ (DAG‐) type DNNs. The second scheduling method takes external factors such as congestion and queuing delay at the cloud side into consideration, which dynamically selects the cutting position of each DNN to optimize the overall delay and thus is applicable to chain DNN models. The experimental results show that, compared with the baseline method, our proposed scheduling method can reduce the delay by up to 6.48x.

Traditionally Deep Neural Networks (DNNs) services are deployed in the cloud, due to the computation-intensive DNN models. In recent years, as emerging edge computing provides new possibilities for DNN applications, we have opportunities to process DNN models in the cloud and on the device collaboratively, i.e., edge-cloud computing. Since cloud and edge devices demonstrate significant diversity on inference latency, network transmission overhead, memory capacity and power consumption, it is a big challenge to determine the DNN model deployment in the cloud and on edge devices. In this paper, we characterize the behaviours of three types of DNN models, i.e., CNN, LSTM and MLP, on four types of platforms, i.e., server-class CPU, server-class GPU, embedded device with GPU, and smart-phones. Our experimental results demonstrate that we can carefully tune a deployment strategy for DNN models in the cloud, and on big and (or) little cores of the edge device, to balance performance and power consumption.

Deep learning is being incorporated in many modern software systems. Deep learning approaches train a deep neural network (DNN) model using training examples, and then use the DNN model for prediction. While the structure of a DNN model as layers is observable, the model is treated in its entirety as a monolithic component. To change the logic implemented by the model, e.g. to add/remove logic that recognizes inputs belonging to a certain class, or to replace the logic with an alternative, the training examples need to be changed and the DNN needs to be retrained using the new set of examples. We argue that decomposing a DNN into DNN modules— akin to decomposing a monolithic software code into modules—can bring the benefits of modularity to deep learning. In this work, we develop a methodology for decomposing DNNs for multi-class problems into DNN modules. For four canonical problems, namely MNIST, EMNIST, FMNIST, and KMNIST, we demonstrate that such decomposition enables reuse of DNN modules to create different DNNs, enables replacement of one DNN module in a DNN with another without needing to retrain. The DNN models formed by composing DNN modules are at least as good as traditional monolithic DNNs in terms of test accuracy for our problems.

Machine learning has surged in popularity in recent years, with deep neural networks (DNNs) becoming a cornerstone of applications such as autonomous driving, augmented reality, and natural language processing, owing to their remarkable accuracy and scalability. As the demand for greater portability and flexibility grows, there is a strong push to optimize these applications for resource-constrained edge devices like smartphones and IoT devices.However, deploying DNN models on such devices presents significant challenges, primarily due to the large size of these models, which is a key contributor to their high performance. For instance, the DeepSpeech 2 model outperforms its predecessor, DeepSpeech 1, but its size is eight times larger, encompassing approximately 68 million parameters. Similarly, other networks designed for various tasks, including convolutional neural networks like VGG and transformer-based networks like BERT and GPT, also exhibit substantial sizes. For example, the GPT-3 model, a predecessor to ChatGPT, consists of an astonishing 175 billion parameters. Consequently, fitting DNN models onto resource-limited edge devices is a formidable challenge. My thesis addresses these challenges with a three-part approach: * High-Performance DNN Accelerators: I will develop DNN accelerators that excel in high performance, low latency, and energy efficiency through an algorithm and hardware co-design across diverse FPGA architectures (Chapter 2). * Real-Time DNN Execution: I aim to achieve exceptional accuracy and real-time execution for various large-scale DNNs on off-the-shelf mobile devices, IoT devices, and other platforms (Chapter 3). * CompVQC Framework: I will explore a systematic framework, CompVQC, to reduce the circuit length of quantum neural networks (QNNs) through model compression, which can achieve up to a 20% improvement in accuracy. This reveals that compilation-aware compression can enhance the robustness of QNNs on near-term noisy quantum devices (Chapter 4). The field of AI has recently undergone a significant transformation, ushering in the era of generative AI, where models like ChatGPT and Stable Diffusion create realistic and creative content for a wide range of applications. As generative AI becomes more prevalent, it is crucial to adapt algorithm-hardware co-design strategies to meet the specific needs of these models, particularly on edge devices. In the final chapter (Chapter 5), I will introduce my ongoing and future work focused on activating generative AI on the edge.--Author's abstract

Deep neural network (DNN) models typically have many hyperparameters that can be configured to achieve optimal performance on a particular dataset. Practitioners usually tune the hyperparameters of their DNN models by training a number of trial models with different configurations of the hyperparameters, to find the optimal hyperparameter configuration that maximizes the training accuracy or minimizes the training loss. As such hyperparameter tuning usually focuses on the model accuracy or the loss function, it is not clear and remains under-explored how the process impacts other performance properties of DNN models, such as inference latency and model size. On the other hand, standard DNN models are often large in size and computing-intensive, prohibiting them from being directly deployed in resource-bounded environments such as mobile devices and Internet of Things (IoT) devices. To tackle this problem, various model optimization techniques (e.g., pruning or quantization) are proposed to make DNN models smaller and less computing-intensive so that they are better suited for resource-bounded environments. However, it is neither clear how the model optimization techniques impact other performance properties of DNN models such as inference latency and battery consumption, nor how the model optimization techniques impact the effect of hyperparameter tuning (i.e., the compounding effect). Therefore, in this paper, we perform a comprehensive study on four representative and widely-adopted DNN models, i.e., CNN image classification , Resnet-50 , CNN text classification , and LSTM sentiment classification , to investigate how different DNN model hyperparameters affect the standard DNN models, as well as how the hyperparameter tuning combined with model optimization affect the optimized DNN models, in terms of various performance properties (e.g., inference latency or battery consumption). Our empirical results indicate that tuning specific hyperparameters has heterogeneous impact on the performance of DNN models across different models and different performance properties. In particular, although the top tuned DNN models usually have very similar accuracy, they may have significantly different performance in terms of other aspects (e.g., inference latency). We also observe that model optimization has a confounding effect on the impact of hyperparameters on DNN model performance. For example, two sets of hyperparameters may result in standard models with similar performance but their performance may become significantly different after they are optimized and deployed on the mobile device. Our findings highlight that practitioners can benefit from paying attention to a variety of performance properties and the confounding effect of model optimization when tuning and optimizing their DNN models.

As training a high-performance deep neural network (DNN) model requires a large amount of data, powerful computing resources and expert knowledge, protecting well-trained DNN models from intellectual property (IP) infringement has raised serious concerns in recent years. Most existing methods using DNN watermarks to verify the ownership of the models after IP infringement occurs, which is reactive in the sense that they cannot prevent unauthorized users from using the model in the first place. Different from these methods, in this article, we propose an active authorization control and user’s fingerprint tracking method for the IP protection of DNN models by utilizing sample-specific backdoor attack. The proposed method inversely and multiplely exploits sample-specific trigger as the key to implement authorization control for DNN model, in which the generated triggers are imperceptible and sample-specific for clean images. Specifically, a U-Net model is used to generate backdoor instances. Then, the target model is trained on the clean images and backdoor instances, which are inversely labeled as wrong classes and correct classes, respectively. Only authorized users can use the target model normally by pre-processing the clean images through the U-Net model. Moreover, the images processed by the U-Net model will contain unique fingerprint that can be extracted to verify and track the corresponding user’s identity. This article is the first work that utilizes the sample-specific backdoor attack to implement active authorization control and user’s fingerprint management for DNN model under black-box scenarios. Extensive experimental results on ImageNet dataset and YouTube Aligned Face dataset demonstrate that the proposed method is effective in protecting the DNN model from unauthorized usage. Specifically, the protected model has a low inference accuracy (1.00%) for unauthorized users, while maintaining a normal inference accuracy (97.67%) for authorized users. Besides, the proposed method can achieve 100% fingerprint tracking success rates on both the ImageNet and YouTube Aligned Face datasets. Moreover, it is demonstrated that the proposed method is robust against fine-tuning attack, pruning attack, pruning attack with retraining, reverse-engineering attack, adaptive attack, and JPEG compression attack. The code is available at https://github.com/nuaaaisec/SSAT .

The cloud-only and edge-computing approaches have recently been proposed to satisfy the requirements of complex neural networks. However, the cloud-only approach generates a latency challenge because of the high data volumes that must be sent to a centralized location in the cloud. Less-powerful edge computing resources require a compression model for computation reduction, which degrades the model trading accuracy. To address this challenge, deep neural network (DNN) partitioning has become a recent trend, with DNN models being sliced into head and tail portions executed at the mobile edge devices and cloud server, respectively. We propose Edgepipe, a novel partitioning method based on pipeline inference with an image stream to automatically partition DNN computation between the edge device and cloud server, thereby reducing the global latency and enhancing the system-wide real-time performance. This method adapts to various DNN architectures, hardware platforms, and networks. Here, when evaluated on a suite of five DNN applications, Edgepipe achieves average latency speedups of 1.241× and 1.154× over the cloud-only approach and the state-of-the-art approach known as “Neurosurgeon”, respectively.

Although deep neural network (DNN) models have been extensively studied, they are often too complex to directly execute in real time on smart cameras with limited resources. Time-domain (TD) DNN models that incorporate frequency-domain (FD) layers, also known as hybrid-domain models, have been developed to mitigate the above problem; however, these require time/FD transforms. Currently, a pure FD DNN does not exist. Thus, our study proposes the first of such, along with our lightweight time/FD transform. Our model ensures that the networks perform faster on smart cameras and are more memory and energy efficient, both of which are important for smart cameras utilizing edge computing. Unlike existing TD or hybrid-domain studies, our model optimizes several internal neural network layers and implements a lightweight time/FD transform to reduce the number of calculations. More importantly, our study is the first to realize an FD fully connected layer, which can better represent a spectral feature distribution. The experimental results show that our accuracy slightly outperforms that of the existing time and hybrid-domain studies. In addition, our model’s inference speed on the edge computing platform was shown to be faster by a maximum of 52.01% for the MNIST data set and 52.00% for the CIFAR-10 data set. Furthermore, our model can improve frames per second (FPS) by at least 52.00% and memory usage by 43.64%, and save approximately 26.09% of power consumption for the MNIST data set.

Interconnected devices are becoming attractive solutions to integrate physical parameters and making them more accessible for further analysis. Edge devices, located at the end of the physical world, measure and transfer data to the remote server using either wired or wireless communication. The exploding number of sensors, being used in the Internet of Things (IoT), medical fields, or industry, are demanding huge bandwidth and computational capabilities in the cloud, to be processed by Artificial Neural Networks (ANNs) – especially, processing audio, video and images from hundreds of edge devices. Additionally, continuous transmission of information to the remote server not only hampers privacy but also increases latency and takes more power. Deep Neural Network (DNN) is proving to be very effective for cognitive tasks, such as speech recognition, object detection, etc., and attracting researchers to apply it in edge devices. Microcontrollers and single-board computers are the most commonly used types of edge devices. These have gone through significant advancements over the years and capable of performing more sophisticated computations, making it a reasonable choice to implement DNN. In this thesis, a DNN model is trained and implemented for Keyword Spotting (KWS) on two types of edge devices: a bare-metal embedded device (microcontroller) and a robot car. The unnecessary components and noise of audio samples are removed, and speech features are extracted using Mel-Frequency Cepstral Co-efficient (MFCC). In the bare-metal microcontroller platform, these features are efficiently extracted using Digital Signal Processing (DSP) library, which makes the calculation much faster. A Depth wise Separable Convolutional Neural Network (DSCNN) based model is proposed and trained with an accuracy of about 91% with only 721 thousand trainable parameters. After implementing the DNN on the microcontroller, the converted model takes only 11.52 Kbyte (2.16%) RAM and 169.63 Kbyte (8.48%) Flash of the test device. It needs to perform 287,673 Multiply-and-Accumulate (MACC) operations and takes about 7ms to execute the model. This trained model is also implemented on the robot car, Jetbot, and designed a voice-controlled robotic vehicle. This robot accepts few selected voice commands-such as “go”, “stop”, etc. and executes accordingly with reasonable accuracy. The Jetbot takes about 15ms to execute the KWS. Thus, this study demonstrates the implementation of Neural Network based KWS on two different types of edge devices: a bare-metal embedded device without any Operating System (OS) and a robot car running on embedded Linux OS. It also shows the feasibility of bare-metal offline KWS implementation for autonomous systems, particularly autonomous vehicles.

Recently, deep neural networks (DNNs) have been widely used in various fields such as autonomous vehicles and smart homes. Since these DNNs can be directly implemented on edge devices, they offer advantages such as real-time processing in low-power and low-bandwidth environments. However, deployment of DNNs in embedded systems, including edge devices, exposes them to threats such as fault injection attacks. In this paper, we introduce two methods to induce misclassifications by using clock and voltage glitch-based fault attacks in devices where DNN models are executed. As a result of experiments on a microcontroller implemented with a DNN for image classification, we show that glitch injection attacks can lead with high probability to serious misclassifications. Furthermore, we propose a countermeasure to glitch attacks on the softmax function, and confirm that this method is effective in preventing misclassifications.

CICIDS2017 is a well-known benchmark dataset for intrusion detection system (IDS) model creation. However, several studies revealed that the CICIDS2017 dataset still has several weaknesses. This research aims to improve R-CICIDS2017 by removing unrealistic features. After removing unrealistic features, we use a deep neural network (DNN) to predict the R-CICIDS-2017 dataset. We focus our predictions on attacks labeled denial of service (DoS) and distributed DoS (DDoS) in the dataset, as these attacks are the most frequent in the Internet of Things (IoT) world. Because implementation in IoT must be lightweight, we only use 0.1 of the entire dataset. We develop quantization for several data types to determine which model is best suited for running on IoT edge devices. The test results show that the Shannon entropy of the sampled and original datasets for each label is identical. This answer indicates that the DNN model for IDS on IoT with R-CICIDS2017 uses a low-complexity algorithm. The DNN test on CICIDS2017 and R-CICIDS2017 reveals that DNNs using unrealistic datasets exhibit better performance; however, this is not feasible for implementation. DNN with R-CICIDS2017 is optimal because its performance surpasses that of GNB and uses features that can be implemented in the IoT environment. Finally, the test shows that dynamic quantization has better accuracy loss than static int 8 quantization, and its model size is better than static float 16 quantization. We obtain a DNN model that achieves optimum efficiency for IDS deployment on edge devices within IoT networks, utilizing R-CICIDS2017.

Background and Objectives: This study investigated the usefulness of deep neural network (DNN) models based on 18F-fluorodeoxyglucose positron emission tomography (FDG-PET) and blood inflammatory markers to assess the therapeutic response in pyogenic vertebral osteomyelitis (PVO). Materials and Methods: This was a retrospective study with prospectively collected data. Seventy-four patients diagnosed with PVO underwent clinical assessment for therapeutic responses based on clinical features during antibiotic therapy. The decisions of the clinical assessment were confirmed as ‘Cured’ or ‘Non-cured’. FDG-PETs were conducted concomitantly regardless of the decision at each clinical assessment. We developed DNN models depending on the use of attributes, including C-reactive protein (CRP), erythrocyte sedimentation ratio (ESR), and maximum standardized FDG uptake values of PVO lesions (SUVmax), and we compared their performances to predict PVO remission. Results: The 126 decisions (80 ‘Cured’ and 46 ‘Non-cured’ patients) were randomly assigned with training and test sets (7:3). We trained DNN models using a training set and evaluated their performances for a test set. DNN model 1 had an accuracy of 76.3% and an area under the receiver operating characteristic curve (AUC) of 0.768 [95% confidence interval, 0.625–0.910] using CRP and ESR, and these values were 79% and 0.804 [0.674–0.933] for DNN model 2 using ESR and SUVmax, 86.8% and 0.851 [0.726–0.976] for DNN model 3 using CRP and SUVmax, and 89.5% and 0.902 [0.804–0.999] for DNN model 4 using ESR, CRP, and SUVmax, respectively. Conclusions: The DNN models using SUVmax showed better performances when predicting the remission of PVO compared to CRP and ESR. The best performance was obtained in the DNN model using all attributes, including CRP, ESR, and SUVmax, which may be helpful for predicting the accurate remission of PVO.

Automated image classification is an essential task of the computer vision field. The tagging of images into a set of predefined groups is referred to as image classification. The implementation of computer vision to automate image classification would be beneficial because manual image evaluation and identification can be time-consuming, particularly when there are many images of different classes. Deep learning approaches are proven to overperform existing machine learning techniques in many fields in recent years, and computer vision is one of the most notable examples. The very deep neural network (VDCNN) is a powerful deep learning model for image classification, and this paper examines it briefly using MNIST handwritten digit dataset. This dataset is used to prove the efficacy of very deep neural networks over other deep learning models. The proposed study aims to comprehend the very deep neural network architecture used to accomplish a handwritten digit recognition task. The feasibility of the proposed model is evaluated using mean accuracy, validation accuracy, and standard deviation. The study results of the very deep neural network model are compared to a convolutional neural network and convolutional neural network with batch normalization. According to the results of the comparison study, very deep neural networks achieve high accuracy of 99.1% for a handwritten dataset. The outcome of the proposed work is used to interpret how well a very deep neural network performs when compared to the other two models of deep neural networks. This proposed architecture may be used to automate the classification of handwritten digits dataset.

Decision of the model complexity is a significant challenge in contemporary data-driven modeling applications. Designing neural architecture involves the process of determining the optimal model complexity for deep neural networks (DNNs) models in order to uncover relationships in real-world data patterns. Consequently, optimizing DNN architectures is crucial for enhancing the practical approximation performance of DNN models to real-world data. This study implements a data-driven neuroevolution scheme for the optimal neural architecture search (NAS) and demonstrates a data-driven engineering application for cooling load estimation. The proposed neuroevolution scheme aims at evolving to the best generalizing DNN model that well suits the modeling complexity requirements of the dataset. To this end, the objective function for the neural architecture optimization process is simplified to the mean square error of the test dataset, which enables to reduce the risk of insufficient generalization during NAS. By employing this objective function for evolution field optimization (EFO), the proposed neuroevolution process can automatically achieve the optimal model complexity, preventing overfitting and underfitting cases, and thereby attaining almost the best generalization for the dataset. For this purpose, this approach combines parametric learning with the backpropagation algorithm and structural learning with EFO-based neural architecture search to address data-driven, optimal complexity DNN model generation problems. Effectiveness of the method is demonstrated in the cooling load estimation problem of residential buildings, and performances of the optimal DNN models with four objective functions are analyzed. The design of objective function for the best generalizing model is also elaborated.

It is usually infeasible to fit and train an entire large deep neural network (DNN) model using a single edge device due to the limited resources. To facilitate intelligent applications across edge devices, researchers have proposed partitioning a large model into several sub-models, and deploying each of them to a different edge device to collaboratively train a DNN model. However, the communication overhead caused by the large amount of data transmitted from one device to another during training, as well as the sub-optimal partition point due to the inaccurate latency prediction of computation at each edge device can significantly slow down training. In this paper, we propose AccEPT, an acceleration scheme for accelerating the edge collaborative pipeline-parallel training. In particular, we propose a light-weight adaptive latency predictor to accurately estimate the computation latency of each layer at different devices, which also adapts to unseen devices through continuous learning. Therefore, the proposed latency predictor leads to better model partitioning which balances the computation loads across participating devices. Moreover, we propose a bit-level computation-efficient data compression scheme to compress the data to be transmitted between devices during training. Our numerical results demonstrate that our proposed acceleration approach is able to significantly speed up edge pipeline parallel training up to 3 times faster in the considered experimental settings