SecDeep

As the security issue of cloud-offloaded Deep Learning (DL) inference is drawing increasing attention, running DL inference in Trusted Execution Environments (TEEs) has become a common practice. Latency prediction of TEE-hosted DL model inference is essential for many scenarios, such as DNN model architecture searching with a latency constraint or layer scheduling in model-parallelism inference. However, existing solutions fail to address the memory over-commitment issue in resource-constrained environments inside TEEs.This paper presents Sectum, an accurate latency predictor for DL inference inside TEE enclaves. We first perform a synthetic empirical study to analyze the relationship between inference latency and memory occupation. Sectum predicts inference latency following a two-stage design based on some critical observations. First, Sectum uses a Graph Neural Network (GNN)-based model to detect whether a given model would trigger memory over-commitment in TEEs. Then, combining operator-level latency modeling with linear regression, Sectum could predict the latency of a model. To evaluate Sectum, we design a large dataset that contains the latency information of over 6k CNN models. Our experiments demonstrate that Sectum could achieve over 85% ±10% accuracy of latency prediction. To our knowledge, Sectum is the first method to predict TEE-hosted DL inference latency accurately.

The code base of system software is growing fast, which results in a large number of vulnerabilities: for example, 296 CVEs have been found in Xen hypervisor and 2195 CVEs in Linux kernel. To reduce the reliance on the trust of system software, many researchers try to provide trusted execution environments (TEEs), which can be categorized into two types: non-privileged TEEs and privileged TEEs. Non-privileged TEEs (e.g., Intel SGX) are extensible, but cannot protect security services like virtual machine introspection (VMI) due to the lack of system-level semantics. On the contrary, privileged TEEs (e.g., the secure world of ARM TrustZone) have system-level semantics, but any additional service implemented in the privileged TEE directly increases the TCB of the entire system. In this article, we propose a new design of TEE to support system-level security services and achieve better extensibility with a small TCB. Each TEE instance of the proposed design is named a <small>Colony</small>. Specifically, we introduce a <i>secure monitor</i> for isolation and capability management. Each <small>Colony</small> is assigned capabilities to access only necessary system-level semantics. We use the new TEE to build four security services, including secure device accessing, VMI tools, a system call tracer, and a much more complex service to virtualize ARM TrustZone with multiple <small>Colonies</small>. We have implemented the system on ARMv7 and ARMv8 platforms, in Xen hypervisor and Linux kernel, and perform a detailed evaluation to show its efficiency.<xref rid="fn1" ref-type="fn">¹</xref><fn id="fn1"><label>1.</label> This paper is an extended version of the conference paper published in USENIX Security&#x2019;17: vTZ: Virtualizing ARM TrustZone <xref ref-type="bibr" rid="ref29">[29]</xref> . A brief summary of differences is in Section<xref ref-type="sec" rid="sec8">8</xref> . </fn>

Data security has become the focus of public concern in widely used Deep Learning (DL) applications. Existing attacks can accurately recover any input entered the models. Therefore, it is of the same importance to protect DL models as well as data. Although service providers may offer Trusted Execution Environment (TEE) such as Trusted Software Guard eXtensions (SGX) for model security. The additional delay introduced by security computation cannot be neglected even compared with the delay introduced by DL inferences. In this paper, we propose an online SGX-based system to protect the DL inference process using crowd sourced SGXs. To motivate devices to contribute their SGXs, we apply an online auction mechanism. We decompose the long-term problem into multi-rounds and solve the decomposed problem in an online manner. The evaluation results show that the proposed algorithm of the online system outperforms the baseline algorithms by 160% in terms of social cost.

Analog in-memory computing (AIMC) using memristive devices is considered a promising Non-von Neumann approach for deep learning (DL) inference tasks. However, inaccuracies in the programming of devices, that are attributed to conductance variations, pose a key challenge toward achieving sufficient compute precision for DL inference. Fortunately, conduction variations in memristive devices, such as phase-change memory (PCM) devices, exhibit a strong state dependence. This state dependence can be exploited in synaptic unit cells that comprise more than one memristive device, to encode positive or negative weights. In such multi-memristive unit cells, we propose a method that optimally maps the weights to the device conductance values, by maximizing the number of devices at the stable SET and RESET states. We demonstrate that this method reduces the matrix-vector multiplication (MVM) error and is more resilient to non-ideal device retention characteristics. With this approach, we increase the mean experimental inference accuracy of a network trained for MNIST classification by 0.71% on two PCM-based AIMC cores, and the hardware-realistic simulated top-1 accuracy of a network trained for ImageNet classification by 0.28%, while significantly reducing variability across multiple experiment instances.

Pontis: A decentralized framework for unifying remote attestation and enabling interoperability between heterogeneous TEEs

We examine data-intensive real-time applications, such as forest fire detection, medical emergency services, oil pipeline monitoring, etc., that require relatively low response time in processing data from the Internet of Things (IoT) devices. Typically, in such circumstances, the edge computing paradigm is utilised to drastically reduce the processing delay of such applications. However, with the growing IoT devices, the edge device cluster needs to be configured properly such that the real-time requirements are met. Therefore, the cluster configuration must be dynamically adapted to the changing network topology of the edge cluster in order to minimise the observed overall communication delay incurred by edge devices when processing data from IoT devices. To this end, we propose an intelligent assignment of IoT devices to edge devices based on Reinforcement Learning such that communication delay is minimised and none of the edge devices is overloaded. We demonstrate, with some preliminary results, that our algorithm outperforms the state-of-the-art.

Deep Learning is ubiquitous today and is increasingly moving from the cloud down to the edge of networked infrastructures, where it enables embedded applications to perform complex inference tasks close to the data sources, reducing long-distance data movement and alleviating the need for a powerful cloud infrastructure. Edge-class multi-processor system on chip (MPSoC) devices featuring an on-chip FPGA fabric offer key advantages for Deep Learning inference tasks, especially for complex applications where multiple models may be run concurrently in the same platform. In this work, we propose an approach and a practical framework for the systematic characterization of multithreaded Deep Learning inference on edge FPGA MPSoCs. We instantiate the framework into a real-world MPSoC platform, targeting Xilinx Vitis-AI as a representative example of a commercial Deep Learning acceleration toolkit for edge environments. We design a comprehensive experimental campaign and apply it to the platform for several convolutional neural networks, each trained on three different datasets. We show that our approach can be used for both hardware- and software-level analysis of a target system. Among other findings, the analysis revealed a suboptimal behavior of the underlying toolkit runtime, involving the utilization of the accelerator cores and the uneven software latency of the support library, influenced by the shapes of the input tensors.

With the rapid growth of the Internet of Things (IoT), there is a massive number of constrained devices connected to the internet, resulting in the generation of large data volume. The deep learning (DL) has become a promised solution to extract more valuable knowledge from collected data but struggle to execute DL algorithms due to limited computing resources of IoT devices. Leveraging the computation power of cloud computing, we could offload the data to cloud servers for processing and response results to the devices. But, this paradigm leads to a significant increase in latency and security risks. Therefore, there have been many efforts to perform DL algorithms on edge devices, which are close to the data sources. Deploying various DL systems to such edge devices a complex process due to the diversity of DL models and running environment configuration. In this paper, we propose a light-weight framework supporting to deploy deep learning as a service running on edge devices, namely DLASE. There are two key advantages of our proposal. The first one is that we enable remote deployment of various deep learning models to edge devices. The other one is that these models are run as a service to serve requests from IoT devices. To demonstrate the light-weight and flexibility of DLASE, we practically evaluate the framework performances (consumed memory, CPU, latency) by running various combinations of DL models and edge devices.

Trusted Execution Environments (TEEs) such as ARM TrustZone are in widespread use in both mobile and embedded devices, and they are used to protect sensitive secrets while often sharing the same computational hardware as untrusted code. Although there has been limited research in the area, the threat of microarchitectural attacks against ARM TrustZone has not been thoroughly studied. This is not the case for other TEEs, such as Intel SGX, where the security promises of the TEE have been violated numerous times by the academic community, showing that it is possible to use side-channel attacks to gain detailed insight into the microarchitectural behavior of trusted code. In this work, we show that TrustZone is susceptible to similar attacks, and we demonstrate the ability to achieve cache attacks with high temporal precision, high spatial precision, and low noise. These tools make it easy to monitor the data flow and code flow of TrustZone code with great resolution, and we apply our techniques to investigate the security of a real-world application. We examine ECDSA signing in Qualcomm's implementation of Android's hardware-backed keystore and identify a series of vulnerabilities that leak sensitive cryptographic information through shared microarchitectural structures. By using the powerful attacks developed in this paper, we are able to successfully extract this sensitive information and fully recover a 256-bit private key from Qualcomm's version of the hardware-backed keystore.

Recent years have seen an exploration of data volumes from a myriad of IoT devices, such as various sensors and ubiquitous cameras. The deluge of IoT data creates enormous opportunities for us to explore the physical world, especially with the help of deep learning techniques. Traditionally, the Cloud is the option for deploying deep learning based applications. However, the challenges of Cloud-centric IoT systems are increasing due to significant data movement overhead, escalating energy needs, and privacy issues. Rather than constantly moving a tremendous amount of raw data to the Cloud, it would be beneficial to leverage the emerging powerful IoT devices to perform the inference task. Nevertheless, the statically trained model could not efficiently handle the dynamic data in the real in-situ environments, which leads to low accuracy. Moreover, the big raw IoT data challenges the traditional supervised training method in the Cloud. To tackle the above challenges, we propose In-situ AI, the first Autonomous and Incremental computing framework and architecture for deep learning based IoT applications. We equip deep learning based IoT system with autonomous IoT data diagnosis (minimize data movement), and incremental and unsupervised training method (tackle the big raw IoT data generated in ever-changing in-situ environments). To provide efficient architectural support for this new computing paradigm, we first characterize the two In-situ AI tasks (i.e. inference and diagnosis tasks) on two popular IoT devices (i.e. mobile GPU and FPGA) and explore the design space and tradeoffs. Based on the characterization results, we propose two working modes for the In-situ AI tasks, including Single-running and Co-running modes. Moreover, we craft analytical models for these two modes to guide the best configuration selection. We also develop a novel two-level weight shared In-situ AI architecture to efficiently deploy In-situ tasks to IoT node. Compared with traditional IoT systems, our In-situ AI can reduce data movement by 28-71%, which further yields 1.4X-3.3X speedup on model update and contributes to 30-70% energy saving.

Efforts to leverage the benefits of Deep Learning(DL) models for performing inference in resource constrained embedded devices is very popular nowadays. Researchers worldwide are trying to come up with software and hardware accelerators that make pre-trained DL models suitable for running the inference phase in these devices. Apart from software and hardware accelerators, DL model partitioning and offloading to Cloud or Edge network servers is becoming more and more practicable with increasing importance of Edge Computing. DL inference workflow partitioning and offloading can augment software / hardware acceleration for improved latency and energy efficiency in resource constrained embedded systems. Efficacy of a computation offloading to system is dependent on proper profiling of timing and energy required for processing DL algorithms. In this work we implement a DL inference offloading system using Raspberry Pi 3 based robot vehicle with a hardware accelerator from Intel (Neural Compute Stick). We report the workload partitioning approach, detailed experimental results and performance improvements achieved. We demonstrate that the current approach of DL execution profiling without considering dynamic system load of the edge device results in sub-optimal partitioning of DL algorithm and provide a solution approach to that.

Deep learning is one of the hottest research directions in the field of artificial intelligence. It has achieved results which subvert these of traditional methods. However, the demand for computing ability of hardware platform is also increasing. The academia and industry mainly use heterogeneous GPUs to accelerating computation. ARM is relatively more open than GPUs. The purpose of this paper is to study the performance and related acceleration techniques of ThunderX high-performance many-core ARM chips under large-scale inference tasks. In order to study the computational performance of the target platform objectively, several deep models are adapted for acceleration. Through the selection of computational libraries, adjustment of parallel strategies, application of various performance optimization techniques, we have excavated the computing ability of many-core ARM platforms deeply. The final experimental results show that the performance of single-chip ThunderX is equivalent to that of the i7 7700 K chip, and the overall performance of dual-chip can reach 1.77 times that of the latter. In terms of energy efficiency, the former is inferior to the latter. Stronger cooling system or bad power management may lead to more power consumption. Overall, high-performance ARM chips can be deployed in the cloud to complete large-scale deep learning inference tasks which requiring high throughput.

Internet-of-Things (IoT) devices are becoming both intelligent and green. On the one hand, Deep Neural Network (DNN) compression techniques make it possible to run deep learning applications on devices equipped with low-end microcontrollers (MCUs). By performing deep learning on IoT devices, in-situ decision-making can be made, which can improve the responsiveness of such devices to the environment and reduce data uploading to edge servers or clouds to save valuable network bandwidth. On the other hand, many IoT devices in the future will be powered by energy harvesters instead of batteries to reduce environmental pollution and achieve permanent service free of battery maintenance. As the energy output of energy harvesters is tiny and unstable, energy harvesting IoT (EH-IoT) devices will experience frequent power failures during their execution, making the software task hard to progress. The deep learning tasks running on such devices must face this challenge and, at the same time, ensure satisfactory execution efficiency. We believe deploying deep learning on EH-IoT devices that execute intermittently will be a challenging yet promising research direction. To motivate research in this direction, this paper summarizes existing solutions and provides an in-depth discussion of future challenges that deserve further investigation. With IoT devices becoming more intelligent and green, DNN inference on EH-IoT devices will generate a much more significant impact in the future in academia and industry.

Homomorphic encryption (HE) is the ultimate tool for performing secure computations even in untrusted environments. Application of HE for deep learning (DL) inference is an active area of research, given the fact that DL models are often deployed in untrusted environments (e.g., third-party servers) yet inferring on private data. However, existing HE libraries [somewhat (SWHE), leveled (LHE) or fully homomorphic (FHE)] suffer from extensive computational and memory overhead. Few performance optimized high-speed homomorphic libraries are either suffering from certain approximation issues leading to decryption errors or proven to be insecure according to recent published attacks. In this article, we propose architectural tricks to achieve performance speedup for encrypted DL inference developed with exact HE schemes without any approximation or decryption error in homomorphic computations. The main idea is to apply quantization and suitable data packing in the form of bitslicing to reduce the costly noise handling operation, Bootstrapping while achieving a functionally correct and highly parallel DL pipeline with a moderate memory footprint. Experimental evaluation on the MNIST dataset shows a significant ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$37\times$ </tex-math></inline-formula> ) speedup over the nonbitsliced versions of the same architecture. Low memory bandwidths (700 MB) of our design pipelines further highlight their promise toward scaling over larger gamut of Edge-AI analytics use cases.

Internet of Things (IoT) devices can apply mobile edge computing (MEC) and energy harvesting (EH) to provide high-level experiences for computational intensive applications and concurrently to prolong the lifetime of the battery. In this paper, we propose a reinforcement learning (RL) based offloading scheme for an IoT device with EH to select the edge device and the offloading rate according to the current battery level, the previous radio transmission rate to each edge device, and the predicted amount of the harvested energy. This scheme enables the IoT device to optimize the offloading policy without knowledge of the MEC model, the energy consumption model, and the computation latency model. Further, we present a deep RL-based offloading scheme to further accelerate the learning speed. Their performance bounds in terms of the energy consumption, computation latency, and utility are provided for three typical offloading scenarios and verified via simulations for an IoT device that uses wireless power transfer for energy harvesting. Simulation results show that the proposed RL-based offloading scheme reduces the energy consumption, computation latency, and task drop rate, and thus increases the utility of the IoT device in the dynamic MEC in comparison with the benchmark offloading schemes.