XpuTEE: A High-Performance and Practical Heterogeneous Trusted Execution Environment for GPUs

With the extensive deployment and application of the Internet of Things (IoT), 5G and 6G technologies and edge intelligence, the volume of data generated by IoT and the number of intelligence applications derived from these data are rapidly growing. However, the absence of effective mechanisms to safeguard the vast data generated by IoT, along with the security and privacy of edge intelligence applications, hinders their further development and adoption. In recent years, Trusted Execution Environment (TEE) has emerged as a promising technology for securing cloud data storage and cloud processing, demonstrating significant potential for ensuring data and application confidentiality in more scenarios. Nevertheless, applying TEE technology to enhance security in IoT and edge intelligence scenarios still presents several challenges. This paper investigates the technical challenges faced by current TEE solutions, such as performance overhead and I/O security issues, in the context of the resource constraints and data mobility that are inherent to IoT and edge intelligence applications. Using Intel Software Guard Extensions (SGX) technology as a case study, this paper validates these challenges through extensive experiments. The results provide critical assessments and analyses essential for advancing the development and usage of TEE in IoT and edge intelligence scenarios.

Confidential Compute Architecture (CCA) is the latest Trusted Execution Environment (TEE) system on Arm. It offers a VM-level execution environment designed to host applications that manage security-sensitive tasks and safeguard them from malicious system software. Although this VM-level design simplifies TEE adoption, it introduces a large attack surface. Attackers can break isolation by exploiting vulnerabilities in any component of the VM. In this paper, we present HiveTEE, a scalable intra-TEE isolation architecture that leverages Realm Management Extension (RME) and Memory Tagging Extension (MTE). HiveTEE allows developers to partition applications into multiple isolated domains (SDoms), preventing a compromise in one part of the application from propagating across the entire TEE. To evaluate the performance overhead introduced by HiveTEE, we apply it to three real-world applications: OpenSSL, SQLite, and Memcached. The evaluation results show that HiveTEE incurs a small performance overhead (<3%).

With the rapid development of computer and communication technology, embedded systems are widely used in smart devices. The increasing connectivity of these systems and the difficulties in providing comprehensive security have made such devices vulnerable to malicious attacks. Passive defense technologies and traffic-based intrusion detection technologies are not fully effective against such attacks. Trusted execution environment (TEE) technology can ensure system security against unknown attacks to some extent. Most researchers use TrustZone to implement TEE. However, the problem is that the API interface of the TEE module which provides the service is not secure. Therefore, to actively defend against attacks, we developed a trusted computing active measurement architecture based on TrustZone. To overcome the serious problem that modules in the trusted execution environment need to be passively invoked to provide services, we have proposed an active measurement closed-loop immune mechanism. To reduce the trusted computing base and reduce the performance overhead, we removed certain functional modules from the trusted execution environment. In addition, based on this architecture, we developed a trust chain and dynamic measurement method to ensure the security of the target applications. We changed the traditional attack response method, which requires the entire system to be restarted after an attack, by developing a fallback mechanism that is more suitable for the system. Finally, we verified the effectiveness of the architecture by developing an attack model. Performance testing and analysis showed that the architecture reduced the impact of the security mechanisms on the system. In the future, we will extend our research to more fine-grained measurements.

Distributed analytics systems enable users to efficiently perform computations over large distributed data sets. Recently, systems have been proposed that can additionally protect the data's privacy by keeping it encrypted even in memory and by performing the computations using trusted execution environments (TEEs). This approach has the potential to make it much safer to outsource analytics jobs to an untrusted cloud platform or to distribute it across multiple parties. TEEs, however, suffer from side channels, such as timing, memory access patterns, and message sizes that weaken their privacy guarantees. Existing privacy-preserving analytics systems only address a subset of these channels, such as memory access patterns, while largely neglecting size and timing. Moreover, previous attempts to close size and timing channels suffer from high performance costs, impracticality, or a lack of rigorous privacy guarantees.

The maturing blockchain technology has gradually promoted decentralized data storage from cryptocurrencies to other applications, such as trust management, resulting in new challenges based on specific scenarios. Taking the mobile trust blockchain within a vehicular network as an example, many users require the system to process massive traffic information for accurate trust assessment, preserve data reliably, and respond quickly. While existing vehicular blockchain systems ensure immutability, transparency, and traceability, they are limited in terms of scalability, performance, and security. To address these issues, this paper proposes a novel decentralized vehicle trust management solution and a well-matched blockchain framework that provides both security and performance. The paper primarily addresses two issues: i) To provide accurate trust evaluation, the trust model adopts a decentralized and peer-review-based trust computation method secured by trusted execution environments (TEEs). ii) To ensure reliable trust management, a multi-shard blockchain framework is developed with a novel hierarchical Byzantine consensus protocol, improving efficiency and security while providing high scalability and performance. The proposed scheme combines the decentralized trust model with a multi-shard blockchain, preserving trust information through a hierarchical consensus protocol. Finally, real-world experiments are conducted by developing a testbed deployed on both local and cloud servers for performance measurements.

With its huge real-world demands, large-scale confidential computing still cannot be supported by today’s Trusted Execution Environment (TEE), due to the lack of scalable and effective protection of high-throughput accelerators like GPUs, FPGAs, and TPUs etc. Although attempts have been made recently to extend the CPU-like enclave to GPUs, these solutions require change to the CPU or GPU chips, may introduce new security risks due to the side-channel leaks in CPU-GPU communication and are still under the resource constraint of today’s CPU TEE.To address these problems, we present the first Heterogeneous TEE design that can truly support large-scale compute or data intensive (CDI) computing, without any chip-level change. Our approach, called HETEE, is a device for centralized management of all computing units (e.g., GPUs and other accelerators) of a server rack. It is uniquely designed to work with today’s data centres and clouds, leveraging modern resource pooling technologies to dynamically compartmentalize computing tasks, and enforce strong isolation and reduce TCB through hardware support. More specifically, HETEE utilizes the PCIe ExpressFabric to allocate its accelerators to the server node on the same rack for a non-sensitive CDI task, and move them back into a secure enclave in response to the demand for confidential computing. Our design runs a thin TCB stack for security management on a security controller (SC), while leaving a large set of software (e.g., AI runtime, GPU driver, etc.) to the integrated microservers that operate enclaves. An enclaves is physically isolated from others through hardware and verified by the SC at its inception. Its microserver and computing units are restored to a secure state upon termination.We implemented HETEE on a real hardware system, and evaluated it with popular neural network inference and training tasks. Our evaluations show that HETEE can easily support the CDI tasks on the real-world scale and incurred a maximal throughput overhead of 2.17% for inference and 0.95% for training on ResNet152.

Prospects of novel flash-based, crash-tolerant, non-volatile memory (NVM) such as Intel’s Optane DC memory [17] and future CXL-based persistent memory [28] bring about new and exciting usage scenarios for both general-purpose and embedded computing systems involving FPGA-enabled Trusted Execution Environment (TEE) [35], [43]. However, the NVM modules demonstrate high write latency and limited write endurance and therefore, are more suitable for a hybrid NVM + volatile DRAM setup [15]. Furthermore, different memory-based adversaries in NVM including integrity-based attacks demand the use of a robust authentication method such as Bonsai Merkle Tree (BMT) [4]. Conventional BMT authentication schemes should not be directly applied to such hybrid, embedded NVM platforms as the typical frequent update process of a BMT affects runtime performance even when persistence is unnecessary. On the contrary, the latest intermittent BMT update techniques can provide better run-time throughput, but lack crash-consistency [27]. Therefore, a heterogeneous memory-based system would greatly benefit from an authentication mechanism that can change its update method on-the-fly and provide a good balance between the persistence and run-time performance.In this paper, we propose a unified and hardware-friendly BMT framework called opportunistic Merkle tree (OMT). OMT is both modular and run-time adaptive by 1) merging the logic for two different BMT update schemes while still allowing for parallel updates through separate update cores and 2) streamlining the BMT read/verification for both of the update methods with a common datapath to support both recovery-critical and general data, therefore eliminating the need for individual authentication subsystems for different memory modules in a heterogeneous memory platform. Most interestingly, through the use of its adaptive Data and Address Management Unit (DAMU), OMT allows for a run-time switch between the update methods depending on the request type (persistent/intermittent). Extensive testing of OMT in a heterogeneous embedded memory system provides 44% lower memory overhead & up to 22% faster execution in synthetic benchmarks compared to a baseline.

Custom Android ROMs offer enhanced security solutions for enterprise, automotive, and government applications by addressing vulnerabilities inherent in the open-source Android platform. These specialized distributions implement multilayered defensive mechanisms including kernel hardening, permission vulnerability mitigation, and firmware-level zero-trust architecture to combat sophisticated threats. Organizations benefit from reduced attack surfaces, secure deployment options tailored to specific threat models, improved regulatory compliance capabilities, and specialized device security for critical applications. Key innovations include continuous verification of system processes, minimal privilege enforcement, hardware-backed security through trusted execution environments, and runtime integrity monitoring. Despite implementation challenges related to performance overhead, compatibility concerns, and update management, custom ROMs provide significant security advantages by addressing vulnerabilities at their foundation rather than attempting to compensate through application-layer controls. As mobile devices increasingly store sensitive data, these hardened implementations play a crucial role in securing enterprise ecosystems while establishing security patterns that may eventually improve mobile security more broadly.

Mobile devices, which users habitually carry along, have become the main data gateway for the majority of the online services. Any device is able to collect at any time various types of data through its sensors. At the same time, modern identification techniques ask users to send photos of their ID documentation in order to be verified by an online service. Those photos are captured by the device's camera and are considered extremely sensitive. They must be secured and establish that they will not be modified. This paper describes a security framework that preserves the authenticity of a captured photo and ensures that it remains intact while transferred to a remote server. The key inside is to use a background service that is tied to the photo-capturing application and uses secure key storing and cryptographic computation capabilities offered by the Trusted Execution Environment (TEE) of commodity Android devices. At the same time, we leverage Playstore's Licencing Verification Library (LVL) to remotely attest the authenticity of the photo-capturing application at registration time. We have implemented our framework as an Android application on a Nexus 5X, which is powered by a Qualcomm processor with ARM TrustZone Technology. The evaluation of our prototype implementation demonstrates the efficacy of the proposed framework in terms of performance overhead and usability.

There is an increasing emphasis on securing deep learning (DL) inference pipelines for mobile and IoT applications with privacy-sensitive data. Prior works have shown that privacy-sensitive data can be secured throughout deep learning inferences on cloud-offloaded models through trusted execution environments such as Intel SGX. However, prior solutions do not address the fundamental challenges of securing the resource-intensive inference tasks on low-power, low-memory devices (e.g., mobile and IoT devices), while achieving high performance. To tackle these challenges, we propose SecDeep, a low-power DL inference framework demonstrating that both security and performance of deep learning inference on edge devices are well within our reach. Leveraging TEEs with limited resources, SecDeep guarantees full confidentiality for input and intermediate data, as well as the integrity of the deep learning model and framework. By enabling and securing neural accelerators, SecDeep is the first of its kind to provide trusted and performant DL model inferencing on IoT and mobile devices. We implement and validate SecDeep by interfacing the ARM NN DL framework with ARM TrustZone. Our evaluation shows that we can securely run inference tasks with 16× to 172× faster performance than no acceleration approaches by leveraging edge-available accelerators.

Hardware enclaves assist in constructing a trusted execution environment (TEE) to store private code and data and thus become an appealing solution to enhance applications’ security. Nevertheless, state-of-the-art enclave implementations like Intel Software Guard Extensions (SGX) have severe performance issues and hinder the deployment of more complicated applications, especially those written in high-level languages like Java. To reduce the performance overhead, prior work has partitioned applications or rebuilt lightweight language runtimes, but they either require manual labor from developers or fail to provide full-fledged support for existing applications. This work instead provides <i>SAJ</i> , a runtime built upon a full-fledged Java virtual machine (JVM) and thus requires no modifications to applications. <i>SAJ</i> first analyzes the performance of vanilla JVMs running in enclaves and finds that the memory management overhead and boot phase are culprits for performance slowdown. For memory management, <i>SAJ</i> introduces SGX-aware heap layout and garbage collector, which reduces both GC and application execution time. As for the boot phase, <i>SAJ</i> introduces an address-conscious launching mechanism to improve the boot performance. The evaluation under representative Java applications shows that <i>SAJ</i> can reduce the overall GC pause time, application time, and boot time by 2.93 <inline-formula><tex-math notation="LaTeX">$\boldsymbol{\times}$</tex-math></inline-formula> , 2.58 <inline-formula><tex-math notation="LaTeX">$\boldsymbol{\times}$</tex-math></inline-formula> , and 2.73 <inline-formula><tex-math notation="LaTeX">$\boldsymbol{\times}$</tex-math></inline-formula> on average, respectively.

Third-party storage services pose the risk of integrity and confidentiality violations as the current storage policy enforcement mechanisms are spread across many layers in the system stack. To mitigate these security vulnerabilities, we present the design and implementation of Pesos, a Policy Enhanced Secure Object Store (Pesos) for untrusted third-party storage providers. Pesos allows clients to specify per-object security policies, concisely and separately from the storage stack, and enforces these policies by securely mediating the I/O in the persistence layer through a single unified enforcement layer. More broadly, Pesos exposes a rich set of storage policies ensuring the integrity, confidentiality, and access accounting for data storage through a declarative policy language. Pesos enforces these policies on untrusted commodity platforms by leveraging a combination of two trusted computing technologies: Intel SGX for trusted execution environment (TEE) and Kinetic Open Storage for trusted storage. We have implemented Pesos as a fully-functional storage system supporting many useful end-to-end storage features, and a range of effective performance optimizations. We evaluated Pesos using a range of micro-benchmarks, and real-world use cases. Our evaluation shows that Pesos incurs reasonable performance overheads for the enforcement of policies while keeping the trusted computing base (TCB) small.

We propose IMPACC, an MPI+OpenACC framework for heterogeneous accelerator clusters. IMPACC tightly integrates MPI and OpenACC, while exploiting the shared memory parallelism in the target system. IMPACC dynamically adapts the input MPI+OpenACC applications on the target heterogeneous accelerator clusters to fully exploit target system-specific features. IMPACC provides the programmers with the unified virtual address space, automatic NUMA-friendly task-device mapping, efficient integrated communication routines, seamless streamlining of asynchronous executions, and transparent memory sharing. We have implemented IMPACC and evaluated its performance using three heterogeneous accelerator systems, including Titan supercomputer. Results show that IMPACC can achieve easier programming, higher performance, and better scalability than the current MPI+OpenACC model.

With the increasing availability of inexpensive yet powerful personal computers and servers, there has been an increasing trend toward personal computer cluster systems for network services. In this paper, we present a small-scale personal computer based cluster system for network servers as a low-cost alternative to traditional high-performance computing systems. The enhanced cluster system for scalable network services (CSSNS) consists of the parallel virtual file system (PVFS), the Linux virtual server (LVS), the director, and several high-end Pentium PC connected by high-speed switched Ethernet networks for I/O nodes and cluster nodes. The design and development of the cluster system for scalable network services is characterized by its high performance, high availability, high scalable file system, low cost, load balance on cluster servers, and convenient managing/setup for a broad range of applications. The transmission performance experiments for the entire system further validates and confirms that CSSNS is truly a self-made high-performance, high reliability and scalability personal computer based cluster system for scalable network servers.

The added value of artificial intelligence to LI-RADS categorization: A systematic review