Towards Application-Specific Address Mapping for Emerging Memory Devices

Two-level main memory co-design: Multi-threaded algorithmic primitives, analysis, and simulation

GPGPUs utilize multi-dimensional memory subsystems to provide the bandwidth needed by their multi-dimensional parallelism. However, an unfavorable address mapping leads to imbalanced memory request distribution across the memory resources, causing degraded performance and poor power efficiency. The optimal mapping is both application- and hardware-dependent. This paper provides a software-hardware co-design to dynamically reconfigure the address mapping according to the trace of the targeted application. First, a circuit to sample the entropy of address bits is proposed to capture the optimal address mapping. Second, a dynamic reconfiguration mechanism is designed to apply the optimal address mapping. Simulation results show up to 45% performance improvement over fixed address mappings.

Recently, dramatic improvements in memory performance have been highly required for data demanding application services such as deep learning, big data, and immersive videos. To this end, the throughput-oriented memory such as high bandwidth memory (HBM) and hybrid memory cube (HMC) has been introduced to provide a high bandwidth. For its effective use, various research efforts have been conducted. Among them, the near-memory-processing (NMP) is a concept that utilizes bandwidth and power consumption by placing computation logic near the memory. In the NMP-enabled system, a processor hierarchy consisting of hosts and NMPs is formed based on the distance from the main memory. In this paper, an evaluation tool is proposed to obtain the optimal design decision considering the power-time trade-off in the processor hierarchy. Every time the operating condition and constraints change, the decision of task-level offloading is dynamically made. For the realistic NMP-enabled system environment, the relationship among HBM, host, and NMP should be carefully considered. Hosts and NMPs are almost hidden from each other and the communications between them are extremely limited. In the simulation results, popular benchmarks and a machine learning application are used to demonstrate power-time trade-offs depending on applications and system conditions.

The recent advent of stacked memory devices has led to a resurgence of research associated with the fundamental memory hierarchy and associated memory pipeline. The bandwidth advantages provided by stacked logic and DRAM devices have inspired research associated with eliminating the bandwidth bottlenecks associated with many applications in high performance computing. Further, recent efforts have focused on utilizing stacked memory devices as last-level caches. In addition to the two aforementioned focus areas, a third area of research is emerging to explore augmenting the stacked memory logic layer with additional operations. This first generation of Hybrid Memory Cube (HMC) devices provided rudimentary atomic memory operations. The Gen2 Hybrid Memory Cube devices provide more expressive atomic memory operations that include primitive integer arithmetic operations. Despite the inclusion of more expressive arithmetic operations, many users have expressed interest in more complex and potentially orthogonal custom memory cube, or CMC, operations in future revisions of the Hybrid Memory Cube specification. This work presents recent development associated with the HMC-Sim Hybrid Memory Cube simulation framework that provides users a powerful infrastructure to experiment and research augmented custom memory cube, or CMC, operations within the current Gen2 Hybrid Memory Cube device infrastructure. We provide an overview of extending the original HMC-Sim simulation infrastructure to include support for CMC operations with requiring users to modify the core simulation code base. We also present three examples of building and utilizing custom, user-defined CMC operations in sample simulations to exhibit potential application speedup with future HMC device specifications. In doing so, we present a model to replace traditional thread mutexes with custom HMC mutex commands.

GPUs are being widely used to accelerate different workloads and multi-GPU systems can provide higher performance with multiple discrete GPUs interconnected together. However, there are two main communication bottlenecks in multi-GPU systems -- accessing remote GPU memory and the communication between GPU and the host CPU. Recent advances in multi-GPU programming, including unified virtual addressing and unified memory from NVIDIA, has made programming simpler but the costly remote memory access still makes multi-GPU programming difficult. In order to overcome the communication limitations, we propose to leverage the memory network based on hybrid memory cubes (HMCs) to simplify multi-GPU memory management and improve programmability. In particular, we propose scalable kernel execution (SKE) where multiple GPUs are viewed as a single virtual GPU as a single kernel can be executed across multiple GPUs without modifying the source code. To fully enable the benefits of SKE, we explore alternative memory network designs in a multi-GPU system. We propose a GPU memory network (GMN) to simplify data sharing between the discrete GPUs while a CPU memory network (CMN) is used to simplify data communication between the host CPU and the discrete GPUs. These two types of networks can be combined to create a unified memory network (UMN) where the communication bottleneck in multi-GPU can be significantly minimized as both the CPU and GPU share the memory network. We evaluate alternative network designs and propose a sliced flattened butterfly topology for the memory network that scales better than previously proposed alternative topologies by removing local HMC channels. In addition, we propose an overlay network organization for unified memory network to minimize the latency for CPU access while providing high bandwidth for the GPUs. We evaluate trade-offs between the different memory network organization and show how UMN significantly reduces the communication bottleneck in multi-GPU systems.

The initial location of data in DRAMs is determined and controlled by the 'address-mapping' and even modern memory controllers use a fixed and run-time-agnostic address mapping. On the other hand, the memory access pattern seen at the memory interface level will dynamically change at run-time. This dynamic nature of memory access pattern and the fixed behavior of address mapping process in DRAM controllers, implied by using a fixed address mapping scheme, means that DRAM performance cannot be exploited efficiently. DReAM is a novel hardware technique that can detect a workload-specific address mapping at run-time based on the application access pattern which improves the performance of DRAMs. The experimental results show that DReAM outperforms the best evaluated address mapping on average by 9%, for mapping-sensitive workloads, by 2% for mapping-insensitive workloads, and up to 28% across all the workloads. DReAM can be seen as an insurance policy capable of detecting which scenarios are not well served by the predefined address mapping.

Processing-In-Memory (PIM) architectures based on recent technology advances (e.g., Hybrid Memory Cube) demonstrate great potential for graph processing. However, existing solutions did not address the key challenge of graph processing---irregular data movements.

Memory cubes (MCs), following the general concept of Micron's Hybrid Memory Cube (HMC), represent a promising memory architecture for high scalability and energy efficiency due to their partitioned 3D-stacked DRAM, high-speed serial links, abstract packet-switched interface, and on-die switching fabric connecting the host processor with the MC's different partitions ('vaults'). While previous studies have shown that implementing processor-to-MC links with silicon-photonic (SiP) integrated optical links offers higher energy efficiency and bandwidth density, they keep the electrical switching fabric inside the MC die and perform signal conversion prior to routing packets across the switch. We believe that the technological limitations of electrical interconnects in terms of energy consumption, the large size of MC dies, the high radix of the on-die switch, and the bandwidth demands will all ultimately turn the on-MC switching fabric into a critical issue in terms of energy and latency. Using an integrated optical switching fabric alleviates all of these issues and allows the host processor to directly communicate with the MC vaults by exploiting wavelength routing, thereby eliminating the need for electrical switch traversal. In particular, we propose to use Arrayed Waveguide Grating Routers (AWGRs) which offer a compact SiP switching fabric with a connectivity pattern that is ideal as the on-MC switch. Our simulation results show that exploiting AWGRs and direct processor-to-vault communication reduces both MC access energy and latency by up to 40% (on average) on PARSEC/SPLASH-2 workloads.

To meet the ever-increasing demand of global data traffic, the Hybrid Memory Cube (HMC) specification and High-Bandwidth Memory standard call for an aggregate bandwidth of 8 Tb/s in next-generation memory-to-processor links [1], [2]. At the same time, the data rates processed by individual line cards in data center switches will soon go beyond 25 Tb/s [3]. However, as the data rate of serial links over copper interconnects increases, the links appear highly lossy. Further, the power consumption of a wireline transceiver, including all clocking and equalization circuits, increases. Publications suggest a tenfold increase in power consumption with a 30-dB increase in channel loss [4]. Hence, energyefficient communication equipment should be engineered to keep the channel loss as low as possible.

Modern FPGAs targeting data centers are designed to accelerate problems with large data. They offer many different types of memory including on-chip and on-board memories. A recent addition is High Bandwidth Memory (HBM), whose advantages have been demonstrated by others. However, there is little research that looks at how interactions among different memory types impact application performance. We investigate how a combination of HBM and on-chip memory (BRAM or URAM) impact clock rate and overall application latency. In these designs, the on-chip memory is used as an on-chip cache for the larger amounts of data stored in HBM. Our experiments show that as the size of data stored in BRAM or URAM increases, the achievable clock speed is reduced. This in turn may result in degraded performance. We examine Garbled Circuits, an implementation of Secure Function Evaluation (SFE) with high memory demands and out-of-order data access, and examine how different choices of BRAM, URAM and HBM usage alters its performance.

The tremendous growth in smartphones and tablets has been fueled by consumer demand for increased mobility, functionality, and ease of use. This, in turn, has been driving an increase in functional convergence and 3D integration of integrated circuit (IC) devices, resulting in the need for more advanced and sophisticated packaging techniques. In particular, the integration of the application processor (AP) and dedicated memory has become increasingly complex. Smartphones and tablets have become virtual streaming devices requiring low power, high bandwidth memory (HBM). For quality, high-speed video and multi-tasking applications, the memory interface to the AP must have superior signal integrity to minimize crosstalk and insertion/return losses. In addition, the thermal management of the processor must allow the maximum clock speed and duty cycle for high-performance applications. Finally, to conserve battery life for the mobility device, the power distribution to the processor must be as efficient as possible. This creates a significant challenge for the semiconductor packaging industry to meet these demanding requirements for smartphone and tablet products. This paper introduces a leading-edge high-density fan-out (HD-FO) semiconductor packaging technology that addresses the need for higher levels of integration and improved electrical and thermal performance for mobile applications. This new innovative technique, called Silicon Wafer Integrated Fan-out Technology (SWIFT™) packaging, leverages the fine feature size and thin-film circuit patterning capabilities of wafer-level packaging. In addition, thin-film dielectric photolithography and pattern plating provides a significant reduction in package z-height, which is critical for advanced mobile devices. By incorporating a redistribution layer (RDL)-first/chip-last process flow, there is also an opportunity for yield optimization and cycle-time reduction. This work compares the electrical and thermal modeling results between a conventional Package-on-Package (PoP) and HD-FO SWIFT devices. The results reveal a highly integrated PoP structure with exceptional electrical, mechanical, and thermal performance benefits – compared to conventional organic laminate-based technologies – to meet the growing need for high-performance mobile applications.

Two hardware buffer architectures for digital state feedback controllers based on controllable canonical model are presented. One is linear buffer architecture and the other is circular buffer architecture. The architectures take advantage of cascaded structure of the canonical form to simplify state calculations. At each sampling time, only the n-th state needs to be re-calculated. This makes the architecture a lot simpler than full state calculation method. For linear buffer architecture, a new "backtapping" technique is introduced to sneak in state shifting during multiplication. For circular buffer architecture, "address mapping" is used to map linear state addresses to coefficient addresses. The buffer architectures simplify hardware realizations of modern state space controllers and provide low-cost low-power solutions to multiple control systems needed in applications such as self-driving and robotics technologies.

NVM (Non-Volatile Memory) that gains attention as an alternative memory technology these days has different memory characteristics in terms of latency and bandwidth, compared with the traditional DRAM. In this paper, we analyze how these different characteristics affect the performance of applications. For this purpose, we design a new memory analysis framework that can control the memory latency and bandwidth through hardware and software techniques in commercial computer systems. The framework allows an application to be executed as it is without any modification, leading to fast analysis under diverse configurations. In addition, we devise a novel memory performance model that investigates performance with the viewpoint of LLC (Last Level Cache) and MLP (Memory Level Parallelism). Real implementation based experiments show that our model analyzes performance changes accurately when we compare with the results measured by the proposed framework

Parallel computers now start to adopt Bandwidth-Asymmetric Memory architecture that consists of traditional DRAM memory and new High Bandwidth Memory (HBM) for high memory bandwidth. However, existing task schedulers suffer from low bandwidth usage and poor data locality problems in bandwidth-asymmetric memory architectures. To solve the two problems, we propose a Bandwidth and Locality Aware Task-stealing (BATS) system, which consists of an HBM-aware data allocator, a bandwidth-aware traffic balancer, and a hierarchical task-stealing scheduler. Leveraging compile-time code transformation and run-time data distribution, the data allocator enables HBM usage automatically without user interference. According to data access hotness, the traffic balancer migrates data to balance memory traffic across memory nodes proportional to their bandwidth. The hierarchical scheduler improves data locality at runtime without a priori program knowledge. Experiments on an Intel Knights Landing server that adopts bandwidth-asymmetric memory show that BATS reduces the execution time of memory-bound programs up to 83.5% compared with traditional task-stealing schedulers.

Inspired by recent advances in multistatic radar systems, the problem of sensor placement is investigated. Our study is motivated by the fact that it is not always clear what the placement of the radars giving the best performance for the targets of interest might be. To account for the issue, we derive the multistatic Cramer-Rao lower bounds (CRLBs) for range and velocity estimation as the fitness function. Then we use genetic algorithms (GAs) to perform the optimized sensor placement as one of global optimization. The simulation example indicates that our proposed approach is a flexible and effective tool and is capable of suggesting optimal sensor placement strategies to meet required radar performance goals.