Two-level main memory co-design: Multi-threaded algorithmic primitives, analysis, and simulation

Recently, dramatic improvements in memory performance have been highly required for data demanding application services such as deep learning, big data, and immersive videos. To this end, the throughput-oriented memory such as high bandwidth memory (HBM) and hybrid memory cube (HMC) has been introduced to provide a high bandwidth. For its effective use, various research efforts have been conducted. Among them, the near-memory-processing (NMP) is a concept that utilizes bandwidth and power consumption by placing computation logic near the memory. In the NMP-enabled system, a processor hierarchy consisting of hosts and NMPs is formed based on the distance from the main memory. In this paper, an evaluation tool is proposed to obtain the optimal design decision considering the power-time trade-off in the processor hierarchy. Every time the operating condition and constraints change, the decision of task-level offloading is dynamically made. For the realistic NMP-enabled system environment, the relationship among HBM, host, and NMP should be carefully considered. Hosts and NMPs are almost hidden from each other and the communications between them are extremely limited. In the simulation results, popular benchmarks and a machine learning application are used to demonstrate power-time trade-offs depending on applications and system conditions.

High Bandwidth Memory (HBM) Dynamic Random Access Memory (DRAM) has emerged as a preferred choice for leading-edge graphics, networking and highperformance computing applications. HBM is also finding usage in artificial intelligence (AI), machine learning (ML) and other advanced applications that demand high bandwidth and high power efficiency. HBM systems are implemented in 2.5D technology in which the memory stack and System-on-Chip (SoC) are integrated using a silicon interposer in a single package. Unlike conventional Double Data Rate (DDR) systems, the memory channel in 2.5D HBM systems are not accessible outside the package, posing multiple challenges in post-silicon characterization, system validation and SoC bring up. This paper discusses the functional at-speed test challenges and solutions using IEEE1500 based test structures implemented in the PHY, complementing those in DRAM. The paper also discusses a novel method to allow channel pin access for receiver and transmitter characterization with minimal impact to normal operation. Silicon results are presented at different levels of system hierarchy that consists of Memory Controller (MC), Physical Layer (PHY) and DRAM. Together, these results demonstrate excellent test coverage of the complete HBM memory subsystem and efficient silicon debug support.

Based on High Bandwidth Memory (HBM) structure and taking into consideration the direct interconnection between the micro-bump and the end of the Cu-filled TSV, there is an interaction between the micro-bump joint and the Cu protrusion. To clarify this issue, in this study two geometric finite element models (i.e., 4-hi stack HBM structure and 8-hi stack HBM structure) are constructed and used to characterize the von Mises stress, volume fraction of plastic deformation region of the Cu-filled TSV, height of Cu protrusion and fatigue life of micro-bump joints with taking into account the interaction between the micro-bump joint and Cu protrusion. The simulation results manifest that micro-bump joints in 4-hi stack HBM model suffer larger average stress than those in 8-hi stack HBM model, and the volume fraction of plastic deformation region of Cu-filled TSV in the former model is also higher than that in the latter one. Interestingly, the height of Cu protrusion becomes equal at upper and lower ends of Cu-filled TSVs in upper TSV layers of 8-hi stack HBM model. From fatigue property point of view, 8-hi stack HBM model is superior to 4-hi stack HBM model.

In this paper, we firstly proposed and electrical and thermal co-analyzed a novel through mold plate (TMP) for thermal management of high bandwidth memory (HBM). Thermal issues are bottlenecks in the development of HBM and 2.5-D/3-D ICs. The proposed TMP can be one of the promising solutions for advanced thermal management. Thermal coupling from the graphic processing unit (GPU) is the dominant factor for HBM thermal issues. A new thermal path can be achieved by installing the TMP in the epoxy molding compound (EMC) between GPU and HBM to significantly prevent thermal coupling from the GPU. By using 3D electromagnetic and fluent simulation, we verified the proposed TMP based on co-analysis of signal integrity (SI) and thermal integrity (TI). SI analysis considering both TMP and operating temperature provided the thermal and electrical superiority of TMP. As a result, the results showed the improvement of thermal integrity and signal integrity of the HBM by not only reducing HBM junction temperature by 10.3 °C but also decreasing jitter by 4.54 % compared to a conventional HBM graphic module.

Spurred by the industrial demands for terabyte/s bandwidth graphics module, high bandwidth memory (HBM) has been emerged to overcome the limitations of conventional DRAMs. Additionally, due to the fine pitch and high density interconnect routing between GPU and 4 HBMs in 2.5D terabyte/s bandwidth graphics module, HBM interposer has also been to the force. However, several signal integrity issues of the HBM interposer occur due to the manufacturing process constraints. In this paper, we design the HBM interposer using 6 layers redistribution layer (RDL) and TSVs in 2.5D terabyte/s bandwidth graphics module. And then, in the designed HBM interposer, electrical performance of the HBM interposer channels using M1, M3, and M5 layer is analyzed by simulation in the frequency-and time-domain. With the simulation results, it is observed that the designed HBM interposer shows good signal integrity.

Recent advancements in 3D-stacked DRAM such as hybrid memory cube (HMC) and high-bandwidth memory (HBM) promise higher bandwidth and lower power consumption compared to traditional DDR-based DRAM. However, taking advantage of this additional bandwidth for improving the performance of real-world applications requires carefully laying out the data in memory which incurs significant programmer effort. To alleviate this programmer burden, we investigate application-specific address mapping to improve performance while minimizing manual effort. Our approach is guided by the following insights: (i) toggling activity of address bits can help determine strategies to improve parallelism within memory but this metric underestimates conflicts and (ii) modern memory controllers reorder address requests and therefore any toggling activity measured from an address trace is non-deterministic. Furthermore, our position is that analyzing individual address bits results in poor estimates for actual conflicts and exploited parallelism and that entropy needs to be calculated for groups of address bits. Therefore, we calculate window-based probabilistic entropy for groups of address bits to determine a near-optimal address mapping. We present simulation results for ten applications that show a performance improvement up to 25% over fixed address-mapping and up to 8% over previous application-specific address mapping for our proposed approach.

Aquarius-II is a cache coherent multiprocessor system designed for the parallel execution of Prolog programs. It contains two tiers of memory: synchronization memory and high bandwidth (HB) memory. The synchronization memory consists of snooping caches connected to a bus and is used to store rendezvous points, synchronization bits, synchronization variables such as locks and semaphores and most of the write shared data. The HB memory is used to store the bulk of the application program code and data. It contains caches and an inexpensive VLSI chip based crossbar interconnection network to memory. The caches connected to the crossbar do not have full snooping capability. The architecture is evaluated by a full simulation of parallel execution of Prolog programs on Aquarius-II. The design details of the components of the architecture and simulation results are presented. Simulation results indicate that the two tier memory system significantly reduces memory interference and speeds up synchronization when compared to a single bus multi. This shared memory multiprocesor architecture has the potential to support other parallel programming paradigms.

In this paper, we firstly proposed a thermal transmission line (TTL) based embedded cooling structure for advanced thermal management of a next-generation high bandwidth memory (HBM) module. Thermal issues are critical to the development of HBM and 2.5D/3D ICs. The proposed TTL based embedded cooling structures can be one of the promising thermal management solutions for the 2.5D/3D ICs. The previous embedded cooling structures have thermal management limitations of the difficulties of cooling the internal heat of the 2.5D/3D ICs each layer. The proposed TTL transfers internal heat to the coolant to lowering junction temperature. Moreover, we checked the fabrication feasibility of the TTL with through silicon vias (TSVs). By using 3D electromagnetic (EM) and 3D fluent simulations, we analyzed the proposed TTL considering signal integrity (SI) and thermal integrity (TI). SI analysis showed the TTL does not contribute critical SI issues for HBM on-chip TSV channels. TI analysis provided the thermal management superiority of the TTL. As a result, it showed the improvement of TI of HBM module decreased HBM junction temperature by 4.789°C compared to the previous embedded cooling structure.

In this article, for the first time, we propose channel characteristic-based deep neural network (DNN) models for accurate eye-height (EH) and eye-width (EW) estimation of high bandwidth memory (HBM) silicon interposer channels. The proposed models preprocess the design parameters that are highly relevant to the characteristics of the HBM channels. By taking account of the contribution of each design parameter to the eye diagram, the proposed models can accurately estimate the EH and EW of the channels even with a limited number of datasets. For verification, the proposed DNN models were applied to the microstrip and stripline channels of an HBM silicon interposer. Only redistributed layer (RDL) was used to clearly see the effect of reflecting the channel characteristics of the proposed method. We compared the proposed DNN models with various regression methods and a conventional fully connected multilayer DNN model. As a result, the proposed DNN models reduced the EH and EW error rates by 22.7 and 43.9% compared to the other regression methods. In addition, the proposed DNN models not only reduced the error rates by 22.0–28.4% but also reduced the computing cost by 8.0–9.4%, compared to the conventional DNN model. Moreover, we compared the proposed models with various DNN models having other preprocessing structures. By showing 26.7 and 28.8% lower EH and EW error rates than the other DNN models, we validated that the proposed models properly consider the most dominant design factors in preprocessing.

Parallel computers now start to adopt Bandwidth-Asymmetric Memory architecture that consists of traditional DRAM memory and new High Bandwidth Memory (HBM) for high memory bandwidth. However, existing task schedulers suffer from low bandwidth usage and poor data locality problems in bandwidth-asymmetric memory architectures. To solve the two problems, we propose a Bandwidth and Locality Aware Task-stealing (BATS) system, which consists of an HBM-aware data allocator, a bandwidth-aware traffic balancer, and a hierarchical task-stealing scheduler. Leveraging compile-time code transformation and run-time data distribution, the data allocator enables HBM usage automatically without user interference. According to data access hotness, the traffic balancer migrates data to balance memory traffic across memory nodes proportional to their bandwidth. The hierarchical scheduler improves data locality at runtime without a priori program knowledge. Experiments on an Intel Knights Landing server that adopts bandwidth-asymmetric memory show that BATS reduces the execution time of memory-bound programs up to 83.5% compared with traditional task-stealing schedulers.

Parallel joins have been widely studied during the past decade and a number of efficient algorithms were presented. While it is known that the performance of these algorithms may suffer greatly in the presence of skewed input data, the work on load balancing schemes for parallel join has been limited. The main contribution of this paper is the development and analysis of a new distributed data structure and an effective load balancing scheme for parallel main memory hash join on NUMA architecture. Multiprocessors based on this architecture are scalable in both size of main memory and number of processors, and provide very high memory bandwidth. The load balancing scheme is based on random probing to avoid the hot spot problems caused by probing sequentially. We have modeled this load balancing scheme both analytically and experimentally. The experiments were run on a BBN TC2000 multiprocessor system.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

Modern computing systems are overwhelmingly designed to move data to computation. This design choice goes directly against at least three key trends in computing that cause performance, scalability and energy bottlenecks: (1) data access is a key bottleneck as many important applications are increasingly data-intensive, and memory bandwidth and energy do not scale well, (2) energy consumption is a key limiter in almost all computing platforms, especially server and mobile systems, (3) data movement, especially off-chip to on-chip, is very expensive in terms of bandwidth, energy and latency, much more so than computation. These trends are especially severely-felt in the data-intensive server and energy-constrained mobile systems of today. At the same time, conventional memory technology is facing many technology scaling challenges in terms of reliability, energy, and performance. As a result, memory system architects are open to organizing memory in different ways and making it more intelligent, at the expense of higher cost. The emergence of 3D-stacked memory plus logic, the adoption of error correcting codes inside the latest DRAM chips, proliferation of different main memory standards and chips, specialized for different purposes (e.g., graphics, low-power, high bandwidth, low latency), and the necessity of designing new solutions to serious reliability and security issues, such as the RowHammer phenomenon, are an evidence of this trend. This chapter discusses recent research that aims to practically enable computation close to data, an approach we call processing-in-memory (PIM). PIM places computation mechanisms in or near where the data is stored (i.e., inside the memory chips, in the logic layer of 3D-stacked memory, or in the memory controllers), so that data movement between the computation units and memory is reduced or eliminated. While the general idea of PIM is not new, we discuss motivating trends in applications as well as memory circuits/technology that greatly exacerbate the need for enabling it in modern computing systems. We examine at least two promising new approaches to designing PIM systems to accelerate important data-intensive applications: (1) processing using memory by exploiting analog operational properties of DRAM chips to perform massively-parallel operations in memory, with low-cost changes, (2) processing near memory by exploiting 3D-stacked memory technology design to provide high memory bandwidth and low memory latency to in-memory logic. In both approaches, we describe and tackle relevant cross-layer research, design, and adoption challenges in devices, architecture, systems, and programming models. Our focus is on the development of in-memory processing designs that can be adopted in real computing platforms at low cost. We conclude by discussing work on solving key challenges to the practical adoption of PIM.

GPUs execute thousands of active threads simultaneously, requiring high memory bandwidth to handle multiple memory requests efficiently. The memory bandwidth in GPUs has always been increasing, but it is still insufficient for the demands of fine-grained threads, necessitating a higher memory bandwidth. Important workloads like deep learning and data analytics demand terabytes of data processing, necessitating high memory capacity and bandwidth to avoid performance overheads. True-3D stacking of non-volatile memory layers on GPUs can provide the required higher bandwidth and capacity, enhancing performance and energy efficiency. We propose a high-bandwidth high-capacity hybrid 3D memory (H3DM) that doubles bandwidth through true-3D integration compared to the baseline GPU architecture and affords 272 GB of total memory capacity by stacking 8 PCM layers (each of 32 GB) and two DRAM layers (each of 8 GB).

Wide and slow bus interfaces such as High Bandwidth Memory (HBM) and Advanced Interface Bus (AIB) are the main drivers behind the development and implementation of technologies such as Embedded Multi-die Interconnect Bridge (EMIB) and Chip-on-Wafer-on-Substrate (CoWoS). These technologies can create very dense interconnect structures using back end of line (BEOL) approaches used to interconnect chiplets and to form functional system in package applications. With recent improvements in fabrication and process technologies organic based high density redistribution layer approaches such HD-FOWLP have become popular alternatives to EMIB and CoWoS. Redistribution layers with $2\text{um}\times 2\text{um}$ cross section and 2um space between adjacent lines or $1\text{um}\times 1\text{um}$ cross section with a minimum spacing of 1um have become achievable with the current fabrication technology. In this work the electrical design challenges in organic based fine RDL packages have been studied via simulation and electrical measurements.

Most data-intensive operations for multimedia applications such as image processing, vision, and 3D graphics require high external memory bandwidth. In augmented-reality (AR) processors [1], both 3D graphics and vision operations are required, so memory bandwidth becomes even more critical. In [1], however, memory bandwidth is not considered, floating-point processing is not supported, and there is no cache memory for texturing, which is a performance bottleneck of common graphics pipelines. In this work, a heterogeneous multimedia processor is presented to process various mobile multimedia applications in a single chip on Si-interposer for high memory bandwidth. The implemented processor has 4 key features: (1) A transceiver pool (TRx) that reconfigures strength of output drivers according to the channel loss for IC-stacking on Si-interposer, (2) A mode-configurable vector processing unit (MCVPU) for framelevel parallelism, (3) An energy-efficient unified filtering unit (UFU) with adaptive block selection (ABS) algorithm for memory-access-efficient texturing, and (4) a unified shader (US) with floating-point scalar processing elements (SPE) and partial special function units (PSFU) to enhance graphics processing performance and quality. With these techniques, we achieve 1.7× frame rate and 8× memory bandwidth improvement in full AR operation.