Evaluating hybrid memory cube infrastructure to support high-performance sparse algorithms

The recent advent of stacked memory devices has led to a resurgence of research associated with the fundamental memory hierarchy and associated memory pipeline. The bandwidth advantages provided by stacked logic and DRAM devices have inspired research associated with eliminating the bandwidth bottlenecks associated with many applications in high performance computing. Further, recent efforts have focused on utilizing stacked memory devices as last-level caches. In addition to the two aforementioned focus areas, a third area of research is emerging to explore augmenting the stacked memory logic layer with additional operations. This first generation of Hybrid Memory Cube (HMC) devices provided rudimentary atomic memory operations. The Gen2 Hybrid Memory Cube devices provide more expressive atomic memory operations that include primitive integer arithmetic operations. Despite the inclusion of more expressive arithmetic operations, many users have expressed interest in more complex and potentially orthogonal custom memory cube, or CMC, operations in future revisions of the Hybrid Memory Cube specification. This work presents recent development associated with the HMC-Sim Hybrid Memory Cube simulation framework that provides users a powerful infrastructure to experiment and research augmented custom memory cube, or CMC, operations within the current Gen2 Hybrid Memory Cube device infrastructure. We provide an overview of extending the original HMC-Sim simulation infrastructure to include support for CMC operations with requiring users to modify the core simulation code base. We also present three examples of building and utilizing custom, user-defined CMC operations in sample simulations to exhibit potential application speedup with future HMC device specifications. In doing so, we present a model to replace traditional thread mutexes with custom HMC mutex commands.

The Hybrid Memory Cube (HMC) is a promising alternative to DDRx memory due to its potential to achieve significantly higher bandwidth. However, the high static power of an HMC device compromises power efficiency when the device is lightly utilized. Activating a sleeping HMC takes over 2µs, which makes it challenging to manage HMC power without a substantial degradation in system performance. We introduce a new technique that alleviates the long wakeup penalty of an HMC by employing erasure codes. Inaccessible data stored in a sleeping HMC module can be reconstructed by decoding related data retrieved from other active HMCs, rather than waiting for the sleeping HMC module to become active. This approach makes it possible to tolerate the latency penalty incurred when switching an HMC between active and sleep modes, thereby enabling a power-capped HMC system. Simulations show that the proposed architecture outperforms a current HMC-based multicore system by 6.2×, and reduces the system energy by 5.3× under the same power budget as the multicore baseline.

Hybrid Memory Cube (HMC) is a revolutionary standard in DRAM architecture based on 3D integration. It provides marvelous concurrency and reduced latency. HMC uses CRC32 for data integrity, but conventional Serial CRC calculation is very slow and has long latencies, here we propose three methods to implement parallel CRC to be very fast. The first method uses symbolic toolbox in MATLAB to generate the final equations of the CRC, and then these equations are exported to VERILOG so that we are able to calculate it in only one clock cycle. The second method is depending on using an existing tool that can generate parallel CRC but this tool has a limitation on the input data width as it is less than the maximum allowed data width in HMC which is 1152 bits, so we were able to find a work around method that enable us to calculate CRC32 for large data widthwith this tool. The third method is based on using the polynomial mathematics for CRC, as the CRC can be calculated using long division method. Method 1 latency is one clock cycle, Method 2 latency is 2 clock cycles, and method 3 latency is 37 clock cycles compared to serial CRC which latency is 1152 clock cycles.

One of the main challenges for embedded systems is the transfer of data between memory and processor. In this context, Hybrid Memory Cubes (HMCs) can provide substantial energy and bandwidth improvements compared to traditional memory organizations, while also allowing the execution of simple atomic instructions in the memory. However, the complex memory hierarchy still remains a bottleneck, especially for applications with a low reuse of data, limiting the usable parallelism of the HMC vaults and banks. In this paper, we introduce the HIVE architecture, which allows performing common vector operations directly inside the HMC, avoiding contention on the interconnections as well as cache pollution. Our mechanism achieves substantial speedups of up to 17.3× (9.4× on average) compared to a baseline system that performs vector operations in a 8-core processor. We show that the simple instructions provided by HMC actually hurt performance for streaming applications.

One of the main challenges for computer architects is how to hide the high average memory access latency from the processor. In this context, Hybrid Memory Cubes (HMCs) can provide substantial energy and bandwidth improvements compared to traditional memory organizations. However, it is not clear how this reduced average memory access latency will impact the LLC. For applications with high cache miss ratios, the latency to search for the data inside the cache memory will impact negatively on the performance. The importance of this overhead depends on the memory access latency. In this paper, we present an evaluation of the L3 cache importance on a high performance processor using HMC also exploring chip area tradeoffs between the cache size and number of processor cores. We show that the high bandwidth provided by HMC memories can eliminate the need for L3 caches, removing hardware and making room for more processing power. Our evaluations show that performance increased 37% and the EDP improved 12% while maintaining the same original chip area in a wide range of parallel applications, when compared to DDR3 memories.

As throughputs of digital networks and memory interfaces are on a constant rise, there is a need for ever-faster implementations of error-detecting codes. Cyclic redundancy checks (CRC) are a common and widely used type of codes to ensure consistency or detect accidental changes of transferred data. We propose a novel FPGA architecture for the computation of the CRC values designed for general high-speed data transfers. Its key feature is allowing a processing of multiple independent data packets (transactions) in each clock cycle, what is a necessity for achieving high overall throughput on very wide data buses. The proposed approach can be effectively used in Ethernet MACs for different speeds, in Hybrid Memory Cube (HMC) controller, and in many other technologies utilizing any kind of CRC. Experimental results confirm that the proposed architecture enables reaching an effective throughput sufficient for utilization in multi-terabit Ethernet networks (over 2 Tbps or over 3000 Mpps) on a single Xilinx UltraScale+ FPGA. Furthermore, a better utilization of FPGA resources is achieved compared to existing CRC implementation for HMC controller (up to 70% savings).

Memory cubes (MCs), following the general concept of Micron's Hybrid Memory Cube (HMC), represent a promising memory architecture for high scalability and energy efficiency due to their partitioned 3D-stacked DRAM, high-speed serial links, abstract packet-switched interface, and on-die switching fabric connecting the host processor with the MC's different partitions ('vaults'). While previous studies have shown that implementing processor-to-MC links with silicon-photonic (SiP) integrated optical links offers higher energy efficiency and bandwidth density, they keep the electrical switching fabric inside the MC die and perform signal conversion prior to routing packets across the switch. We believe that the technological limitations of electrical interconnects in terms of energy consumption, the large size of MC dies, the high radix of the on-die switch, and the bandwidth demands will all ultimately turn the on-MC switching fabric into a critical issue in terms of energy and latency. Using an integrated optical switching fabric alleviates all of these issues and allows the host processor to directly communicate with the MC vaults by exploiting wavelength routing, thereby eliminating the need for electrical switch traversal. In particular, we propose to use Arrayed Waveguide Grating Routers (AWGRs) which offer a compact SiP switching fabric with a connectivity pattern that is ideal as the on-MC switch. Our simulation results show that exploiting AWGRs and direct processor-to-vault communication reduces both MC access energy and latency by up to 40% (on average) on PARSEC/SPLASH-2 workloads.

Two-level main memory co-design: Multi-threaded algorithmic primitives, analysis, and simulation

To meet the ever-increasing demand of global data traffic, the Hybrid Memory Cube (HMC) specification and High-Bandwidth Memory standard call for an aggregate bandwidth of 8 Tb/s in next-generation memory-to-processor links [1], [2]. At the same time, the data rates processed by individual line cards in data center switches will soon go beyond 25 Tb/s [3]. However, as the data rate of serial links over copper interconnects increases, the links appear highly lossy. Further, the power consumption of a wireline transceiver, including all clocking and equalization circuits, increases. Publications suggest a tenfold increase in power consumption with a 30-dB increase in channel loss [4]. Hence, energyefficient communication equipment should be engineered to keep the channel loss as low as possible.

Recent advancements in 3D-stacked DRAM such as hybrid memory cube (HMC) and high-bandwidth memory (HBM) promise higher bandwidth and lower power consumption compared to traditional DDR-based DRAM. However, taking advantage of this additional bandwidth for improving the performance of real-world applications requires carefully laying out the data in memory which incurs significant programmer effort. To alleviate this programmer burden, we investigate application-specific address mapping to improve performance while minimizing manual effort. Our approach is guided by the following insights: (i) toggling activity of address bits can help determine strategies to improve parallelism within memory but this metric underestimates conflicts and (ii) modern memory controllers reorder address requests and therefore any toggling activity measured from an address trace is non-deterministic. Furthermore, our position is that analyzing individual address bits results in poor estimates for actual conflicts and exploited parallelism and that entropy needs to be calculated for groups of address bits. Therefore, we calculate window-based probabilistic entropy for groups of address bits to determine a near-optimal address mapping. We present simulation results for ten applications that show a performance improvement up to 25% over fixed address-mapping and up to 8% over previous application-specific address mapping for our proposed approach.

Recently, dramatic improvements in memory performance have been highly required for data demanding application services such as deep learning, big data, and immersive videos. To this end, the throughput-oriented memory such as high bandwidth memory (HBM) and hybrid memory cube (HMC) has been introduced to provide a high bandwidth. For its effective use, various research efforts have been conducted. Among them, the near-memory-processing (NMP) is a concept that utilizes bandwidth and power consumption by placing computation logic near the memory. In the NMP-enabled system, a processor hierarchy consisting of hosts and NMPs is formed based on the distance from the main memory. In this paper, an evaluation tool is proposed to obtain the optimal design decision considering the power-time trade-off in the processor hierarchy. Every time the operating condition and constraints change, the decision of task-level offloading is dynamically made. For the realistic NMP-enabled system environment, the relationship among HBM, host, and NMP should be carefully considered. Hosts and NMPs are almost hidden from each other and the communications between them are extremely limited. In the simulation results, popular benchmarks and a machine learning application are used to demonstrate power-time trade-offs depending on applications and system conditions.

Memory bandwidth and latency constitutes a major performance bottleneck for many data-intensive applications. While high-locality access patterns take advantage of the deep cache hierarchies available in modern processors, unpredictable low-locality patterns cause a significant part of the execution time to be wasted waiting for data. An example of those memory bound applications is the exact matching algorithm based on FM-index, used in some well-known sequence alignment applications. Processing-Near-Memory (PNM) has been proposed as a strategy to overcome the memory wall problem, by placing computation close to data, speeding up memory bound workloads by reducing data movements. This paper presents a performance and energy evaluation of two classes of processor architectures when executing the FM-index exact matching algorithm, as a reference algorithm for exact sequence alignment. One architecture class is processor-centric, based on complex cores and DDR3/4 SDRAM memory technology. The other architecture class is memory-centric, based on simple cores and ultra-high-bandwidth hybrid memory cube (HMC) 3D-stacked memory technologies. The results show that the PNM solution improves performance between 1.26 $$\times$$ and 3.7 $$\times$$ and the energy consumption per operation is reduced between 21 $$\times$$ and 40 $$\times$$ . In addition, a synthetic benchmark RANDOM was developed that mimics the memory access pattern of the FM-index exact matching algorithm, but with a user configurable operational intensity. This benchmark allows us to extend the evaluation to the class of algorithms with similar memory behaviour but running over a range of operational intensity values.

Semiconductor industry, for more than four decades, has rigorously followed Moore's Law in scaling down the CMOS technologies. Although several new materials and processes are being developed to address the challenges of future technology nodes, in the coming years they will be limited with respect to functionalities that future devices will require. As a consequence a clear trend of moving from CMOS to package and system architecture can be observed. Three-dimensional (3D) technology using the well-known Through Silicon Via (TSV) interconnect is one the emerging option, considered today the most advanced technology, that could enable various heterogeneous integration. Indeed such technology is not limited to the CMOS scaling in itself, it is rather based on bringing more functionalities by stacking different type of devices (Logic, Memory, Analog, MEMS, Passive component...) while reducing the form factor of the packaging. This functional diversification is also known as More-than-Moore. In addition, considering Known Good Die approach, each component of the 3D package could have a different manufacturer using different wafer sizes and node technology, thus bringing more complexity but also more opportunities and responsibilities to the supply chain. There are several business models identified, either using vertical integration or collaborative approach, if a dominant one will emerge or several tactics will co-exist, it is still remains a key question that need to be answered. The supply chain interaction and key players will be addressed in this presentation, including current and future standardization needs. This is today a key for the manufacturing of advanced 3D devices. 3D integration is considered today a new paradigm for the semiconductor industry, since it will drive evolution for packages for the coming decades. Due to several advantages that TSV technology can bring, several platforms have started. 3D WLCSP, 2.5D interposers & 3DIC are the main platforms that will be studied in this paper. Market forecasts in terms of wafer starts, market revenue, segments and end-products as well as supply chain activities and major player interactions will be presented. The industry has enthusiastically been waiting for mass production of 3D ICs. Although some small level of production has already been reported, the adoption rate in high volume manufacturing (HVM) is still low due to unresolved challenges that the industry still needs to address. Process technology is not fully mature, there are still many challenges in bonding and de-bonding, testing as well as thermal management that have to be overcome. Furthermore, design tools have to be fully released to enable proper 3D integration design. Looking at the time to market it is foreseen that device such as the Hybrid Memory Cube, combining high-speed logic with a multiple stacks of TSV bonded memories, will come into high volume production in 2014. This will definitely change the world of the memory market and will significantly speed up the adoption of 3D technologies. Technology roadmaps for 3D integration will also be included in the manuscript and reviewed during the presentation.

GPUs are being widely used to accelerate different workloads and multi-GPU systems can provide higher performance with multiple discrete GPUs interconnected together. However, there are two main communication bottlenecks in multi-GPU systems -- accessing remote GPU memory and the communication between GPU and the host CPU. Recent advances in multi-GPU programming, including unified virtual addressing and unified memory from NVIDIA, has made programming simpler but the costly remote memory access still makes multi-GPU programming difficult. In order to overcome the communication limitations, we propose to leverage the memory network based on hybrid memory cubes (HMCs) to simplify multi-GPU memory management and improve programmability. In particular, we propose scalable kernel execution (SKE) where multiple GPUs are viewed as a single virtual GPU as a single kernel can be executed across multiple GPUs without modifying the source code. To fully enable the benefits of SKE, we explore alternative memory network designs in a multi-GPU system. We propose a GPU memory network (GMN) to simplify data sharing between the discrete GPUs while a CPU memory network (CMN) is used to simplify data communication between the host CPU and the discrete GPUs. These two types of networks can be combined to create a unified memory network (UMN) where the communication bottleneck in multi-GPU can be significantly minimized as both the CPU and GPU share the memory network. We evaluate alternative network designs and propose a sliced flattened butterfly topology for the memory network that scales better than previously proposed alternative topologies by removing local HMC channels. In addition, we propose an overlay network organization for unified memory network to minimize the latency for CPU access while providing high bandwidth for the GPUs. We evaluate trade-offs between the different memory network organization and show how UMN significantly reduces the communication bottleneck in multi-GPU systems.

Processing-In-Memory (PIM) architectures based on recent technology advances (e.g., Hybrid Memory Cube) demonstrate great potential for graph processing. However, existing solutions did not address the key challenge of graph processing---irregular data movements.