Impulse: building a smarter memory controller

: This report describes the design of the Impulse architecture and shows how an Impulse memory system can be used in a variety of ways to improve the performance of data-intensive applications. The Impulse design does not require any modification to processor, cache, or bus designs - all novel hardware functionality resides at the memory controller. As a result, Impulse optimizations can be adopted in conventional systems without major system changes. Impulse can be used to: (1) dynamically create superpages cheaply, (2) dynamically recolor physical pages, to perform strided fetches, (3) perform gathers and scatters through indirection vectors, and (4) dynamically gather cache lines from randomly dispersed data. Impulse improved the performance of six DARPA Data Intensive System (DIS) program Stressmarks from 1 .25X to 16X (or 470X in the case of in-place CornerTurn, which was unusually well-suited for Impulse). Impulse sped up the twenty-two programs in the benchmark suite by a geometric mean of 2.4X on 2002-class hardware, and 3.3X on 2007-class hardware. In addition to its applicability for data-intensive applications, Impulse can also be used by the OS for dynamic superpage creation, which is useful for arbitrary applications.

Image processing applications tend to access their data non-sequentially and reuse that data infrequently. As a result, they tend to perform poorly on conventional memory systems due to high cache and TLB miss rates and are particularly sensitive to the growing latency of main memory. In this paper, we analyze the memory performance of three image processing algorithms (volume rendering, image warping, and image filtering) on both a conventional memory system and on the Impulse memory system. The Impulse memory system allows application software to control how, when, and where data are loaded into a conventional processor cache. It does this by letting software configure how the memory controller interprets the physical addresses exported by the processor, which enables an application to dynamically change how data are fetched. Sparse data can be accessed densely, which improves both cache and TLB utilization, and memory latency is hidden by prefetching data within the memory controller. We find that for these image processing codes, using an Impulse memory system yields speedups of 40% to 226% over an otherwise identical machine with a conventional memory system.

Chapter 13 - DRAM Memory Controller

Modern computing systems are overwhelmingly designed to move data to computation. This design choice goes directly against at least three key trends in computing that cause performance, scalability and energy bottlenecks: (1) data access is a key bottleneck as many important applications are increasingly data-intensive, and memory bandwidth and energy do not scale well, (2) energy consumption is a key limiter in almost all computing platforms, especially server and mobile systems, (3) data movement, especially off-chip to on-chip, is very expensive in terms of bandwidth, energy and latency, much more so than computation. These trends are especially severely-felt in the data-intensive server and energy-constrained mobile systems of today. At the same time, conventional memory technology is facing many technology scaling challenges in terms of reliability, energy, and performance. As a result, memory system architects are open to organizing memory in different ways and making it more intelligent, at the expense of higher cost. The emergence of 3D-stacked memory plus logic, the adoption of error correcting codes inside the latest DRAM chips, proliferation of different main memory standards and chips, specialized for different purposes (e.g., graphics, low-power, high bandwidth, low latency), and the necessity of designing new solutions to serious reliability and security issues, such as the RowHammer phenomenon, are an evidence of this trend. This chapter discusses recent research that aims to practically enable computation close to data, an approach we call processing-in-memory (PIM). PIM places computation mechanisms in or near where the data is stored (i.e., inside the memory chips, in the logic layer of 3D-stacked memory, or in the memory controllers), so that data movement between the computation units and memory is reduced or eliminated. While the general idea of PIM is not new, we discuss motivating trends in applications as well as memory circuits/technology that greatly exacerbate the need for enabling it in modern computing systems. We examine at least two promising new approaches to designing PIM systems to accelerate important data-intensive applications: (1) processing using memory by exploiting analog operational properties of DRAM chips to perform massively-parallel operations in memory, with low-cost changes, (2) processing near memory by exploiting 3D-stacked memory technology design to provide high memory bandwidth and low memory latency to in-memory logic. In both approaches, we describe and tackle relevant cross-layer research, design, and adoption challenges in devices, architecture, systems, and programming models. Our focus is on the development of in-memory processing designs that can be adopted in real computing platforms at low cost. We conclude by discussing work on solving key challenges to the practical adoption of PIM.

On multicore systems contention for shared resources occurs when memory-intensive threads are co-scheduled on cores that share parts of the memory hierarchy, such as last-level caches and memory controllers. Previous work investigated how contention could be addressed via scheduling. A contention-aware scheduler separates competing threads onto separate memory hierarchy domains to eliminate resource sharing and, as a consequence, mitigate contention. However, all previous work on contention-aware scheduling assumed that the underlying system is UMA (uniform memory access latencies, single memory controller). Modern multicore systems, however, are NUMA, which means that they feature non-uniform memory access latencies and multiple memory controllers. We discovered that contention management is a lot more difficult on NUMA systems, because the scheduler must not only consider the placement of threads, but also the placement of their memory. This is mostly required to eliminate contention for memory controllers contrary to the popular belief that remote access latency is the dominant concern. In this work we quantify the effects on performance imposed by resource contention and remote access latency. This analysis inspires the design of a contention-aware scheduling algorithm for NUMA systems. This algorithm significantly outperforms a NUMA-unaware algorithm proposed before as well as the default Linux scheduler. We also investigate memory migration strategies, which are the necessary part of the NUMA contention-aware scheduling algorithm. Finally, we propose and evaluate a new contention management algorithm that is priority-aware.

Probabilistic Timing Analysis (PTA) reduces the amount of information needed to provide tight WCET estimates in real-time systems with respect to classic timing analysis. PTA imposes new requirements on hardware design that have been shown implementable for single-core architectures. However, no support has been proposed for multicores so far. In this paper, we propose several probabilistically-analysable bus designs for multicore processors ranging from 4 cores connected with a single bus, to 16 cores deploying a hierarchical bus design. We derive analytical models of the probabilistic timing behaviour for the different bus designs, show their suitability for PTA and evaluate their hardware cost. Our results show that the proposed bus designs (i) fulfil PTA requirements, (ii) allow deriving WCET estimates with the same cost and complexity as in single-core processors, and (iii) provide higher guaranteed performance than single-core processors, 3.4x and 6.6x on average for an 8-core and a 16-core setup respectively.

Memory system performance is an important factor in determining overall system performance. The design of key components of the memory system, such as the memory controller, becomes more important as memory performance becomes a limiting factor in high performance computing. This work focuses on the design of a unit which sends control signals to Double Data Rate Synchronous Dram (DDR SDRAM). The design is based on established concepts such as access reordering. A novel, adaptive page policy based on a machine learning algorithm has been developed in this work and evaluated with traditional page policies. the work illustrates some of the design trade-offs in a memory controller and the performance of the designs when using real application address traces.The results show that access reordering improves the performance of DDR SDRAM compared to in-order scheduling (up to 50% improvement) and that scheduling multiple requests can result in latency hiding. The dynamic page policy approximates the best static page policy in most cases.

Consider constrained-deadline sporadic tasks scheduled on a multiprocessor where (i) each task is characterized by its execution requirement, deadline, and minimum inter-arrival time, (ii) each task generates a sequence of jobs, (iii) the execution requirement of a job and its potential for parallel execution is described by one or many stages with a stage having one or many segments and different segments in the same stage can execute in parallel and a segment is only allowed to start execution if all segments of previous stages have finished execution, and (iv) there is contention for shared resources in the memory system (cache eviction, reordering in memory controller, memory bus contention). We present an algorithm that (i) performs schedulability testing for tasks scheduled with global-Earliest-Deadline-First (gEDF), (ii) configures the virtual-to-physical address translation so that a cache block fetched to the last-level cache by one task cannot be evicted by another task, (iii) configures the virtual-to-physical address translation to attempt to eliminate the extra execution time caused by the reordering effect in the memory controller and if this is not possible, then the reordering effect is considered in the schedulability analysis, and (iv) considers the effect of contention for the memory bus. Our solution is based on formulating this problem as a Mixed-Integer Linear Program (MILP). We have implemented a tool based on this theory and validated its output against measurements on a real computer.

Cores in a chip-multiprocessor (CMP) system share multiple hardware resources in the memory subsystem. If resource sharing is unfair, some applications can be delayed significantly while others are unfairly prioritized. Previous research proposed separate fairness mechanisms in each individual resource. Such resource-based fairness mechanisms implemented independently in each resource can make contradictory decisions, leading to low fairness and loss of performance. Therefore, a coordinated mechanism that provides fairness in the entire shared memory system is desirable. This paper proposes a new approach that provides fairness in the entire shared memory system , thereby eliminating the need for and complexity of developing fairness mechanisms for each individual resource. Our technique, Fairness via Source Throttling (FST), estimates the unfairness in the entire shared memory system. If the estimated unfairness is above a threshold set by system software, FST throttles down cores causing unfairness by limiting the number of requests they can inject into the system and the frequency at which they do. As such, our source-based fairness control ensures fairness decisions are made in tandem in the entire memory system. FST also enforces thread priorities/weights, and enables system software to enforce different fairness objectives and fairness-performance tradeoffs in the memory system. Our evaluations show that FST provides the best system fairness and performance compared to four systems with no fairness control and with state-of-the-art fairness mechanisms implemented in both shared caches and memory controllers.

Chapter 5 - Memory Systems

A large number of criteria have been proposed for determining when a behavior has become automatic. Almost all of these were developed before the widespread acceptance of multiple memory systems. Consequently, popular frameworks for studying automaticity often neglect qualitative differences in how different memory systems guide initial learning. Unfortunately, evidence suggests that automaticity criteria derived from these frameworks consistently misclassify certain sets of initial behaviors as automatic. Specifically, criteria derived from cognitive science mislabel much behavior still under the control of procedural memory as automatic, and criteria derived from animal learning mislabel some behaviors under the control of declarative memory as automatic. Even so, neither set of criteria make the opposite error-that is, both sets correctly identify any automatic behavior as automatic. In fact, evidence suggests that although there are multiple memory systems and therefore multiple routes to automaticity, there might nevertheless be only one common representation for automatic behaviors. A number of possible cognitive and cognitive neuroscience models of this single automaticity system are reviewed. WIREs Cogn Sci 2012, 3:363-376. doi: 10.1002/wcs.1172 For further resources related to this article, please visit the WIREs website.

Thanks to recent progresses in optical device integration, nanophotonic interconnect can be applied to build optical network-on-chip (ONoC) for providing connectivity among increasing number of cores and memory modules in future many-core processors. However, there are still challenges in architecture and protocol design of ONoC for massive parallel memory accesses in CMPs, The on-chip memory access traffic exhibits heterogeneity for different nodes, i.e. Cores and memory controllers and different memory packets. This paper analyses on-chip communication characteristics and proposes a novel ONoC named SAMNoC. In SAMNoC, memory controllers are able to make self-arbitration for network resources to support node heterogeneity, and different packets are transmitted in dedicated sub-networks to optimize memory traffic. The memory request sub-network applies efficient global communication via slot and coded tokens to exploit bank level parallelism and the response sub-network can eliminate network bottleneck and process memory transactions quickly when the data is ready. Simulation results on real applications show that our SAMNoC can significantly improve memory performance, compared with other bus-based ONoC alternatives, with lower static power consumption and less number of optical waveguides and micro-rings.

Today's systems are overwhelmingly designed to move data to computation. This design choice goes directly against at least three key trends in systems that cause performance, scalability and energy bottlenecks: 1) data access from memory is already a key bottleneck as applications become more data-intensive and memory bandwidth and energy do not scale well, 2) energy consumption is a key constraint in especially mobile and server systems, 3) data movement is very expensive in terms of bandwidth, energy and latency, much more so than computation [1-3]. These trends are especially severely-felt in the data-intensive server and energy-constrained mobile systems of today. At the same time, conventional memory technology is facing many scaling challenges in terms of reliability, energy, and performance [4-5]. As a result, memory system architects are open to organizing memory in different ways and making it more intelligent, at the expense of slightly higher cost. The emergence of 3D-stacked memory plus logic [6], the adoption of error correcting codes inside the latest DRAM chips [7-8], and intelligent memory controllers to solve the RowHammer problem [9-11] are an evidence of this trend. In this talk, I will discuss some recent research that aims to practically enable computation close to data. After motivating trends in applications as well as technology, we will discuss at least two promising directions: 1) performing massively-parallel bulk operations in memory by exploiting the analog operational properties of DRAM, with low-cost changes [3, 12-14, 23], 2) exploiting the logic layer in 3D-stacked memory technology in various ways to accelerate important data-intensive applications [2, 15-22]. In both approaches, we will discuss relevant cross-layer research, design, and adoption challenges in devices, architecture, systems, applications, and programming models [1]. Our focus will be the development of in-memory processing designs that can be adopted in real computing platforms and real data-intensive applications, spanning machine learning, graph processing, data analytics, and genome analysis, at low cost [26]. If time permits, we will also discuss simulation and evaluation infrastructures that can enable exciting and forward-looking research in future memory systems, including Ramulator [24] and SoftMC [25].

Requirements for a new Cray Inc. computer system mean that the system's memory cards have high-speed SerDes interfaces and multiple memory controllers in a semicustom IC close to the memory chips. As a result, the functionality of a card is much greater than is current practice, and the cards can not be connected to existing memory testers. An embedded test system is designed for the card that can test all aspects of the card including the SerDes paths, a large L3 cache, the memory controllers, and the memory parts mounted on the cards, with testing driven from a JTAG port. The test capability is based in a microcoded controller, with data packets being generated and checked rather than directly controlling the logic being exercised. Some of the test implementation discussed here is patent pending.

Conventional main memory can no longer meet the requirements of low energy consumption and massive data storage in an artificial intelligence Internet of Things (AIoT) system. Moreover, the efficiency is decreased due to the swapping of data between the main memory and storage. This paper presents a hybrid storage class memory system to reduce the energy consumption and optimize IO performance. Phase change memory (PCM) brings the advantages of low static power and a large capacity to a hybrid memory system. In order to avoid the impact of poor write performance in PCM, a migration scheme implemented in the memory controller is proposed. By counting the write times and row buffer miss times in PCM simultaneously, the write-intensive data can be selected and migrated from PCM to dynamic random-access memory (DRAM) efficiently, which improves the performance of hybrid storage class memory. In addition, a fast mode with a tmpfs-based, in-memory file system is applied to hybrid storage class memory to reduce the number of data movements between memory and external storage. Experimental results show that the proposed system can reduce energy consumption by 46.2% on average compared with the traditional DRAM-only system. The fast mode increases the IO performance of the system by more than 30 times compared with the common ext3 file system.