Locality aware memory assignment and tiling

Multi-level caches are widely used to improve the memory access speed of multiprocessor systems. Deciding on a suitable set of cache memories for an application specific embedded system's memory hierarchy is a tedious problem, particularly in the case of MPSoCs. To accurately determine the number of hits and misses for all the configurations in the design space of an MPSoC, researchers extract the trace first using Instruction set simulators and then simulate using a software simulator. Such simulations take several hours to months. We propose a novel method based on specialized hardware which can quickly simulate the design space of cache configurations for a shared memory multiprocessor system on an FPGA, by analyzing the memory traces and calculating the cache hits and misses simultaneously. We demonstrate that our simulator can explore the cache design space of a quad-core system with private L <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub> caches and a shared L <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sub> cache, over a range of standard benchmarks, taking as less as 0.106 seconds per million memory accesses, which is up to 456 times faster than the fastest known software based simulator. Since we emulate the program and analyze memory traces simultaneously, we eliminate the need to extract multiple memory access traces prior to simulation, which saves a significant amount of time during the design stage.

State-of-the-art applications, such as convolutional neural networks, demand specialized hardware accelerators that address performance and efficiency constraints. An efficient memory hierarchy is mandatory for such hardware systems. While the memory architectures of general-purpose processors (e.g., CPU or GPUs) are based on cache systems, dedicated accelerators have mostly adopted the DMA (Direct Memory Access) concept due to the application field of image processing. DMA features like 2D data transfers or data padding can optimize the memory accesses of image processing. However, DMA lacks the capability to exploit temporal and spatial data reuse, a feature common in cache systems, particularly when multiple DMAs operate in parallel. This article proposes a novel Direct Cached Memory Access (DCMA) architecture, combining both DMA and cache methodologies and their respective advantages. Optimized for image-based AI algorithms, the DCMA architecture facilitates enhanced memory access by integrating multiple, parallel DMA ports with caching capabilities. This design allows for efficient data reuse and parallel memory access. Optimal parameters for the DCMA are determined through a comprehensive design space exploration. The DCMA is evaluated on a state-of-the-art Xilinx UltraScale+ FPGA board coupled with a massive-parallel vertical vector co-processor, called V 2 PRO. The results show the mitigation of the vector processor’s memory bottleneck. By using the proposed DCMA, speedups of up to ×17 for the ResNet-50 CNN can be achieved.

The performance of modern processors is often limited by execution stalls resulting from long memory access latencies. Compile-time optimizations, deep cache hierarchies and prefetching mechanisms already provide significant performance gains, by performing memory accesses in parallel with computation. However, they are reaching a throughput improvement limit. Hence, new solutions that effectively exploit the memory access patterns to improve processing throughput are required. To achieve this objective, a new compiler-assisted data streaming method is proposed. It leverages static analysis and code transformations with an on-chip data streaming support as a viable alternative to prefetching mechanisms for regular code structures. Static analysis is used to identify and encode memory accesses with a dedicated representation. Then, a code transformation algorithm detaches data indexation and address calculation from computation, allowing for a significant code reduction. An on-chip data stream controller, attached to the L1 data cache, is used to autonomously generate memory accesses from the pattern representation and reorganize the data transfers in streams, with the aid of stream buffers. When compared with state-of-the-art prefetchers, the proposed solution provides up to 26 percent of code reduction, an IPC improvement of 2.4x, and an average performance improvement of 40 percent.

Due to limited battery power that devices have, an efficient DVFS (Dynamic Voltage and Frequency Scaling) algorithm is important. DVFS applies not only to the CPU, also the shared cache and memory associated with the CPU. The CPU and these memory hierarchy interact with each other, however, their DVFS methods are operated independently. Although there have been many discussions to reduce the memory access latency of CPU, the conventional DVFS technique does not relieve memory stall (the state during which CPU is stalled waiting for a memory access) effectively due to individual DVFS methods. In this article, we propose a new DVFS method, which alleviates the memory stall and reduces CPU power consumption utilizing microarchitectural information. Our proposed method identifies the layer on the memory hierarchy that mainly generates the memory stall and determines the optimal frequency of CPU and memory hierarchy that achieve a system-wide power optimization, without degrading the performance. In our experiments, our proposed method improved IPC (Instruction Per Cycle) by 11%, reduced MSPC (Memory Stall Per Cycle) by 26% and reduced power consumption by 5% compared to a conventional DVFS method.

Two performance gaps in the memory hierarchy, between CPU cache and main memory, and main memory and mass storage, will become increasingly severe bottlenecks for computing-system performance. Although it is necessary to increase memory capacity to fill these gaps, power also increases when conventional volatile memories are used. A new nonvolatile memory for this purpose has been anticipated. Storage class memory is used to fill the second gap. Many candidates exist: ReRAM, PRAM, and 3D-cross point type with resistive change RAM. However, nonvolatile last level cache (LLC) is used to fill the first gap. Advanced STT-MRAM has achieved sub-4ns read and write accesses with perpendicular magnetic tunnel junctions (p-MTJ) [1–2]. Furthermore, mature integration processes have been developed and 8Mb STT-MRAM with sub-5ns operation has shown high reliability [3]. Moreover, because of its non-volatility, STT-MRAM can reduce operation energy by more than 81% compared to SRAM for cache [1]. This paper presents STT-MRAM-based last level cache memory (LLC) including MRAM memory core, peripherals and cache logic circuits, using novel power optimization with high-speed power gating (HS-PG),considering processor architectures and cache memory accesses. The STT-MRAM-based cache has high reliability to reduce the write-error rate with novel write-verify-write. Furthermore, a read-modify-write scheme is implemented to reduce active power without penalty. Figure 7.2.1 presents a block diagram of a 4Mb STT-MRAM based cache.

Current high-performance multicore processors provide users with a non-uniform memory access model (NUMA). These systems perform better when threads access data on memory banks next to the core where they run. However, ensuring data locality is difficult. In this paper, we propose compiler analyses and code generation methods to support a lightweight runtime system that dynamically migrates memory pages to improve data locality. Our technique combines static and dynamic analyses and is capable of identifying the most promising pages to migrate. Statically, we infer the size of arrays, plus the amount of reuse of each memory access instruction in a program. These estimates rely on a simple, yet accurate, trip count predictor of our own design. This knowledge lets us build templates of dynamic checks, to be filled with values known only at runtime. These checks determine when it is profitable to migrate data closer to the processors where this data is used. Our static analyses are quadratic on the number of variables in a program, and the dynamic checks are O(1) in practice. Our technique does not require any form of user intervention, neither the support of a third-party middleware, nor modifications in the operating system's kernel. We have applied our technique on several parallel algorithms, which are completely oblivious to the asymmetric memory topology, and have observed speedups of up to 4x, compared to static heuristics. We compare our approach against Minas, a middleware that supports NUMA-aware data allocation, and show that we can outperform it by up to 50% in some cases.

In-memory Key-Value stores (IMKVs) provide significantly higher performance than traditional disk-based counterparts. As memory technologies advance, IMKVs become practical for modern Big Data processing, which include financial services, e-commerce, telecommunication network, etc. Recently, various IMKVs have been proposed from both academia and industrial. In order to leverage high performance random access capability of main memory, most IMKVs employ hashing based index structures to retrieve data according to keys. Consequently, a regular memory access pattern can be observed in data retrieval from those IMKVs. Normally speaking, one access to index (hash table), which is also located in main memory, is followed by another memory access to value data. Such a regular access pattern provides a potential opportunity that data prefetching techniques can be employed to improve memory access efficiency for data retrieval in these IMKVs. Based on this observation, we explore various data prefetching techniques with proper architecture level modifications on memory controller considering trade-off between design overhead and performance. Specifically, we focus on two key design issues of prefetching techniques: (1) where to fetch data (i.e. data address)? and (2) how many data to fetch (i.e. data size)? Experimental results demonstrate that memory access performance can be substantially improved up to 35.4 %. In addition, we also demonstrate the overhead of prefetching on power consumption.

Data-parallel programs are both growing in importance and increasing in diversity, resulting in specialized processors targeted at specific classes of these programs. This paper presents a classification scheme for data-parallel program attributes, and proposes micro-architectural mechanisms to support applications with diverse behavior for using a single reconfigurable architecture. We focus on the following four broad kinds of data-parallel programs - DSP/multimedia, scientific, networking, and real-time graphics workloads. While all of these programs exhibit high computational intensity, coarse-grain regular control behavior, and some regular memory access behavior, they show wide variance in the computation requirements, fine grain control behavior, and the frequency of other types of memory accesses. Based on this study of application attributes, this paper proposes a set of general micro-architectural mechanisms that enable a baseline architecture to be dynamically tailored to the demands of a particular application. These mechanisms provide efficient execution across a spectrum of data-parallel application and can be applied to diverse architectures ranging from vector cores to conventional superscalar cores. Our results using a baseline TRIPS processor show that the configurability of the architecture to the application demands provides harmonic mean performance improvement of 5%-55% over scalable yet less flexible architectures, and performs competitively against specialized architectures.

Exploiting runtime memory access traces can be a complementary approach to compiler optimizations for the energy reduction in memory hierarchy. This is particularly important for emerging multimedia applications since they usually have input-sensitive runtime behavior which results in dynamic and/or irregular memory access patterns. These types of applications are normally hard to optimize by static compiler optimizations. The reason is that their behavior stays unknown until runtime and may even change during computation. To tackle this problem, we propose an integrated approach of software [compiler and operating system (OS)] and hardware (data access record table) techniques to exploit data reusability of multimedia applications in Multiprocessor Systems on Chip. Guided by compiler analysis for generating scratch pad data layouts and hardware components for tracking dynamic memory accesses, the scratch pad data layout adapts to an input data pattern with the help of a runtime scratch pad memory manager incorporated in the OS. The runtime data placement strategy presented in this paper provides efficient scratch pad utilization for the dynamic applications. The goal is to minimize the amount of accesses to the main memory over the entire runtime of the system, which leads to a reduction in the energy consumption of the system. Our experimental results show that our approach is able to significantly improve the energy consumption of multimedia applications with dynamic memory access behavior over an existing compiler technique and an alternative hardware technique.

The multiplication of the number of cores inside embedded systems has raised the pressure on the memory hierarchy. The cost of coherence protocol and the scalability problem of the memory hierarchy is nowadays a major issue. In this paper, a specific data management for read-only data is investigated because these data can be duplicated in several memories without being tracked. Based on analysis of standard benchmarks for embedded systems, this analysis shows that read-only data represent 62% of all the data used by applications and 18% of all the memory accesses. A specific data path for read-only data is then evaluated by using simulations. On the first level of the memory hierarchy, removing read-only data of the L1 cache and placing them in another read-only cache improve the data locality of the read-write data by 30% and decrease the total energy consumption of the first level memory by 5%.

Shared memory architectures have recently experienced a large increase in thread-level parallelism, leading to complex memory hierarchies with multiple cache memory levels and memory controllers. These new designs created a Non-Uniform Memory Access (NUMA) behavior, where the performance and energy consumption of memory accesses depend on the place where the data is located in the memory hierarchy. Accesses to local caches or memory controllers are generally more efficient than accesses to remote ones. A common way to improve the locality and balance of memory accesses is to determine the mapping of threads to cores and data to memory controllers based on the affinity between threads and data. Such mapping techniques can operate at different hardware and software levels, which impacts their complexity, applicability, and the resulting performance and energy consumption gains. In this article, we introduce a taxonomy to classify different mapping mechanisms and provide a comprehensive overview of existing solutions.

This paper presents a hybrid cache analysis for the simulation-based evaluation of data caches in embedded systems. The proposed technique uses static analyses at the machine code level to obtain information about the control flow of a program and the memory accesses contained in it. Using the result of these analyses, a high-speed source-level simulation model is generated from the source code of the application, enabling a fast and accurate evaluation of its data cache behavior. As memory accesses are obtained from the binary-level control flow, which is simulated in parallel to the original functionality of the software, even complex compiler optimizations can be modeled accurately. Experimental results show that the presented source-level approach estimates the cache behavior of a program within the same level of accuracy as established techniques working at the machine code level.

In this paper, we present a novel HitME (Hit-MEmory) buffer to reduce the energy consumption of memory hierarchy in embedded processors. The HitME buffer is a small direct-mapped cache memory that is added as additional memory into existing cache memory hierarchies. The HitME buffer is loaded only when there is a hit on L1 cache. Otherwise, L1 cache is updated from the memory and the processor's memory request is served directly from the L1 cache. The strategy works due to the fact that 90% of memory accesses are only accessed once, and these often pollute the cache. Energy reduction is achieved by reducing the number of accesses to the L1 cache memory. Experimental results show that the use of HitME buffer will reduce the L1 cache accesses resulting in a reduction in the energy consumption of the memory hierarchy. This decrease in L1 cache accesses reduces the cache system energy consumption by an average of 60.9% when compared to traditional L1 cache memory architecture and an energy reduction of 6.4% when compared to filter cache architecture for 70nm cache technology.

A pipelined 8 bit-serial single-flux-quantum microprocessor integrating cache memories, calledCORE1γ, has been designedand implemented. The CORE1γ has 16 byte and 8 byte shift-register-based cache memories for instructions and for data,respectively. The microprocessor overlaps four instructions in pipeline executions. Thepipeline stages are optimized to control the cache memories and a new circuit component isdeveloped to manage the pipeline execution according to whether each cache hits or misses.Advanced passive transmission line techniques with use of the cell library of the2.5 kA cm−2 niobium standard process are utilized to implement theCORE1γ accompanied by extremely complex interconnects. TheCORE1γ is made up of 295 passive interconnects, and 22302 Josephson junctions are integrated on a6.36 × 6.36 mm2 area. The logic simulation shows that the peak performance is estimated at 1000 millionoperations per second, and it is kept even during memory access as long as the cachememories hit.

Modern processors rely on advanced memory management techniques such as predicting and prefetching to bridge the processor–memory gap. However, current mechanisms are primarily designed for a specific memory hierarchy level or access pattern, resulting in inefficiencies for diverse workloads with mixed memory access patterns. To address this challenge, this paper proposes DHCM, a Dynamic Hierarchy Coordination Mechanism that intelligently schedules prediction hierarchies and dynamically optimizes memory access processes to enhance system performance. DHCM integrates a hierarchy coordination mechanism driven by a State Trigger. This mechanism dynamically leverages system feedback to prioritize and coordinate memory operations, enabling the simultaneous management of both off-chip load requests and on-chip cache accesses. Through extensive evaluations on the ChampSim simulator, DHCM demonstrates its adaptability and efficiency with an average IPC improvement of 34.08% and 24.09% on single-core and multi-core systems, respectively. Additionally, DHCM contributes a 64.17% miss coverage and 89.33% DRAM-loads reduction.