Reducing energy and delay using efficient victim caches

This paper proposes methods for achieving high energy-delay efficiency in the embedded systems. Particularly, we present adaptive block management schemes for victim cache to reduce the number of accesses to more power consuming memory structures such as L2 caches. Victim cache is a memory element for reducing conflict misses in a direct-mapped L1 cache without affecting its access time. We investigate techniques to use victim cache more efficiently by selecting the blocks to be loaded into it based on the L1 cache history information. According to our simulations, proposed schemes show better performance than the conventional victim cache scheme and also reduce the power consumption.KeywordsComputer ArchitectureBlock ManagementVictim CacheLow Power

Learning the various structures and levels of memory hierarchy by means of conventional procedures is a complex subject. A memory hierarchy environment (Web-MHE) was proposed and developed as part of our undergraduate research, to serve not only as an auxiliary teaching tool for the Computer's Architecture professor, but also as a learning facility tool for the student carrying out his undergraduate or graduate courses. The environment encompasses the knowledge of the various levels of the memory hierarchy, and can be either used by students in his studies or by senior researchers. In conclusion, the Web-MHE offers the user the possibility to simulate, experiment and learn the various concepts of the memory hierarchy.

The emerging Die-stacking technology enables DRAM to be used as a cache to break the “Memory Wall” problem. Recent studies have proposed to use DRAM as a victim cache in both CPU and GPU memory hierarchies to improve performance. DRAM caches are large in size and, hence, when realized as a victim cache, non-inclusive design is preferred. This non-inclusive design adds significant differences to the conventional DRAM cache design in terms of its probe, fill, and writeback policies. Design and verification of a victim DRAM cache can be much more complex than that of a conventional DRAM cache. Hence, without rigorous modeling and formal verification, ensuring the correctness of such a system can be difficult. The major focus of this work is to show how formal modeling is applied to design and verify a victim DRAM cache. In this approach, we identify the agents in the victim DRAM cache design and model them in terms of interacting state machines. We derive a set of properties from the specifications of a victim cache and encode them using Linear Temporal Logic. The properties are then proven using symbolic and bounded model checking. Finally, we discuss how these properties are related to the dataflow paths in a victim DRAM cache.

The need for energy efficiency continues to grow for many classes of processors, including those for which performance remains vital. Data cache is crucial for good performance, but it also represents a significant portion of the processor's energy expenditure. We describe the implementation and use of a tagless access buffer (TAB) that greatly improves data access energy efficiency while slightly improving performance. The compiler recognizes memory reference patterns within loops and allocates these references to a TAB. This combined hardware/software approach reduces energy usage by (1) replacing many level-one data cache (L1D) accesses with accesses to the smaller, more power-efficient TAB; (2) removing the need to perform tag checks or data translation lookaside buffer (DTLB) lookups for TAB accesses; and (3) reducing DTLB lookups when transferring data between the L1D and the TAB. Accesses to the TAB occur earlier in the pipeline, and data lines are prefetched from lower memory levels, which result in a small performance improvement. In addition, we can avoid many unnecessary block transfers between other memory hierarchy levels by characterizing how data in the TAB are used. With a combined size equal to that of a conventional 32-entry register file, a four-entry TAB eliminates 40% of L1D accesses and 42% of DTLB accesses, on average. This configuration reduces data-access related energy by 35% while simultane-ously decreasing execution time by 3%.

In our prior work we explored a cache organization providing architectural support for distinguishing between memory references that exhibit spatial and temporal locality and mapping them to separate caches.That work showed that using separate (data) caches for indexed or stream data and scalar data items could lead to substantial improvements in terms of cache misses. In addition, such a separation allowed for the design of caches that could be tailored to meet the properties exhibited by different data items.In this paper, we investigate the interaction between three established methods: split cache, victim cache and stream buffer. Since significant amounts of compulsory and conflict misses are avoided, the size of each cache (i.e., array and scalar), as well as the combined cache capacity can be reduced. Our results show that on average 55% reduction in miss rates over the base configuration.

On-chip cache memories play an important role in resource-constrained embedded systems by filtering out most off-chip memory accesses. Because cache latency and energy consumption are generally proportional to cache sizes, a small cache at the top level of the memory hierarchy is desirable. Previous work has presented a novel cache architecture called a filter cache to reduce hit time and energy consumption of the L1 instruction cache. However, consideration to the data cache requires a different approach and has not been researched much. In this paper, we propose a filter data cache architecture to effectively adopt the filter cache to the data cache hierarchy. We observed that cache misses occur considerably and they are likely to be continuous when the filter cache is used for the data cache. Those misses cost performance and energy consumption by increasing cache latency and uploading unnecessary data. The proposed filter data cache architecture reduces miss costs using three schemes: early cache hit predictor (ECHP), locality-based allocation (LA), and No Tag Matching Write (NTW). Experimental results show that the proposed filter data cache reduces energy consumption of the data caches by 21% compared with the filter cache, and the energy consumption of the ALU by 27.2 percentage on average. The overheads in terms of area and leakage power are small and the proposed filter data cache architecture does not hurt performance.

The paper presents a performance model that can be used to optimally distribute computations over heterogeneous computers. This model is application-centric representing the speed of each computer by a function of the problem size. This way it takes into account the processor heterogeneity, the heterogeneity of memory structure, and the memory limitations at each level of memory hierarchy. A problem of optimal partitioning of ann-element set overpheterogeneous processors using this performance model is formulated, and its efficient solution of the complexity O(p3× log2n) is given.

SummaryOpenACC is a directive‐based programming model which allows programmers to write graphic processing unit (GPU) programs by simply annotating parallel loops. However, OpenACC has poor support for irregular nested parallel loops, which are natural choices to express nested parallelism. We propose PFACC, a programming model similar to OpenACC. PFACC directives can be used to annotate parallel loops and to guide data movement between different levels of memory hierarchy. Parallel loops can be arbitrarily nested or be placed inside functions that would be (possibly recursively) called in other parallel loops. The PFACC translator translates C programs with PFACC directives into CUDA programs by inserting runtime iteration‐sharing and memory allocation routines. The PFACC runtime iteration‐sharing routine is a two‐level mechanism. Thread blocks dynamically organize loop iterations into batches and execute the batches in a depth‐first order. Different thread blocks share iterations among one another with an iteration‐stealing mechanism. PFACC generates CUDA programs with reasonable memory usage because of the depth‐first execution order. The two‐level iteration‐sharing mechanism is implemented purely in software and fits well with the CUDA thread hierarchy. Experiments show that PFACC outperforms CUDA dynamic parallelism in terms of performance and code size on most benchmarks.

Among the beyond-complementary metal-oxide- semiconductor (CMOS) devices being explored, ferroelectric field-effect transistors (FeFETs) are considered as one of the most promising. FeFETs are being studied by all major semiconductor manufacturers, and experimentally, FeFETs are making rapid progress. FeFETs also stand out with the unique hysteretic Ids-Vgs characteristic that allows a device to function as both a switch and a nonvolatile (NV) storage element. We exploit this FeFET property to build two categories of fine-grained logic-in-memory (LiM) circuits: 1) ternary content addressable memory (TCAM) which integrates efficient and compact logic/processing elements into various levels of memory hierarchy; 2) basic logic function units for constructing larger and more complex LiM circuits. Two writing schemes (with and without negative supply voltages respectively) for FeFETs are introduced in our LiM designs. The resulting designs are compared with existing LiM approaches based on CMOS, magnetic tunnel junctions (MTJs), resistive random access memories (ReRAMs), ferrorelectric tunnel junctions (FTJs), etc., that afford the same circuit-level functionality. Simulation results show that FeFET-based NV TCAMs offer lower area overhead than MTJ (79%) and CMOS (42% less) equivalents, as well as better search energy-delay products (EDPs) than TCAM designs based on MTJ (149×), ReRAM (1.7×), and CMOS (1.3×) in array evaluations. NV FeFET-based LiM basic circuit blocks are also more efficient than functional equivalents based on MTJs in terms of propagation delay (4.2×) and dynamic power (2.5×). A case study for an FeFET-based LiM accumulator further demonstrates that by employing FeFET as both a switch and an NV storage element, the FeFET-based accumulator can save area (36%) and power consumption (40%) when compared with a conventional CMOS accumulator with the same structure.

Petascale computing will require coordinating the actions of 100,000+ processors, and directing the flow of data between up to six levels of memory hierarchy and along channels that differ by over a factor of 100 in bandwidth. Amdahl's law requires that petascale applications have less than 0.001% sequential or replicated work in order to be at least 50% efficient. These are profound challenges for all but the most regular or embarrassingly parallel applications, yet we also demand that not just bigger and better, but fundamentally new science. In this presentation I will discuss how we are attempting to confront simultaneously the complexities of petascale computation while increasing our scientific productivity. I hope that I can convince you that our development of MADNESS (multiresolution adaptive numerical scientific simulation) is not as crazy as it sounds.

Due to the increasing gap between processor speed and memory access time, a large fraction of a program's execution time is spent in accesses to the various levels in the memory hierarchy. Hence, cache-aware programming is of prime importance. For efficiently utilizing the memory subsystem, many architecture-specific characteristics must be taken into account: cache size, replacement strategy, access latency, number of memory levels, etc.In this paper, we present a simulator for the accurate performance prediction of sequential and parallel programs on shared memory systems. It assists the programmer in locating the critical parts of the code that have the greatest impact on the overall performance. Our simulator is based on the Latency-of-Data-Access Model, that focuses on the modeling of the access times to different memory levels.We describe the design of our simulator, its configuration and its usage in an example application.

This paper presents a simple dictionary structure designed for a hierarchical memory. The proposed data structure is cache oblivious and locality preserving. A cache-oblivious data structure has memory performance optimized for all levels of the memory hierarchy even though it has no memory-hierarchy-specific parameterization. A locality-preserving dictionary maintains elements of similar key values stored close together for fast access to ranges of data with consecutive keys.The data structure presented here is a simplification of the cache-oblivious B-tree of Bender, Demaine, and Farach-Colton. Like the cache-oblivious B-tree, this structure supports search operations using only O(logBN) block operations at a level of the memory hierarchy with block size B. Insertion and deletion operations use O(logBN + log2N/B) amortized block transfers. Finally, the data structure returns all k data items in a given search range using O(logBN + k/B) block operations.This data structure was implemented and its performance was evaluated on a simulated memory hierarchy. This paper presents the results of this simulation for various combinations of block and memory sizes.

Transactional Memory (TM) is an emerging technology which simplifies the concurrency control in a parallel program. In this paper we propose Quick TM, a new hardware transactional memory (HTM) architecture. It incorporates three features to address known bottlenecks in the existing HTM architectures. First, we propose hardware-only dynamic detection of true-shared variables. Our result shows that true-shared variables account for only about 20% in the commit set of any transaction. Rest can be completely disregarded from the commit phase. This shortens every commit phase drastically resulting in a significant overall speed-up. Second, we keep both the speculative and the last committed version local to each processor. This benefits when a transaction is repeated in a loop. The processor request gets satisfied from the L1 data cache(L1D) itself. Furthermore, since both the versions are locally maintained, the commit action involves only broadcast of addresses. Third, we have proposed a mechanism to address overflow in transactions. In our proposal, each processor continues to run transactions even if one processor has overflown its L1D. Our technique eliminates the stall of a thread even if it conflicts with the overflown transaction. Overflown transaction commits in-place and periodically broadcasts its write set addresses, termed “partial commit”. This gradually reduces conflicts and allows other threads to progress towards commit. Moreover, the technique does not require any additional hardware at any memory hierarchy level beyond L1. Quick TM outperforms the state-of-the-art scalable HTM architecture, Scalable-TCC, on average by 20% in the latest TM benchmark suite STAMP. It outperforms the original TCC proposal with serialized commit by 28% on average. Maximum speed-up achieved in these two cases are 43% and 67% respectively. Our proposal handles transaction overflow gracefully and outperforms the current overflow-aware HTM proposal, One TM-concurrent by 12% on average.

Energy efficiency has become the first priority when designing high-performance computing systems, and caches play a key role in improving the performance. However, as caches represent a dominant proportion of the total area in a processor, they dissipate a significant amount of the total energy consumption. A non-unified data cache architecture is suggested to tackle this issue without sacrificing performance. This architecture is based on the principle of providing a smaller cache for a group of frequently used references, which does not require a large cache to provide a high hit rate. Thus, the non-unified design reduces the number of accesses to a relatively large L1 data cache, thereby saving energy and enabling efficient performance. This paper examines the energy consumption of a non-unified data cache design as compared to that of a conventional unified data cache. As a powerful system requires allocation of a significant number of cores on a single die, minimizing the L1 caches in each core helps to incorporate more cores in one chip. Therefore, we conduct experiments for different sizes of L1 data caches, ranging from quite large to small. The experimental results show that compared to a conventional L1 data cache, the non-unified design reduces data cache dynamic energy consumption, on average, by up to 22% for a small data cache (such as a 4KB cache), and by up to 82% for a relatively large (such as a 32KB) cache.

Ubiquitous embedded systems are revolutionizing our daily lives. Whole systems on a chip deliver unprecedented computation power at ever decreasing costs. However, their complexity makes their design with traditional RTL-based flows extremely challenging. Complexity in such systems arises not only from the diversity of the technologies, from RF front-ends to baseband DSP software, that must be integrated on-chip, but also from the fact that such systems must be increasingly built from parts that have been designed separately and using different tools and flows. High abstraction levels and component reuse are essential. Often such systems are highly networked and rely on sophisticated communication mechanisms. Architectural design and performance analysis of such networked system-chips is a crucial part of the embedded system design process.Two basic methods for tackling the growing complexity of system-on-chip design are emerging. Formal specification and design methods, and platform-based design. Formal methods are essential to capture precisely the designer intent at the highest possible level of abstraction, and to guide him down towards an implementation. Design errors are reduced by the use of synthesis techniques, as well as by the availability of a variety of validation techniques. Moreover, the orthogonalization of concerns (e.g., computation vs. communication, function vs. architecture) promotes extensive reuse of both parts of an application (derivative designs) and of previously used architectures.A platform in this context is the precise specification of a high-level implementation target that supports both a variety of lower level implementations, and a variety of higher-level applications. A classical example is the PC, that has supported an explosive growth of both applications and implementations. In embedded systems-on-chip platform clearly need to be application-specific. A typical platform includes one or more processors, memory, one or more bus hierarchy levels, and peripherals devoted to application-specific functions (e.g., MPEG decoding, filtering, A/D conversion, ...). Mapping to a platform involves identifying which portions of the application will be implemented on each computation and communication resource. Communication must be refined from high-level primitives (e.g., FIFOs) down to platform primitives (e.g., interrupts, memory buffers, ...). Functionality must be synthesized and scheduled based on platform-specific cost, memory and performance constraints.In this tutorial we will cover both aspects in detail, illustrating first the languages and Models of Computation available to the system-level designer to capture precisely and unambiguously the requirements. We will discuss for what application domain and platform each language is most appropriate, focusing mostly on platform-independent languages and MOCs, since they support the greatest freedom in mapping choice. We will then discuss how the architecture of the platform, and the services it offers to the application designer, can also be formally and compactly captured and specified. We will show how the mapping paradigm can be used to select an implementation for the functional blocks and their communication, and how simulation and implementation methods can be derived automatically. In particular, we will describe how software estimation and synthesis for reactive real-time systems can be competitive with hand design, while retaining the ease of re-use typical of high-level specifications.In the next part of the tutorial, we will establish the importance of on-chip communication architectures in determining the performance of System-on-Chips (SoCs). We will track several on-chip communication architectures that are in use today, including different types of bus architectures, token ring architectures, and crossbar switches. We will analyze the performance of the existing communication architectures under different classes of on-chip communication traffic. We will also describe novel on-chip communication architectures that are being developed to satisfy the growing needs of fast and concurrent on-chip communication in very high-performance systems like network processors. Next, we will describe efficient methodologies that can be used to analyze and design customized on-chip communication architectures for new application-specific platforms, as well as optimally map an application's communication requirements to the communication architecture present in an existing platform. We will also discuss reconfigurable on-chip communication architectures that allow runtime adaptation of the communication protocols to changing communication demands of the platform. Besides performance, we will analyze the energy consumed by on-chip communication architectures, and describe techniques to minimize such energy consumption. Lastly, we will analyze the impact of deep sub-micron technologies on SoC communication, and discuss the needs for development of noise-aware on-chip communication.