Increasing hardware data prefetching performance using the second-level cache

This paper discusses the potential of an on-chip cache memory for modern vector supercomputers. The vector supercomputers can achieve the high computational efficiency for compute-intensive scientific applications. The most important factor affecting the computational performance is high memory bandwidth to provide a sufficient amount of data to the rich arithmetic units in time; the modern vector supercomputers such as NEC SX-7 and SX-8 have 4 bytes per flop (4B/FLOP) on the ratio of memory bandwidth to floating-point operations. However, the gap in performance between memory and processors has become remarkably exposed year by year in high performance computing. Therefore, it is getting harder to keep the 4B/FLOP memory bandwidth in design of future vector supercomputers. As a promising solution to cover a lack of the memory bandwidths of vector load/store units of the future vector supercomputers, we design an on-chip vector cache for the NEC SX vector processor architecture. This paper evaluates the performance of the on-chip cache memory system on the SX-7 system with 2B/FLOP or lower memory bandwidth across two kernel loops and five leading scientific applications. The results of the kernel loops demonstrate that a 2B/FLOP memory system with the on-chip cache whose hit ratio is 50% can achieve a performance comparable to that of a 4B/FLOP system without the cache. The results of the four applications indicate that the on-chip cache can improve sustained performance of the four applications by 20% to 98%. The experimental results regarding the last one show a conflicting effect of loop unrolling with vector caching, resulting in a poor hit rate. However, when loop-unrolling is disabled, its cache hit rate is improved, and the sustained performance comparable to that of the 4B/FLOP memory bandwidth without the loop-unrolling is obtained. In addition, selective caching, in which only a part of data with the high locality of reference are cached, is also effective for efficient use of the limited cache capacity.

It is well known that memory latency is a major deterrent to achieving the maximum possible performance of a today's high speed RISC processors. Techniques to reduce or tolerate large memory latencies become essential for achieving high processor utilization. Many methods, ranging from software to hardware solutions, have been studied with varying amounts of success. Most techniques have concentrated on data prefetching. However, our simulations show that the CPU is stalled up to 50% of the time waiting for instructions. The instruction memory latency reduction technique typically used in CPU designs today is the one block look-ahead (OBL) method. In this thesis, I present a new hardware prefetching scheme based on dynamic interpretation of the instruction stream. This is done by adding a small pipeline to the cache that scans forward in the instruction stream interpreting each instruction and predicting the future execution path. It then prefetches what it predicts the CPU will be executing in the near future. The pipelined prefetching engine has been shown to be a very effective technique for decreasing the instruction stall cycles in typical on-chip cache memories used today. It performs well, yielding reductions in stall cycles up to 30% or more for both scientific and general purpose programs, and has been shown to reduce the number of instruction stall cycles as compared to the OBL technique as well. The idea of sub-line prefetching was also studied and presented. It was thought that prefetching full cache lines might present too much overhead in terms of bus bandwidth, so prefetches should only fill partial cache lines instead. However it was determined that prefetching partial cache lines does not show any benefit when dealing with cache lines smaller than 128 bytes.

This 3rd-generation, superscalar processor, implementing the SPARC V9 64 b architecture, improves performance over previous processors by improvements in the on-chip memory system and circuit designs enhancing the speed of critical paths beyond the process entitlement. In the on-chip memory system, both bandwidth and latency are scaled. Keys to scaling memory latency are a sum-addressed memory data cache, which allows the average memory latency to scale by more than the clock ratio, and the use of a prefetch data cache. Memory bandwidth is improved by using wave-pipelined SRAM designs for on-chip caches and a write cache for store traffic. The chip operates at 800 MHz and dissipates <60 W from a 1.5 V supply. It contains 23 M transistors (12 M in RAM cells) on a 244mm/sup 2/ die. This paper contrasts this 7-metal-layer-aluminum, 0.15 /spl mu/m CMOS design with the previous generations designs. To deal with the growing microprocessor complexity, more aggressive circuit-techniques, interconnect delay optimization, crosstalk reduction, improved power and clock distribution schemes, and better thermal management are used.

Recent I/O technologies such as PCI-Express and 10Gb Ethernet enable unprecedented levels of I/O bandwidths in mainstream platforms. However, in traditional architectures, memory latency alone can limit processors from matching 10 Gb inbound network I/O traffic. We propose a platform-wide method called Direct Cache Access (DCA) to deliver inbound I/O data directly into processor caches. We demonstrate that DCA provides a significant reduction in memory latency and memory bandwidth for receive intensive network I/O applications. Analysis of benchmarks such as SPECWeb9, TPC-W and TPC-C shows that overall benefit depends on the relative volume of I/O to memory traffic as well as the spatial and temporal relationship between processor and I/O memory accesses. A system level perspective for the efficient implementation of DCA is presented.

The trace cache is a recently proposed solution to achieving high instruction fetch bandwidth by buffering and reusing dynamic instruction traces. This work presents a new block-based trace cache implementation that can achieve higher IPC performance with more efficient storage of traces. Instead of explicitly storing instructions of a trace, pointers to blocks constituting a trace are stored in a much smaller trace table. The block-based trace cache renames fetch addresses at the basic block level and stores aligned blocks in a block cache. Traces are constructed by accessing the replicated block cache using block pointers from the trace table. Performance potential of the block-based trace cache is quantified and compared with perfect branch prediction and perfect fetch schemes. Comparing to the conventional trace cache, the block-based design can achieve higher IPC, with less impact on cycle time.Results: Using the SPECint95 benchmarks, a 16-wide realistic design of a block-based trace cache can improve performance 75% over a baseline design and to within 7% of a baseline design with perfect branch prediction. With idealized trace prediction, it is shown the block-based trace cache with an 1K-entry block cache achieves the same performance of the conventional trace cache with 32K entries.

Artificial intelligence Internet of Things (AIoT) systems need more processing capability and low energy consumption on the terminals. Increasing the cache capacity at the architecture level can effectively reduce the cache missing rate and the number of off-chip memory access, which improves the performance of processors. Because of the limitation of the storage density and high leakage currents, improving SRAM cache capacity will lead to big chip area and high power consumption of processors. Emerging non-volatile memory (NVM), such as spin-transfer torque RAM (STT-RAM), has some characters such as bit width read/write and short read/write latency which are attractive options for replacing or augmenting SRAM. However, the asymmetric read/write latency of NVM brings some new challenges in designing caches. In this paper, we introduce a method to bring STT-RAM into the cache system of AIoT processors. Experimental results show that if we replace SRAM L2 cache with the same sized STT-RAM L2 cache, the out-of-order processor performance is improved up to 10 percent. We first find that for the STT-RAM cache, although the better-performing stream-based data prefetch configuration can improve the processor performance, the amount of prefetching data also increases obviously, which has negative effect on the processor performance because of the cache congestion caused by STT-RAM's long write latency. This paper presents a novel stream-based prefetch method ANCP (Adaptive Non-volatile Cache Prefetch) to reduce the amount of prefetch-write on STT-RAM. Compared with the best-performing stream-based data prefetch configuration, ANCP reduces the prefetching issued by 11 percent on average, which obtains additional up to 8 percent performance improvement of the AIoT processor.

Memory latency and bandwidth are progressing at a much slower pace than processor performance. In this paper, we describe and evaluate the performance of three variations of a hardware function unit whose goal is to assist a data cache in prefetching data accesses so that memory latency is hidden as often as possible. The basic idea of the prefetching scheme is to keep track of data access patterns in a reference prediction table (RPT) organized as an instruction cache. The three designs differ mostly on the timing of the prefetching. In the simplest scheme (basic), prefetches can be generated one iteration ahead of actual use. The lookahead variation takes advantage of a lookahead program counter that ideally stays one memory latency time ahead of the real program counter and that is used as the control mechanism to generate the prefetches. Finally the correlated scheme uses a more sophisticated design to detect patterns across loop levels. These designs are evaluated by simulating the ten SPEC benchmarks on a cycle-by-cycle basis. The results show that 1) the three hardware prefetching schemes all yield significant reductions in the data access penalty when compared with regular caches, 2) the benefits are greater when the hardware assist augments small on-chip caches, and 3) the lookahead scheme is the preferred one cost-performance wise.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

The gap between processor and memory performance has become a focal point for microprocessor research and development over the past three decades. Modern architectures use two orthogonal approaches to help alleviate this issue: (1) Almost every microprocessor includes some form of on-chip storage, usually in the form of caches, to decrease memory latency and make more effective use of limited memory bandwidth. (2) Massively multithreaded architectures, such as graphics processing units (GPUs), attempt to hide the high latency to memory by rapidly switching between many threads directly in hardware. This paper explores the intersection of these two techniques. We study the effect of accelerating highly parallel workloads with significant locality on a massively multithreaded GPU. We observe that the memory access stream seen by on-chip caches is the direct result of decisions made by the hardware thread scheduler. Our work proposes a hardware scheduling technique that reacts to feedback from the memory system to create a more cache-friendly access stream. We evaluate our technique using simulations and show a significant performance improvement over previously proposed scheduling mechanisms. We demonstrate the effectiveness of scheduling as a cache management technique by comparing cache hit rate using our scheduler and an LRU replacement policy against other scheduling techniques using an optimal cache replacement policy.

This paper addresses the scheduling problem for multi-dimensional loops applications on heterogeneous multicore processors. In the multi-dimensional loops scheduling problem, a significant issue is how to hide memory latency to reduce the schedule length. With the increasing CPU speed, the gap between the processor and memory performance is an important bottleneck for modern high-performance computer systems. To solve the bottleneck problem, a variety of techniques have been studied to hide memory latency from intermediate fast memories (caches) to various prefetching and memory management techniques. Although there are a lot of algorithms in the literature to solve the scheduling with memory management problem for multiprocessor systems, they may not deliver good quality with high performance for heterogeneous multicore processors. In this paper, we first propose a scheduling algorithm Recom_Task_Assign to reduce the write activities to main memory. Then, in conjunction with the Recom_Task_Assign algorithm, we present a new partition scheduling algorithm called heterogeneous multiprocessor partition (HMP) based on the prefetching technique for heterogeneous multicore processors, which can hide memory latencies for applications with multi-dimensional loops. This technique takes advantage of memory access pattern information and fully considers the heterogeneity of processors to achieve high processor utilization. Our HMP algorithm selects the appropriate partition size and shape according to different processors, which increases processor utilization and reduces memory latency. Experiments on DSP benchmarks show that our algorithm can efficiently reduce memory latency and enhance parallelism compared with existing methods.

Workload distribution is critical to the performance of network processor based parallel forwarding systems. Scheduling schemes that operate at the packet level, e.g., round-robin, cannot preserve packet-ordering within individual TCP connections. Moreover, these schemes create duplicate information in processor caches and therefore are inefficient in resource utilization. Hashing operates at the flow level and is naturally able to maintain per-connection packet ordering; besides, it does not pollute caches. A pure hash-based system, however, cannot balance processor load in the face of highly skewed flow-size distributions in the Internet; usually, adaptive methods are needed. In this paper, based on measurements of Internet traffic, we examine the sources of load imbalance in hash-based scheduling schemes. We prove that under certain Zipf-like flow-size distributions, hashing alone is not able to balance workload. We introduce a new metric to quantify the effects of adaptive load balancing on overall forwarding performance. To achieve both load balancing and efficient system resource utilization, we propose a scheduling scheme that classifies Internet flows into two categories: the aggressive and the normal, and applies different scheduling policies to the two classes of flows. Compared with most state-of-the-art parallel forwarding schemes, our work exploits flow-level Internet traffic characteristics.

A methodology is devised to simulate and evaluate the performance of RISC processors. The detailed model provides a way to evaluate the performance of the processor for various applications and under a wide spectrum of operation environments and conditions. The instruction flow of each instruction is modeled. The instruction requests the various units and stages of pipelines which are modeled as resources. The model is applied on the Motorola MC88100 RISC processor which includes four processing units and numerous pipelines to speed up the execution of instructions. From the model the processing speed is investigated for various applications, and the contention on some units is studied. Moreover, the utilization of units and mean waiting time are studied and discussed. >

In this work we modify the conventional row buffer allocation mechanism used in DDR2 SDRAM banks to improve average memory latency and overall processor performance. Our method assigns row buffers to different banks dynamically and by taking into account program cyclic behavior and bank row buffer demand.As we show in this work, memory requests go through several phases. In each phase, programs tend to access a single bank most of the time. We exploit this repetitive behavior and improve the concurrency level for memory read and write operations. We do so by assigning idle row buffers to more demanding banks during specific program phases. This improves average memory latency and processor performance by 12.7% and 7.6% respectively.

An integrated prefetching/caching scheme in multimedia servers

We have developed a superscalar RISC processor for the super technical server HITACHI SR8000. The processor includes architectural features specialized in scientific applications, in which massive amounts of data in the main memory must be processed. These features are a slide-windowed-registers scheme and a simultaneous execution of up to 16 prefetch instructions. The slide-windowed-registers scheme enables instructions of the processor to access any of 160 floating point registers (FPRs). The execution mechanism for prefetch instructions produces high efficiency of the out-of-order superscalar processor despite the long latency of the main memory. A logic simulation showed that the performance of the processor reaches over 3 floating-point operations per cycle and the memory throughput of over 12 bytes per cycle.

In processor design, RISC-V has key advantages such as modularity, scalability, and high performance. The five- stage pipelining technique plays a very important role in improving processor utilization and overcoming problems in performance. The purpose of this study introduces the design, implementation, and performance of a five-stage pipelined RISC-V processor. The processor is divided into five stages—Instruction Fetch, Instruction Decode, Execute, Memory Access, and Write Back to achieve faster instruction execution. Pipeline registers are placed between stages to transfer necessary controls and data signals, enabling the overlapping instruction execution. This architecture improves instruction throughput, reduces program execution time, and improves overall processor efficiency.