SPAID: software prefetching in pointer- and call-intensive environments

Current microprocessors aggressively exploit instruction-level parallelism (ILP) through techniques such as multiple issue, dynamic scheduling, and non-blocking reads. Recent work has shown that memory latency remains a significant performance bottleneck for shared-memory multiprocessor systems built of such processors.This paper provides the first study of the effectiveness of software-controlled non-binding prefetching in shared memory multiprocessors built of state-of-the-art ILP-based processors. We find that software prefetching results in significant reductions in execution time (12% to 31%) for three out of five applications on an ILP system. However, compared to previous-generation system, software prefetching is significantly less effective in reducing the memory stall component of execution time on an ILP system. Consequently, even after adding software prefetching, memory stall time accounts for over 30% of the total execution time in four out of five applications on our ILP system.This paper also investigates the interaction of software prefetching with memory consistency models on ILP-based multiprocessors. In particular, we seek to determine whether software prefetching can equalize the performance of sequential consistency (SC) and release consistency (RC). We find that even with software prefetching, for three out of five applications, RC provides a significant reduction in execution time (15% to 40%) compared to SC.

Applications that exhibit regular memory access patterns usually benefit transparently from hardware prefetchers that bring data into the fast on-chip cache just before it is required, thereby avoiding expensive cache misses. Unfortunately, unstructured mesh applications contain irregular access patterns that are often more difficult to identify in hardware. An alternative for such workloads is software prefetching, where special non-blocking instructions load data into the cache hierarchy. However, there are currently few examples in the literature on how to incorporate such software prefetches into existing applications with positive results. This paper addresses these issues by demonstrating the utility and implementation of software prefetching in an unstructured finite volume CFD code of representative size and complexity to an industrial application and across a number of processors. We present the benefits of auto-tuning for finding the optimal prefetch distance values across different computational kernels and architectures and demonstrate the importance of choosing the right prefetch destination across the available cache levels for best performance. We discuss the impact of the data layout on the number of prefetch instructions required in kernels with indirect-access patterns and show how to integrate them on top of existing optimisations such as vectorisation. Through this we show significant full application speed-ups on a range of processors, such as the Intel Xeon Skylake CPU (15%) as well as on the in-order Intel Xeon Phi Knights Corner (1.99x) architecture and the out-of-order Knights Landing (33%) many-core processor.

In automatic memory management programming language, the garbage collector is an important feature which reclaims memory that is no longer referenced by the program. The tracing collector performs a transitive closure across the object reference graph. And this process may result in too many cache misses, because the traverse exhibits poor locality. Previous studies have already proposed using software prefetching to improve memory performance of garbage collector. In this work, we studied the characteristic of prefetch instructions on x86 platforms in detail, and proposed a method to control the prefetch ratios by address thresholds. The experimental results show that a further improvement and optimization is obtained by tuning the prefetch ratio. We use these results to devise some guidelines for optimizing the memory performance as well as minimizing the collection pause times.

Growing application sizes continue to strain the memory system. As more complex applications are developed, and the instruction memory footprint increases, the cache hierarchy cannot contain relevant instructions causing the front end to become idle as it awaits fetched instructions. Hardware instruction prefetchers can alleviate this problem by learning instruction stream behavior and prefetching instructions into the cache before use. Fetch Directed Prefetching (FDP) is a ubiquitous form of hardware instruction prefetching that uses branch predictor to predict future instruction cache references. Modern processors generally implement aggressive, deep FDP to decouple the front-end from the rest of the machine. Instruction accesses, however, provide a small amount of information over long periods due to instruction stream variability and lack of information regarding the context of instruction accesses. Capturing the instruction stream’s context requires significant storage overhead to correlate instruction accesses and ensure timely accesses. Software prefetching techniques overcome this problem by profiling and statically analyzing an application’s behavior. Prior work demonstrates software instruction prefetching’s potential performance benefit but does not evaluate performance in the context of aggressive, decoupled front-ends. While software prefetching provides ~ 20% improvement in conservative front-ends, we find that it does not yield performance benefit when modeling a baseline with an aggressive FDP, in some cases hurting performance. Our analysis finds that using software instruction prefetching negatively impacts an aggressive front-end’s behavior. We investigate this finding and characterize the different states a front-end can be in and how introducing instructions into an application can change the front-end’s behavior resulting in destructive interference with the software prefetcher.

Many modern data processing and HPC workloads are heavily memory-latency bound. A tempting proposition to solve this is software prefetching, where special non-blocking loads are used to bring data into the cache hierarchy just before being required. However, these are difficult to insert to effectively improve performance, and techniques for automatic insertion are currently limited. This article develops a novel compiler pass to automatically generate software prefetches for indirect memory accesses, a special class of irregular memory accesses often seen in high-performance workloads. We evaluate this across a wide set of systems, all of which gain benefit from the technique. We then evaluate the extent to which good prefetch instructions are architecture dependent and the class of programs that are particularly amenable. Across a set of memory-bound benchmarks, our automated pass achieves average speedups of 1.3× for an Intel Haswell processor, 1.1× for both an ARM Cortex-A57 and Qualcomm Kryo, 1.2× for a Cortex-72 and an Intel Kaby Lake, and 1.35× for an Intel Xeon Phi Knight’s Landing, each of which is an out-of-order core, and performance improvements of 2.1× and 2.7× for the in-order ARM Cortex-A53 and first generation Intel Xeon Phi.

The performance monitoring of computer systems is a complex affair, made even more challenging by the increasing gap between hardware and software. Methods that collect and feed data to performance analysis can usually be classified into one of two groups. Methods in the first group (e.g., software instrumentation), can be accurate but have unacceptable overheads and offer no insight into the interaction of the software and the hardware. Methods in the second group (e.g., hardware supported) run in real time and connect code to the response in the architecture, at a considerable decrease in accuracy, with a high degree of specificity to the hardware and therefore with limited actionable information made available. In this work, we focus on the latter group - the collection and analysis of raw performance data sourced from hardware Performance Monitoring Units, built into the majority of processors. The periodic collection of samples (Event Based Sampling), containing code locations, for instance, can be triggered on events such as retired instructions or cache misses. We quantify accuracy improvements to existing methods, propose a better performing method of performance data collection and a new architecture-agnostic method of performance data analysis. In our experiments, we analyze compiled, large-scale production workloads, industry standard benchmarks, and micro-benchmarks reproducing specific architectural behaviors. First, we focus on performance data collection methods and their accuracy, when establishing instruction retirement rates. These vary from least to most advanced and we examine the low level factors influencing accuracy. In particular, we employ a method based on Last Branch Records, a hardware facility present on some Intel architectures, which allows for the sampling of short records of last taken branches leading up to the sample. We compare the most commonly used method, its improved derivatives and the Last Branch Record based method. With respect to the first method, we observe accuracy improvements of up to 18x on synthetic benchmarks and 15x on real applications. Second, to further improve accuracy, we propose a new collection method named Hybrid Basic Block Profiling, fusing those based on Last Branch Records and the widely used Event Based Sampling. We apply simple machine learning techniques to determine the operating criteria. As a demonstration of capability, our method and a profiling tool are used to generate Instruction Mixes, which traditionally require good quality data at the most detailed level of granularity - basic blocks. Compared to software instrumentation, we observe an improvement in runtime of up to 76x, while keeping instruction attribution errors at 2.1% on average. In our third contribution we describe an architecture-agnostic performance analysis methodology called Hierarchical Cycle Accounting. The user is presented with a navigable hierarchy of microarchitectural issues, expressed in a metric that is universal and simple to compare - core cycles. We develop a reference implementation for the Intel Ivy Bridge microarchitecture and provide pointers for implementations on other processor architectures. Our approach and tool are successfully used by non-experts to improve and vectorize complex scientific code. Experts benefit as well - by being able to pinpoint software issues undetected by other methods, and identifying a bug in a family of Intel processors.

Article Free Access Share on The impact of architectural trends on operating system performance Authors: M. Rosenblum Computer Systems Laboratory, Stanford University, Stanford CA Computer Systems Laboratory, Stanford University, Stanford CAView Profile , E. Bugnion Computer Systems Laboratory, Stanford University, Stanford CA Computer Systems Laboratory, Stanford University, Stanford CAView Profile , S. A. Herrod Computer Systems Laboratory, Stanford University, Stanford CA Computer Systems Laboratory, Stanford University, Stanford CAView Profile , E. Witchel Computer Systems Laboratory, Stanford University, Stanford CA Computer Systems Laboratory, Stanford University, Stanford CAView Profile , A. Gupta Computer Systems Laboratory, Stanford University, Stanford CA Computer Systems Laboratory, Stanford University, Stanford CAView Profile Authors Info & Claims SOSP '95: Proceedings of the fifteenth ACM symposium on Operating systems principlesDecember 1995Pages 285–298https://doi.org/10.1145/224056.224078Published:03 December 1995Publication History 188citation2,012DownloadsMetricsTotal Citations188Total Downloads2,012Last 12 Months295Last 6 weeks50 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteeReaderPDF

Modern multi-core processors propose new cache management challenges (more cache conflicts and misses) due to the subtle interactions of simultaneously executing processes sharing on-chip resources. To address this issue, thread group scheduling scheme that cluster threads with high sharing cache as one group to schedule has been proposed. It has led to numerous academic and industrial attentions. However, by analyzing the programming model of Android which account for a large and increasing fraction of operating system of multi-core smart phones, we find it may be unacceptable for numerous interactive applications since previous scheduling works may cause the terrible response time and directly bring an awful user experience. In order to reduce cache competitions while improving application's response time, we present an application-aware group scheduler (AGS) whose key idea is to adopt a thread group scheduling scheme to partition current running application's threads into one group and give each thread a fair chance to occupy CPU time. We implement AGS on practical hardware running a real Android operating system. The results show that our proposed scheduler can improve performance from different parameters while keeping system fairness, which reduces 3.2% in cache miss rate, 5.5% in response time, 5.3% in data cache (D-Cache) misses and 2.5% in instruction cache (I-Cache) misses to the maximum extent.

A real-time software MPEG1 video encoder on a multimedia RISC processor, V830R/AV, is presented. The V830R/AV processor provides 64-bit SIMD media-enhanced instructions. The motion estimation is accelerated by a combination of the partial absolute difference instruction and a method to sum up partial block differences. The IDCT in conformity with the IEEE standard is simply implemented by the multiply-accumulate instruction with symmetric rounding. The encoder program structure is carefully reconstructed to reduce instruction cache misses, which heavily degrade the processor performance. Preload instructions are used to load data to the data cache in parallel with successive instruction execution. Consequently, the cache miss penalties are drastically reduced. The current version of the software MPEG1 video encoder on the 200 MHz V830R/AV processor encodes SIF (352/spl times/240 pels) video at 30 frames/s with I, P and B picture types in 193 M clocks/s.

Cache misses are becoming relatively more expensive in modern processors. This is largely due do the fact that processor clock rates are increasing faster than the latency of main memory is improving. Prefetch has been used to hide memory latency. There are at least two kinds of prefetches - automatic prefetch and instruction-initiated prefetch. This paper described an implementation-independent instruction-initiated prefetch mechanism for I-cache and an automatic prefetch mechanism for D-cache to hide the memory latency associated with cache misses. Simulation results taken from execution traces of 5 commercial relational database management systems were used to illustrate the potential benefit of the proposed mechanisms.

Spin-transfer torque RAM (STT-RAM) has emerged as an energy-efficient and high-density alternative to SRAM for large on-chip caches. However, its high write energy has been considered as a serious drawback. Hybrid caches mitigate this problem by incorporating a small SRAM cache for write-intensive data along with an STT-RAM cache. In such architectures, choosing cache blocks to be placed into the SRAM cache is the key to their energy efficiency. This paper proposes a new hybrid cache architecture called prediction hybrid cache. The key idea is to predict write intensity of cache blocks at the time of cache misses and determine block placement based on the prediction. We design a write intensity predictor that realize the idea by exploiting a correlation between write intensity of blocks and memory access instructions that incur cache misses of those blocks. It includes a mechanism to dynamically adapt the predictor to application characteristics. We also design a hybrid cache architecture in which write-intensive blocks identified by the predictor are placed into the SRAM region. Evaluations show that our scheme reduces energy consumption of hybrid caches by 28 percent (31 percent) on average compared to the existing hybrid cache architecture in a single-core (quad-core) system.

Modern computer systems have increasingly complex memory systems. Common machine models for algorithm analysis do not reflect many of the features of these systems, e.g., large register sets, lockup-free caches, cache hierarchies, associativity, cache line fetching, and streaming behavior. Inadequate models lead to poor algorithmic choices and an incomplete understanding of algorithm behavior on real machines. A key step toward developing better models is to quantify the performance effects of features not reflected in the models. This paper explores the effect of memory system features on sorting performance. We introduce a new cache-conscious sorting algorithm, R-merge, which achieves better performance in practice over algorithms that are theoretically superior under the models. R-merge is designed to minimize memory stall cycles rather than cache misses, considering features common to many system designs.

Hardware transactional memory (HTM) offers a promising parallel programming model for chip multiprocessors. The performance aspect of HTMs has been explored extensively, whereas little research has addressed the power of HTMs. In this paper, we investigate power consumption in HTMs and propose two optimization techniques. The first optimization technique is adaptive snoop granularity (ASG). HTMs rely on cache coherence protocols to detect conflicts and maintain consistency of transactional data. One of the main design issues facing HTMs is the growing number of snoops required to maintain coherency of transactional data. We found that many transactions access consecutive memory locations which are not shared by other transactions. ASG monitors these transactional accesses and dynamically changes snoop granularity to reduce the power of the interconnection network and eliminate needless cache snoops. The second optimization technique is transactional snoop filtering (TSF). TSF dynamically tracks accesses to the coherence caches and eliminates cache snoops that would result in cache misses. TSF relies on small filters to monitor cache addresses. Energy is reduced as accesses to the much more demanding data caches are decreased. We extended the Gem5 simulator to model ASG and TSF. Our simulation results show that ASG and TSF are effective and reduce energy of interconnects and caches up to 44% and 89%, respectively.

SPECFEM3D_GLOBE is a spectral element application enabling the simulation of global seismic wave propagation in 3D anelastic, anisotropic, rotating and self-gravitating Earth models at unprecedented resolution. A fundamental challenge in global seismology is to model the propagation of waves with periods between 1 and 2 seconds, the highest frequency signals that can propagate clear across the Earth. These waves help reveal the 3D structure of the Earth's deep interior and can be compared to seismographic recordings. We broke the 2 second barrier using the 62K processor Ranger system at TACC. Indeed we broke the barrier using just half of Ranger, by reaching a period of 1.84 seconds with sustained 28.7 Tflops on 32K processors. We obtained similar results on the XT4 Franklin system at NERSC and the XT4 Kraken system at University of Tennessee Knoxville, while a similar run on the 28K processor Jaguar system at ORNL, which has better memory bandwidth per processor, sustained 35.7 Tflops (a higher flops rate) with a 1.94 shortest period.Thus we have enabled a powerful new tool for seismic wave simulation, one that operates in the same frequency regimes as nature; in seismology there is no need to pursue periods much smaller because higher frequency signals do not propagate across the entire globe.We employed performance modeling methods to identify performance bottlenecks and worked through issues of parallel I/O and scalability. Improved mesh design and numbering results in excellent load balancing and few cache misses. The primary achievements are not just the scalability and high teraflops number, but a historic step towards understanding the physics and chemistry of the Earth's interior at unprecedented resolution.

Multigrid is widely used as an efficient solver for sparse linear systems arising from the discretization of elliptic boundary value problems. Linear relaxation methods such as Gauss–Seidel and Red–Black Gauss–Seidel form the principal computational component of multigrid, and thus affect its efficiency. In the context of multigrid, these iterative solvers are executed for a small number of iterations (2–8). We exploit this property of the algorithm to develop a cache-efficient multigrid method, by focusing on improving the memory behavior of the linear relaxation methods. The efficiency in our cache-efficient linear relaxation algorithm comes from two sources: reducing the number of data cache and TLB misses, and reducing the number of memory references by keeping values registerresident. Our optimizations are applicable to multigrid applied to linear systems arising from constant coefficient elliptic PDEs on structured grids. Experiments on five modern computing platforms show a performance improvement of 1.15–2.7 times over a standard implementation of Full Multigrid V-Cycle.