MEMORY HIERARCHY PERFORMANCE PREDICTION FOR BLOCKED SPARSE ALGORITHMS

GPUs provide high-bandwidth/low-latency on-chip shared memory and L1 cache to efficiently service a large number of concurrent memory requests. Specifically, concurrent memory requests accessing contiguous memory space are coalesced into warp-wide accesses. To support such large accesses to L1 cache with low latency, the size of L1 cache line is no smaller than that of warp-wide accesses. However, such L1 cache architecture cannot always be efficiently utilized when applications generate many memory requests with irregular access patterns especially due to branch and memory divergences that make requests uncoalesced and small. Furthermore, unlike L1 cache, the shared memory of GPUs is not often used in many applications, which essentially depends on programmers. In this article, we propose Elastic-Cache, which can efficiently support both fine- and coarse-grained L1 cache line management for applications with both regular and irregular memory access patterns to improve the L1 cache efficiency. Specifically, it can store 32- or 64-byte words in non-contiguous memory space to a single 128-byte cache line. Furthermore, it neither requires an extra memory structure nor reduces the capacity of L1 cache for tag storage, since it stores auxiliary tags for fine-grained L1 cache line managements in the shared memory space that is not fully used in many applications. To improve the bandwidth utilization of L1 cache with Elastic-Cache for fine-grained accesses, we further propose Elastic-Plus to issue 32-byte memory requests in parallel, which can reduce the processing latency of memory instructions and improve the throughput of GPUs. Our experiment result shows that Elastic-Cache improves the geometric-mean performance of applications with irregular memory access patterns by 104% without degrading the performance of applications with regular memory access patterns. Elastic-Plus outperforms Elastic-Cache and improves the performance of applications with irregular memory access patterns by 131%.

Irregular access patterns are a major problem for today’s optimizing compilers. In this paper, a novel approach will be presented that enables transformations that were designed for regular loop structures to be applied to linked list data structures. This is achieved by linearizing access to a linked list, after which further data restructuring can be performed. Two subsequent optimization paths will be considered: annihilation and sublimation, which are driven by the occurring regular and irregular access patterns in the applications. These intermediate codes are amenable to traditional compiler optimizations targeting regular loops. In the case of sublimation, a run-time step is involved which takes the access pattern into account and thus generates a data instance specific optimized code. Both approaches are applied to a sparse matrix multiplication algorithm and an iterative solver: preconditioned conjugate gradient. The resulting transformed code is evaluated using the major compilers for the x86 platform, GCC and the Intel C compiler.

In-memory Key-Value stores (IMKVs) provide significantly higher performance than traditional disk-based counterparts. As memory technologies advance, IMKVs become practical for modern Big Data processing, which include financial services, e-commerce, telecommunication network, etc. Recently, various IMKVs have been proposed from both academia and industrial. In order to leverage high performance random access capability of main memory, most IMKVs employ hashing based index structures to retrieve data according to keys. Consequently, a regular memory access pattern can be observed in data retrieval from those IMKVs. Normally speaking, one access to index (hash table), which is also located in main memory, is followed by another memory access to value data. Such a regular access pattern provides a potential opportunity that data prefetching techniques can be employed to improve memory access efficiency for data retrieval in these IMKVs. Based on this observation, we explore various data prefetching techniques with proper architecture level modifications on memory controller considering trade-off between design overhead and performance. Specifically, we focus on two key design issues of prefetching techniques: (1) where to fetch data (i.e. data address)? and (2) how many data to fetch (i.e. data size)? Experimental results demonstrate that memory access performance can be substantially improved up to 35.4 %. In addition, we also demonstrate the overhead of prefetching on power consumption.

Deep neural network inference has become a vital workload for many systems from edge-based computing to data centers. To reduce the performance and power requirements for deep neural networks (DNNs) running on these systems, pruning is commonly used as a way to maintain most of the accuracy of the system while significantly reducing the workload requirements. Unfortunately, accelerators designed for unstructured pruning typically employ expensive methods to either determine non-zero activation-weight pairings or reorder computation. These methods require additional storage and memory accesses compared to the more regular data access patterns seen in structurally pruned models. However, even existing works that focus on the more regular access patterns seen in structured pruning continue to suffer from inefficient designs, which either ignore or expensively handle activation sparsity leading to low performance. To address these inefficiencies, we leverage structured pruning and propose the multiply-and-fire (MnF) technique, which aims to solve these problems in three ways: (a) the use of a novel event-driven dataflow that naturally exploits activation sparsity without complex, high-overhead logic; (b) an optimized dataflow takes an activation-centric approach, which aims to maximize the reuse of activation data in computation and ensures the data are only fetched once from off-chip global and on-chip local memory; and (c) based on the proposed event-driven dataflow, we develop an energy-efficient, high-performance sparsity-aware DNN accelerator. Our results show that our MnF accelerator achieves a significant improvement across a number of modern benchmarks and presents a new direction to enable highly efficient AI inference for both CNN and MLP workloads. Overall, this work achieves a geometric mean of 11.2× higher energy efficiency and 1.41× speedup compared to a state-of-the-art sparsity-aware accelerator.

The memory system is a major performance and power bottleneck in embedded systems especially for multimedia applications. Most multimedia applications access stream type of data structures with regular access patterns. It is observed that conventional caches behave poorly for stream-type data structure. Therefore, prediction-based prefetching techniques have been extensively researched to exploit the regular access patterns. Prefetching, however, may pollute the cache if the prediction is not accurate and needs extra hardware prediction logic. To overcome these problems, we propose a novel hardware prefetching technique that is assisted by static analysis of data access pattern with stream caches. With the proposed stream cache architecture, we could achieve significant performance improvement compared with the conventional cache architecture.

Applications that exhibit regular memory access patterns usually benefit transparently from hardware prefetchers that bring data into the fast on-chip cache just before it is required, thereby avoiding expensive cache misses. Unfortunately, unstructured mesh applications contain irregular access patterns that are often more difficult to identify in hardware. An alternative for such workloads is software prefetching, where special non-blocking instructions load data into the cache hierarchy. However, there are currently few examples in the literature on how to incorporate such software prefetches into existing applications with positive results. This paper addresses these issues by demonstrating the utility and implementation of software prefetching in an unstructured finite volume CFD code of representative size and complexity to an industrial application and across a number of processors. We present the benefits of auto-tuning for finding the optimal prefetch distance values across different computational kernels and architectures and demonstrate the importance of choosing the right prefetch destination across the available cache levels for best performance. We discuss the impact of the data layout on the number of prefetch instructions required in kernels with indirect-access patterns and show how to integrate them on top of existing optimisations such as vectorisation. Through this we show significant full application speed-ups on a range of processors, such as the Intel Xeon Skylake CPU (15%) as well as on the in-order Intel Xeon Phi Knights Corner (1.99x) architecture and the out-of-order Knights Landing (33%) many-core processor.

Analytical modeling is one of the most interesting approaches to evaluate the memory hierarchy behavior. Unfortunately, models have many limitations regarding the structure of the code they can be applied to, particularly when the path of execution depends on conditions calculated at run-time that depend on the input or intermediate data. In this paper we extend in this direction a modular analytical modeling technique that provides very accurate estimations of the number of misses produced by codes with regular access patterns and structures while having a low computing cost. Namely, we have extended this model in order to be able to analyze codes with data-dependent conditionals. In a previous work we studied how to analyze codes with a single and simple conditional sentence. In this work we introduce and validate a general and completely systematic strategy that enables the analysis of codes with any number of conditionals, possibly nested in any arbitrary way, while allowing the conditionals to depend on any number of items and atomic conditions.

Traditional distributed shared memory systems (DSM) suffer from the overhead of communication latencies and data fault handling. This work explores the benefits of reducing these overheads and evaluating the savings. Data fault handling can be reduced by employing a lock-oriented consistency protocol that synchronizes data when acquiring and releasing a lock, which eliminates subsequent faults on data accesses. Communication latencies can be reduced by speculatively replicating or moving data before receiving a request for such an action. Histories of DSM accesses are used to anticipate future data requests.DSM-Threads implement an adaptive runtime history that speculatively distributes data based on access histories. These histories are kept locally on a node and are distinguished by each thread on a node. When regular access patterns on requests are recognized, the data will be speculatively distributed at the closest prior release of a lock. The benefits of this active data distribution and its potential for false speculation are evaluated for two consistency protocols. The data-oriented entry consistency protocol (Bershad at al.) allows an even earlier data distribution at the closest prior release on the same lock. A simple multi-reader single-writer protocol, an improvement of Li/Hudak’s work, uses the closest prior release of a speculatively associated lock for speculative data distribution and also eliminates data faults. In addition, a framework for speculative release consistency is formulated.KeywordsDestination NodeAccess PatternSequential ConsistencyPage FaultDistribute Shared MemoryThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

We are going through the computation from single core to multicore architecture in parallel programming. Graphics Pro- cessor Units (GPUs) have recently emerged as outstanding platforms for data parallel applications with regular data access patterns. However, it is still challenging to optimize computations with irregular data access patterns like sparse matrix-vector multiplication (SPMV). SPMV is one of the most important computational kernels in engineering practice and scientific computation. Various data formats to store the sparse matrix have been implemented on GPUs to maximize the performance. In this paper, we propose and evalu- ate a new implementation of SPMV on GPU based on QCSR storage format which combines the quadtree storage format and CSR format. We also outline some optimization strategies to improve performance. In comparison with previously published implementa- tion, it achieves higher overall performance than BCSR format. The results show that it achieves 1.15 speedup averagely than BCSR format.

This paper describes a new benchmark tool, Spatter, for assessing memory system architectures in the context of a specific category of indexed accesses known as gather and scatter. These types of operations are increasingly used to express sparse and irregular data access patterns, and they have widespread utility in many modern HPC applications including scientific simulations, data mining and analysis computations, and graph processing. However, many traditional benchmarking tools like STREAM, STRIDE, and GUPS focus on characterizing only uniform stride or fully random accesses despite evidence that modern applications use varied sets of more complex access patterns.

Sparse Matrix-Vector Multiplication (SpMV) is the most important sparse linear algebra kernel in both scientific and engineering applications. Due to its irregular control flow and data access pattern, Von Neumann architectures like CPUs and GPUs cannot fully exploit the inherent parallelism of S pMV. Although FPGAs can efficiently accelerate SpMV in a dataflow manner, their performance is degraded in face of large matrices that exceed the capacity of on-chip memory because of excessive rescheduling of data. In this paper we propose ReDESK, a Reconfigurable Dataflow Engine for Sparse Kernels, for emerging tightly-coupled CPU-FPGA heterogeneous platforms. To fully exploit the heterogeneity, we design a novel representation of sparse matrix that is tailored for data prefetching on CPU-side and streaming processing on FPGA-side. In this way ReDESK can fully utilize the memory bandwidth regardless of the scale of SpMV problem. We evaluate ReDESK on Intel HARP-2 platform with a set of matrices from the University of Florida sparse matrix collection. The result demonstrates an average bandwidth utilization of 0.094 GFLOP/GB, which is 1.6-4.3x more efficient than previous SpMV on FPGAs.

Locality analysis of an application helps us extract data access patterns and predict runtime cache behavior. In this paper, we propose a locality-aware dynamic mapping algorithm for multithreaded applications, which assigns computations with similar data access patterns to same cores. We collect the amounts of shared and distinct data used by all computations, called chunks and calculate sharing among those chunks. Then, chunks with the similar data access patterns are grouped into bins, which are subsequently assigned to threads for improving cache reuse and program performance. Our algorithm is illustrated with sparse matrix-vector multiply (SpMV), which is one of the most widely used kernel in engineering and scientific computing and suffers from irregular and indirect memory access patterns. Five inputs with different shapes and characteristics are considered for testing the performance of our algorithm. Based on the results of experimental study, our algorithm outperforms Linux scheduler with an average of 12.5% performance improvement for various scenarios considered.

Intel Xeon Phi acceleration of Hybrid Total FETI solver

In the work reported here, we set out to study the neural systems that detect predictable temporal patterns and departures from them. We used functional magnetic resonance imaging (fMRI) to locate activity in the brains of subjects when they viewed temporally regular and irregular patterns produced by letters, numbers, colors and luminance. Activity induced by irregular sequences was located within dorsolateral prefrontal cortex, including an area that was responsive to irregular patterns regardless of the type of visual stimuli producing them. Conversely, temporally regular arrangements resulted in activity in the right frontal lobe (medial frontal gyrus), in the left orbito-frontal cortex and in the left pallidum. The results show that there is an abstractive system in the brain for detecting temporal irregularity, regardless of the source producing it.

Given the memory wall, the performance of the memory system significantly influences the user experience of mobile phones. The system cache (SC), located on the memory side, is shared among all CPUs and GPUs within the mobile phone, serving as the last line of defense before resorting to time-consuming off-chip memory access. Managing SC is a challenge due to its large working set and irregular access patterns. Despite occupying a substantial on-chip area, SC's effectiveness in terms of hit rate is relatively low. It has been observed that neither state-of-the-art cache replacement policies nor increasing cache size significantly improve SC performance. Prefetchers designed for higher-level caches cannot be seamlessly applied to SC due to the absence of the required program counter (PC) on the memory side and the violation of stringent power constraints in mobile phones by aggressive prefetch traffic.