Constant Distance Prefetching for Linked Data Structure Based on CMP

The memory wall problem is one of the important issues in modern computer system - it affects the system performance in spite of the powerful processor. The emergence of multi-core processors has further exacerbated the problem. On the other hand, the increasing use of the linked data structure in applications aggravates the memory access latency. This paper dispatches two thread prefetching techniques based on CMP which prefetches the demanded data into the shared cache in advance to hide the long memory access latency, and evaluates these two helper thread prefetching algorithms based on three fundamental aspects of data prefetching. The performance evaluation of thread prefetching shows with acceptable coverage, increasing timeliness and accuracy can improve performance of threaded prefetching. However, improving accuracy by issuing prefetches earlier introduces more uncertainty in the prefetching, hence reducing coverage. Performance results of four benchmark applications when applying with two thread prefetching techniques is also provided.

Data prefetching via helper threading has been extensively investigated on Simultaneous Multi- Threading (SMT) or Virtual Multi-Threading (VMT) architectures. Although reportedly large cache latency can be hidden by helper threads at runtime, most techniques rely on hardware support to reduce context switch overhead between the main thread and helper thread as well as rely on static profile feedback to construct the help thread code. This paper develops a new solution by exploiting helper threaded prefetching through dynamic optimization on the latest UltraSPARC Chip-Multiprocessing (CMP) processor. Our experiments show that by utilizing the otherwise idle processor core, a single user-level helper thread is sufficient to improve the runtime performance of the main thread without triggering multiple thread slices. Moreover, since the multiple cores are physically decoupled in the CMP, contention introduced by helper threading is minimal. This paper also discusses several key technical challenges of building a lightweight dynamic optimization/software scouting system on the UltraSPARC/Solaris platform.

The data needs of scientific or commercial applications from a diverse range of fields have been increasing exponentially over the recent years. Although the traditional systems work well for computation that requires limited data handling, the CMPs in cloud computing may below performance for the computation that requires large amounts of intensive data. Conventional helper thread techniques try to improve the high performance overheads, but they can not improve performance of the irregular data intensive applications with small computation workload. Our goal is to provide a novel solution to improve the application performance in data intensive computing environments. By introducing the prepuce look ahead Size K, the prepush block size P and the synchronization block size B three operations to helper thread, we expect to reduce the overheads introduced by the traditional helper thread and leave the computing resources to perform useful prefetch work. As a starting point, we design the KPB interleaved data prepush algorithm, and use Q6600 and IBM 5110 multi-core computers as our test platforms to study behaviors of the benchmarks fromSPEC2006 suite and Olden suite. We construct the helper threads of mcf from SPEC2006, mst and em3d from Olden by using our method, the average result of speedup is 1.23, 1.32and 1.09 separately on the Q6600 machine, and 1.28, 1.35 and1.23 separately on another machine. Compared with the AP and PV methods, our method is less negative impact than both AP and PV, our KPB-method is also better than AP and PV in the prefetching timeliness and control ability.

Linked data structures (LDS), such as lists and trees, are widely used in modern applications. Traversing LDS typically involves a significant amount of pointer chasing. Due to the serial nature of memory access in pointer chasing, the incurred long memory latency of traversing LDS has become a critical performance bottleneck. Furthermore, the poor spatial locality in LDS makes it difficult for spatial prefetchers to predict access addresses. Although temporal prefetchers can handle irregular memory access patterns, hindered by the challenges of collecting semantic information, current state-of-the-art temporal prefetchers suffer from significant metadata redundancy and frequent metadata conflicts. Consequently, there remain substantial opportunities to enhance the LDS prefetching. To solve this problem, we propose Augur, a semantics-aware temporal prefetcher to enhance LDS performance. Augur utilizes a novel pruning method to obtain semantic information and effectively extracts node address correlations from the perspective of nodes in LDS, thereby diminishing the metadata redundancy and conflicts. Additionally, Augur employs efficient metadata management strategies that guarantee a minimal storage overhead. Evaluated on LDS workloads, Augur achieves an average performance speedup of 17.8% and 11.7% over the baseline stride prefetcher and state-of-the-art spatial prefetcher Berti, respectively. Furthermore, Augur outperforms the state-of-the-art temporal prefetcher MISB, Triage, and Triangel, by 17.4%, 12.8%, and 6.3%, respectively, with a significantly lower storage overhead of only 1.26 KB.

In the development process of multi-core processors, the Network on Chip optimizes the storage structure, but the problem of “storage wall” still exists. There are many ways to alleviate this problem. The improvement of memory access performance can be obtained for appropriate address mapping between cache levels. As far as the performance index of address mapping is concerned, it can be considered from the perspectives of memory access bandwidth and latency in different levels of the storage system. Therefore, a module-level detection tool is designed to test the memory access performance. Based on the hardware emulation acceleration platform, this tool solves the online/offline data collection of memory access bandwidth and latency. And it implements the functions for extremely long simulation times, which is data capture, analysis, and processing. Under different address mapping and data types of different application scenarios, This tool tests memory access performance between cache modules. Based on the data-affinity address cross-mapping method, this thesis proposes an address mapping optimization scheme that binds the directory controller unit and the local memory. Through the module-level memory access performance test, the optimized data affinity and low-order address cross-mapping method increases the memory access bandwidth by 8% and the maximum bandwidth by 9% when processing scheduling-intensive data. In a word, this thesis establishes the memory access performance analysis method which is “benchmark test program, module-level memory access performance detection, address mapping optimization and configuration”.

One of the main challenges for computer architects is how to hide the high average memory access latency from the processor. In this context, Hybrid Memory Cubes (HMCs) can provide substantial energy and bandwidth improvements compared to traditional memory organizations. However, it is not clear how this reduced average memory access latency will impact the LLC. For applications with high cache miss ratios, the latency to search for the data inside the cache memory will impact negatively on the performance. The importance of this overhead depends on the memory access latency. In this paper, we present an evaluation of the L3 cache importance on a high performance processor using HMC also exploring chip area tradeoffs between the cache size and number of processor cores. We show that the high bandwidth provided by HMC memories can eliminate the need for L3 caches, removing hardware and making room for more processing power. Our evaluations show that performance increased 37% and the EDP improved 12% while maintaining the same original chip area in a wide range of parallel applications, when compared to DDR3 memories.

The performance of modern processors is often limited by execution stalls resulting from long memory access latencies. Compile-time optimizations, deep cache hierarchies and prefetching mechanisms already provide significant performance gains, by performing memory accesses in parallel with computation. However, they are reaching a throughput improvement limit. Hence, new solutions that effectively exploit the memory access patterns to improve processing throughput are required. To achieve this objective, a new compiler-assisted data streaming method is proposed. It leverages static analysis and code transformations with an on-chip data streaming support as a viable alternative to prefetching mechanisms for regular code structures. Static analysis is used to identify and encode memory accesses with a dedicated representation. Then, a code transformation algorithm detaches data indexation and address calculation from computation, allowing for a significant code reduction. An on-chip data stream controller, attached to the L1 data cache, is used to autonomously generate memory accesses from the pattern representation and reorganize the data transfers in streams, with the aid of stream buffers. When compared with state-of-the-art prefetchers, the proposed solution provides up to 26 percent of code reduction, an IPC improvement of 2.4x, and an average performance improvement of 40 percent.

Modern CPUs operate at GHz frequencies, but the latencies of memory accesses are still relatively large, in the order of hundreds of cycles. Deeper cache hierarchies with larger cache sizes can mask these latencies for codes with good data locality and reuse, such as structured dense matrix computations. However, cache hierarchies do not necessarily benefit sparse scientific computing codes, which tend to have limited data locality and reuse. We therefore propose a new memory architecture with a load miss predictor (LMP), which includes a data bypass cache and a predictor table, to reduce access latencies by determining whether a load should bypass the main cache hierarchy and issue an early load to main memory. Our architecture uses the L2 (and lower caches) as a victim cache for data removed from our bypass cache. We use cycle-accurate simulations, with SimpleScalar and Wattch to show that our LMP improves the performance of sparse codes, our application domain of interest, on average by 14%, with a 13.6% increase in power. When the LMP is used with dynamic voltage and frequency scaling (DVFS), performance can be improved by 8.7% with system power savings of 7.3% and energy reduction of 17.3% at 1800 MHz relative to the base system at 2000 MHz. Alternatively our LMP can be used to improve the performance of SPEC benchmarks by an average of 2.9 % at the cost of 7.1 % increase in average power.

Network on chip (NoC) is a new design paradigm for building scalable core-based systems on chip (SoC). Multithreading is a technique for hiding long latencies of memory accesses, through the overlapped execution of several threads. In this paper, we make a case for using multi-threaded processors (MTP) as resources in NoC and describe a methodology for off-line mapping and scheduling of concurrent applications to NoC with MTPs. The experimental results show a speedup of 15% on average for a 2/spl times/2 NoC when using two MTPs with 3-thread contexts instead of two general processors (GP).

SummaryMulticore NUMA systems present on‐board memory hierarchies and communication networks that influence performance when executing shared memory parallel codes. Characterizing this influence is complex, and understanding the effect of particular hardware configurations on different codes is of paramount importance. In this article, monitoring information extracted from hardware counters at runtime is used to characterize the behavior of each thread for an arbitrary number of multithreaded processes running in a multiprocessing environment. This characterization is given in terms of number of operations per second, operational intensity, and latency of memory accesses. We propose a runtime tool, executed in user space, that uses this information to guide two different thread migration strategies for improving execution efficiency by increasing locality and affinity without requiring any modification in the running codes. Different configurations of NAS Parallel OpenMP benchmarks running concurrently on multicore NUMA systems were used to validate the benefits of our proposal, in which up to four processes are running simultaneously. In more than the 95% of the executions of our tool, results outperform those of the operating system (OS) and produces up to 38% improvement in execution time over the OS for heterogeneous workloads, under different and realistic locality and affinity scenarios.

On multicore systems contention for shared resources occurs when memory-intensive threads are co-scheduled on cores that share parts of the memory hierarchy, such as last-level caches and memory controllers. Previous work investigated how contention could be addressed via scheduling. A contention-aware scheduler separates competing threads onto separate memory hierarchy domains to eliminate resource sharing and, as a consequence, mitigate contention. However, all previous work on contention-aware scheduling assumed that the underlying system is UMA (uniform memory access latencies, single memory controller). Modern multicore systems, however, are NUMA, which means that they feature non-uniform memory access latencies and multiple memory controllers. We discovered that contention management is a lot more difficult on NUMA systems, because the scheduler must not only consider the placement of threads, but also the placement of their memory. This is mostly required to eliminate contention for memory controllers contrary to the popular belief that remote access latency is the dominant concern. In this work we quantify the effects on performance imposed by resource contention and remote access latency. This analysis inspires the design of a contention-aware scheduling algorithm for NUMA systems. This algorithm significantly outperforms a NUMA-unaware algorithm proposed before as well as the default Linux scheduler. We also investigate memory migration strategies, which are the necessary part of the NUMA contention-aware scheduling algorithm. Finally, we propose and evaluate a new contention management algorithm that is priority-aware.

As the application area of convolutional neural networks (CNNs) is fast expanding, the demand for a customized hardware accelerator called a neural processing unit (NPU), is increasing to process them efficiently in terms of execution time and energy consumption. In the design of an NPU, building a fast and accurate virtual prototype enables us to develop a compiler concurrently with the hardware and to explore the micro-architectural design space. Since the memory access latency has a great effect on performance, it is necessary to model the memory access overhead accurately in the virtual prototype. In this work, we propose a novel analytical model for memory access latency, improving the performance estimation accuracy significantly compared with the previous state-of-the-art analytical model by considering the effect of memory access patterns of the NPU on the latency. The proposed high-level virtual prototype achieves an estimated execution time gap within a 6.8% difference from the RTL simulation result. To demonstrate the usefulness of a fast and accurate prototype, we propose a compiler optimization technique and a new DMA logic tailored for the NPU for further performance improvement.

Multi-core architectures are increasingly being used in real-time embedded systems. In general, such systems have more processors than the shared memory modules, potentially causing severe interference over memory accesses. This resource contention could lead to substantial variation on memory access latencies, and thus wide fluctuation in the overall system performance, which is highly undesirable especially for the time-critical applications. In this paper, we address resource contention and timing predictability for multi-core architectures with distributed memory interconnects. We focus on the locally arbitrated interconnect constructed by pipelined multiplexing stages with local arbitration, while the globally arbitrated interconnect employing global scheduling to the same architecture potentially suffers synchronisation issue and requires strict coordination. Our contributions are mainly threefold: (i) We analyse the resource contention across the memory access data path, and report the accurate calculational method to bound the worst-case behaviour. (ii) We compare the average-case behaviour of the locally arbitrated and the globally arbitrated architectures with experiments, demonstrating varying memory latencies caused by the resource sharing issue. (iii) We propose an architectural modification to smooth resource sharing. Evaluations on simulators and FPGA implementations with synthetic memory workload show that the latency variation is significantly reduced, contributing towards timing predictability of multi-core systems.

This paper presents a profiling-based analysis to determine the traversal orientation of link-based tree data structures. Given the very-high memory-hierarchy latencies in modern computers, once the compiler has identified that a pointer-based data structure represents a tree, it would be useful to determine the predominant orientation of traversal for the tree. Optimizing compilers can implement the static shape analysis proposed by Ghiya and Hendren to determine if a linked data structure is a tree [10]. However no techniques have been reported to enable an optimizing compiler to determine the predominant traversal orientation of a tree. This paper describes an analysis that collects data during an instrumented run to determine if the traversal is predominantly breadth-first or depth-first. The analysis determined, with high accuracy, the predominant orientation of traversal of trees in programs written by us as well as in the Olden benchmark suite. This profile-based analysis is storage efficient -- it uses only 7% additional memory in comparison with the non-instrumented version of the code. Determining the predominant orientation of traversal of a tree data structure will enable several client optimizations such as improved software-based prefetching, data-storage remapping and better memory allocators.

Caches play an important role in embedded systems by bridging the performance gap between high speed processors and slow memory. At the same time, caches introduce imprecision in Worst-case Execution Time (WCET) estimation due to unpredictable access latencies. Modern embedded processors often include cache locking mechanism for better timing predictability. As the cache contents are statically known, memory access latencies are predictable, leading to precise WCET estimate. Moreover, by carefully selecting the memory blocks to be locked, WCET estimate can be reduced compared to cache modeling without locking. Existing static instruction cache locking techniques strive to lock the entire cache to minimize the WCET. We observe that such aggressive locking mechanisms may have negative impact on the overall WCET as some memory blocks with predictable access behavior get excluded from the cache. We introduce a partial cache locking mechanism that has the flexibility to lock only a fraction of the cache. We judiciously select the memory blocks for locking through accurate cache modeling that determines the impact of the decision on the program WCET. Our synergistic cache modeling and locking mechanism achieves substantial reduction in WCET for a large number of embedded benchmark applications.