CPU Optimization Techniques for Compiled Languages in Scientific Computing

The paper presents an environment for teaching elements of a computer system memory hierarchy. It is made up of a hierarchical memory system, a reference manual, a software package and a set of laboratory experiments. The hierarchical memory system is devised to cover the virtual memory and translation lookaside buffer, the cache memory, and the main memory. The reference manual provides all implementation details with the appropriate circuits drawings and detailed descriptions. For the devised hierarchical memory system a software package, which includes the graphical simulator with the accompanying tools, is developed. They allow one to carry out the simulation down to the register transfer level by executing a set of laboratory experiments.

Applications executing on multicore processors can now easily offload computations to many core processors, such as Intel Xeon Phi coprocessors. However, it requires high levels of expertise and effort to tune such offloaded applications to realize high-performance execution. Previous efforts have focused on optimizing the execution of offloaded computations on many core processors. However, we observe that the data transfer overhead between multicore and many core processors, and the limited device memories of many core processors often constrain the performance gains that are possible by offloading computations. In this paper, we present three source-to-source compiler optimizations that can significantly improve the performance of applications that offload computations to many core processors. The first optimization automatically transforms offloaded codes to enable data streaming, which overlaps data transfer between multicore and many core processors with computations on these processors to hide data transfer overhead. This optimization is also designed to minimize the memory usage on many core processors, while achieving the optimal performance. The second compiler optimization re-orders computations to regularize irregular memory accesses. It enables data streaming and factorization on many core processors, even when the memory access patterns in the original source codes are irregular. Finally, our new shared memory mechanism provides efficient support for transferring large pointer-based data structures between hosts and many core processors. Our evaluation shows that the proposed compiler optimizations benefit 9 out of 12 benchmarks. Compared with simply offloading the original parallel implementations of these benchmarks, we can achieve 1.16x-52.21x speedups.

Analyzing and optimizing program memory performance is a pressing problem in high-performance computer architectures. Currently, software solutions addressing the processor-memory performance gap include compiler-or programmer-applied optimizations like data structure padding, matrix blocking, and other program transformations. Compiler optimization can be effective, but the lack of precise analysis and optimization frameworks makes it impossible to confidently make optimal, rather than heuristic-based, program transformations. Imprecision is most problematic in situations where hard-to-predict cache conflicts foil heuristic approaches. Furthermore, the lack of a general framework for compiler memory performance analysis makes it impossible to understand the combined effects of several program transformations.The Cache Miss Equation (CME) framework discussed in this paper addresses these issues. We express memory reference and cache conflict behavior in terms of sets of equations. The mathematical precision of CMEs allows us to find true optimal solutions for transformations like blocking or padding. The generality of CMEs also allows us to reason about interactions between transformations applied in concert. Unlike our prior work, this framework applies to caches of arbitrary associativity. This paper also demonstrates the utility of CMEs by presenting precise algorithms for intra-variable padding, inter-variable padding, and selecting tile sizes. Our experiences with CMEs implemented in the SUIF system show that they are a unifying mathematical framework offering the generality and precision imperative for compiler optimizations on current high-performance architectures.

Temporal data mining algorithms are becoming increasingly important in many application domains including computational neuroscience, especially the analysis of spike train data. While application scientists have been able to readily gather multi-neuronal datasets, analysis capabilities have lagged behind, due to both lack of powerful algorithms and inaccessibility to powerful hardware platforms. The advent of GPU architectures such as Nvidia's GTX 280 offers a cost-effective option to bring these capabilities to the neuroscientist's desktop. Rather than port existing algorithms onto this architecture, we advocate the need for algorithm transformation, i.e., rethinking the design of the algorithm in a way that need not necessarily mirror its serial implementation strictly. We present a novel implementation of a frequent episode discovery algorithm by revisiting 'in-the-large' issues such as problem decomposition as well as 'in-the-small' issues such as data layouts and memory access patterns. This is non-trivial because frequent episode discovery does not lend itself to GPU-friendly data-parallel mapping strategies. Applications to many datasets and comparisons to CPU as well as prior GPU implementations showcase the advantages of our approach.

Data locality exists everywhere in the memory hierarchy. Most applications show temporal and spatial locality, and computer system and architecture designers utilize this property to improve the system performance with better data layout, prefetching, and scheduling. The locality property can be represented by memory access patterns, which records the time and frequency of accessed addresses. From the security perspective, if an attacker can trace the access pattern, sensitive information inside of the application could be observed and leaked. Oblivious RAM is one of the most effective solutions to mitigate the access pattern leakage on the system, which adds redundant data blocks in space and time. With ORAM protection, the intrinsic data locality is broken by the randomly stored data. Therefore, the application cannot gain any performance benefits from locality if the ORAM protocol is used. In this work, we would like to study the potential to support multi-range accesses with new storage and access efficient ORAM construction. Our proposed designs include two major schemes: Lite-rORAM, which minimize the storage overhead of existing rORAM; and Hybrid-rORAM, which support multiple ranges accesses with minimum storage overhead. We achieve the goal to preserve the locality for consecutive data blocks with different ranges in the application while obfuscates the access pattern as well. We tested our proposed schemes with different workloads on local and remote backends. The experimental results show that, in the best case, our proposed ORAM construction can reduce the data block retrieval time to 0.24x of the baseline Path ORAM, with 87.5% storage overhead reduction compared to rORAM.

Leveraging large data sets, deep Convolutional Neural Networks (CNNs) achieve state-of-the-art recognition accuracy. Due to the substantial compute and memory operations, however, they require significant execution time. The massive parallel computing capability of GPUs make them as one of the ideal platforms to accelerate CNNs and a number of GPU-based CNN libraries have been developed. While existing works mainly focus on the computational efficiency of CNNs, the memory efficiency of CNNs have been largely overlooked. Yet CNNs have intricate data structures and their memory behavior can have significant impact on the performance. In this work, we study the memory efficiency of various CNN layers and reveal the performance implication from both data layouts and memory access patterns. Experiments show the universal effect of our proposed optimizations on both single layers and various networks, with up to 27.9× for a single layer and up to 5.6× on the whole networks.

The Fast Fourier Transform (FFT) is one of the most widely used computation kernels. Its performance is highly sensitive to the efficiency of its memory operations. In particular, when the FFT is applied on large problem sizes, the overall performance becomes almost totally memory bound. In those cases, the impact of the algorithm’s computational operations is demoted to minimal. This is probably why existing efforts on optimizing large size FFTs have been focused on optimizing the memory operation efficiency. The widely adopted approach in that regard is to adapt parameters external to the FFT algorithm, such as data layout, system software memory management, etc., to the needs of FFT’s memory operation pattern.In this paper, we look into the FFT internals for a way to mitigate FFT’s memory "boundness". The low efficiency of FFT’s memory operation, to a large degree, is incurred by the many strided data shuffles intrinsic to the algorithm, and made worse by the fact that many of those stride sizes are power-of-two. The strided data shuffling is already one of the hard workload patterns for memory system, and the power-of-two stride sizes make such a memory access pattern even more challenging. This power-of-two strided memory access pattern is the key problem that our paper addresses and represents our main contribution. Our key idea is a complete separation of the logic indexes and the physical indexes of the data internally accessed in FFT. By maintaining an efficient mapping between the logic indexes and the physical indexes through all passes, almost any stride sizes can be used and be further tuned. As the result, FFTs can be correctly carried out without the need for most of those worst power-of-two data shuffles. This greatly improves memory performance and in turn raises the overall performance. We compare our Shuffle-Light FFT (SLFFT) library with FFTW, a very widely used FFT library, and achieve the average speedup on 24.7% on problems sizes from 2 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">6</sup> to 2 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">26</sup> , and the maximal speedup of 39.4%.

Abstract— Aim/Background This study aims to systematically evaluate the practical runtime performance of the Insertion Sort algorithm across six widely used programming languages: C, C++, Go, Java, PHP, and Python. While Insertion Sort’s theoretical time complexity is well established, real-world execution performance may differ significantly due to variations in language paradigms, compilation strategies, and runtime environments. The research focuses on identifying how language-specific characteristics influence actual execution time. Methods The Insertion Sort algorithm was implemented uniformly in all six languages using a shared pseudocode to ensure consistency. Experiments were conducted under controlled conditions using input sizes of 1K, 10K, and 100K elements. Each dataset was tested with four different data arrangements: ascending, descending, nearly sorted, and random. Runtime measurements were collected and compared. Additionally, a Relative Execution Time Ratio (RETR) metric was introduced to provide a normalized and quantitative comparison of performance across languages. Results The experimental findings show clear performance differences among the programming languages. Go demonstrated the fastest runtimes across most scenarios, benefiting from its efficient compiled execution model. Java performed second-best, leveraging just-in-time (JIT) compilation and runtime optimizations. C and C++ showed moderate performance, influenced by compiler optimization levels and memory management mechanisms. PHP and Python exhibited significantly slower execution times, particularly for large and poorly ordered datasets, mainly due to interpreter overhead and runtime inefficiencies. The RETR metric effectively highlighted these relative performance gaps. Conclusion The study confirms that programming language choice significantly impacts real-world algorithm performance, even for well-understood algorithms like Insertion Sort. Compiled languages with efficient runtime optimizations generally outperform interpreted languages, especially for larger datasets. These results emphasize the importance of selecting appropriate programming languages for performance-critical applications and aligning algorithm implementation with both data characteristics and execution environments.

Many high performance applications run well below the peak arithmetic performance of the underlying machine, with inefficiencies often attributed to poor memory system behavior. In the context of scientific computing we examine three emerging processors designed to address the well-known gap between processor and memory performance through the exploitation of data parallelism. The VIRAM architecture uses novel PIM technology to combine embedded DRAM with a vector co-processor for exploiting its large bandwidth potential. The DIVA architecture incorporates a collection of PIM chips as smart-memory coprocessors to a conventional microprocessor, and relies on superword-level parallelism to make effective use of the available memory bandwidth. The Imagine architecture provides a stream-aware memory hierarchy to support the tremendous processing potential of SIMD controlled VLIW clusters. First we develop a scalable synthetic probe that allows us to parametize key performance attributes of VIRAM, DIVA and Imagine while capturing the performance crossover points of these architectures. Next we present results for scientific kernels with different sets of computational characteristics and memory access patterns. Our experiments allow us to evaluate the strategies employed to exploit data parallelism, isolate the set of application characteristics best suited to each architecture and show a promising direction towards interfacing leading-edge processor technology with high-end scientific computations.

Strassen's algorithm for matrix multiplication gains its lower arithmetic complexity at the expense of reduced locality of reference, which makes it challenging to implement the algorithm efficiently on a modern machine with a hierarchical memory system. We report on an implementation of this algorithm that uses several unconventional techniques to make the algorithm memory-friendly. First, the algorithm internally uses a non- standard array layout known as Morton order that is based on a quad-tree decomposition of the matrix. Second, we dynamically select the recursion truncation point to minimize padding without affecting the performance of the algorithm, which we can do by virtue of the cache behavior of the Morton ordering. Each technique is critical for performance, and their combination as done in our code multiplies their effectiveness. Performance comparisons of our implementation with that of competing implementations show that our implementation often outperforms the alternative techniques (up to 25%). However, we also observe wide variability across platforms and across matrix sizes, indicating that at this time, no single implementation is a clear choice for all platforms or matrix sizes. We also note that the time required to convert matrices to/from Morton order is a noticeable amount of execution time (5% to 15%). Eliminating this overhead further reduces our execution time.

GPUs provide high-bandwidth/low-latency on-chip shared memory and L1 cache to efficiently service a large number of concurrent memory requests. Specifically, concurrent memory requests accessing contiguous memory space are coalesced into warp-wide accesses. To support such large accesses to L1 cache with low latency, the size of L1 cache line is no smaller than that of warp-wide accesses. However, such L1 cache architecture cannot always be efficiently utilized when applications generate many memory requests with irregular access patterns especially due to branch and memory divergences that make requests uncoalesced and small. Furthermore, unlike L1 cache, the shared memory of GPUs is not often used in many applications, which essentially depends on programmers. In this article, we propose Elastic-Cache, which can efficiently support both fine- and coarse-grained L1 cache line management for applications with both regular and irregular memory access patterns to improve the L1 cache efficiency. Specifically, it can store 32- or 64-byte words in non-contiguous memory space to a single 128-byte cache line. Furthermore, it neither requires an extra memory structure nor reduces the capacity of L1 cache for tag storage, since it stores auxiliary tags for fine-grained L1 cache line managements in the shared memory space that is not fully used in many applications. To improve the bandwidth utilization of L1 cache with Elastic-Cache for fine-grained accesses, we further propose Elastic-Plus to issue 32-byte memory requests in parallel, which can reduce the processing latency of memory instructions and improve the throughput of GPUs. Our experiment result shows that Elastic-Cache improves the geometric-mean performance of applications with irregular memory access patterns by 104% without degrading the performance of applications with regular memory access patterns. Elastic-Plus outperforms Elastic-Cache and improves the performance of applications with irregular memory access patterns by 131%.

This research aims to optimize the instruction scheduling of LLVM back end through improved genetic algorithm to improve the execution efficiency of the object code generated by the compiler. Considering the characteristics of target hardware and optimization objectives, the intermediate representation (LLVM IR) information of the compiler is fully used to generate efficient and optimized target machine code, which ensures the performance of the compiled program on various hardware. Genetic algorithms are introduced as a search and optimization technique to automatically explore different instruction scheduling schemes and find the optimal solution. Through the introduction of adaptive evaluation mechanism, different scheduling schemes are evaluated based on performance indicators, and then evolved generation by generation to obtain better solutions. The experimental results show that the LLVM back-end instruction scheduling optimization based on genetic algorithm is obviously superior to the traditional method, which provides a strong support for improving the performance of compiler generated code, and provides an empirical research basis for the application of genetic algorithm in compiler optimization.

Irregular memory access patterns pose performance and user productivity challenges on distributed-memory systems. They can lead to fine-grained remote communication and the data access patterns are often not known until runtime. The Partitioned Global Address Space (PGAS) programming model addresses these challenges by providing users with a view of a distributed-memory system that resembles a single shared address space. However, this view often leads programmers to write code that causes fine-grained remote communication, which can result in poor performance. Prior work has shown that the performance of irregular applications written in Chapel, a high-level PGAS language, can be improved by manually applying optimizations. However, applying such optimizations by hand reduces the productivity advantages provided by Chapel and the PGAS model. We present an inspector-executor based compiler optimization for Chapel programs that automatically performs remote data replication. While there have been similar compiler optimizations implemented for other PGAS languages, high-level features in Chapel such as implicit processor affinity lead to new challenges for compiler optimization. We evaluate the performance of our optimization across two irregular applications. Our results show that the total runtime can be improved by as much as 52x on a Cray XC system with a low-latency interconnect and 364x on a standard Linux cluster with an Infiniband interconnect, demonstrating that significant performance gains can be achieved without sacrificing user productivity.

Despite a long history of research and achievements in designing sorting routines, new hardware features and application requirements pose advanced challenges that computer scientists are recently tackling through novel data-aware (learned) algorithms. This paper aims to better understand the strengths and limitations of learned sorters for numerical data, by addressing two main research and engineering questions: (Q1) Which is the best trade-off, in sorting items, between the efficacy of a “learned” model in approximating the distribution of the input data and its time-space complexity? and (Q2) Which algorithmic structure for distribution-based sorting is suited to leverage a learned model in order to achieve state-of-the-art performance? To answer Q1, we implement a variant of the best-known learned model (i.e., the two-layers RMI) and, also, design a fully new learned model that turns out to be space-time efficient and efficacious in approximating adversarial input distributions. We experimentally test them over 11 datasets of 200M and 800M 64-bit floating-point items, thus offering a comprehensive answer to Q1. We then address Q2 by plugging the best-learned models from above into a distribution-based sorting scheme that leads to three new sorters whose performance is tested over 33 datasets (real and synthetic) of sizes up to 800M items. Our experimental results will show that our sorters perform better than 6 (classic and learned) best-known sorters on 29 out of those 33 datasets, thus achieving new state-of-the-art tradeoffs.

The efficient use of the cache hierarchy of an execution platform often has a major impact on the performance of an application. It is often difficult for the application programmer or the compiler to determine a suitable memory layout for the application data, since the interactions between the memory accesses cannot be fully anticipated before the program execution. This paper introduces an approach to improve the cache efficiency by dynamically padding memory allocations by a post-compilation tool. Data structures like histograms are used to evaluate cache simulations of memory traces of the application considered and to compute optimized pad sets. These can then be used for later runs of the application with different data sets. The accumulated representation of the references' memory accesses additionally offers a visualization interface to the algorithm-specific memory access pattern of each reference captured. As implied above, the advantage of the method is that it also allows for an improvement of the cache usage of binary only applications for which no source code is available. Post optimization cache behavior analyses as well as run-time measurements show that the cache hit rates of the runtime-modified applications are considerably increased by applying the generated pad set.