FAFNIR: Accelerating Sparse Gathering by Using Efficient Near-Memory Intelligent Reduction

Practical Mechanisms for Reducing Processor–Memory Data Movement in Modern Workloads

The performance and scalability of modern data-intensive systems processing large datasets are limited by unnecessary data movement. Even though near-data processing (NDP) can provably reduce data transfers and increase performance, at present, NDP is utilized primarily in read-only settings. Near-data execution of data-intensive modification operations is currently infeasible due to the lack of transactional consistency and the absence of practicable low-latency synchronization mechanisms between the host database engine and the NDP-engine on smart storage. In this paper, we introduce Update NDP as an approach to offloading modifications to computational storage with transactional guarantees in an NDP database system called neoDBMS . To ensure consistency, we introduce a low-latency shared lock table between the host and computational storage, based on novel cache-coherent interconnects . We also introduce a novel locking protocol that seamlessly integrates the shared lock table within the lock manager of the host NDP-engine. To handle failure recovery, while preserving high and robust performance, we introduce novel extended locking and logging mechanisms that allow the host and computational storage to perform useful work during log-movement. Our evaluation indicates that in-storage modifications in neoDBMS in mixed workload settings are ≥ 6.52 × faster than host-only executions and exhibit robust performance due to lower data movement and better resource utilization.

As applications become more data-intensive, issues like von Neumann's bottleneck and the memory wall became more apparent since data movement is the main source of inefficiency in computer systems. Looking to mitigate this issue, Near-Data Processing (NDP) moves computation from the processor to the memory, thus reducing the data movement required by many data-intensive workloads. In this paper, we look to database query operators, common targets of NDP research as database systems often need to deal with large amounts of data. We investigate the migration of most time-consuming database operators to Vector-In-Memory Architecture (VIMA), a novel 3D-stacked memory-based NDP architecture. We consider the selection, projection, and bloom join database query operators, commonly used by data analytics applications, comparing VIMA to a high-performance x86 baseline. Our results show speedups of up to 8× for selection, 6× for projection, and 16× for join while consuming up to 99% less energy. To the best of our knowledge, these results outperform the state-of-the-art for these operators on NDP platforms.

Most general-purpose distributed storage systems are not designed with near data processing (NDP) in mind. They do not respect semantic data boundaries when writing data, for example splitting a record across servers. This reduces NDP effectiveness by requiring data collation before computation. While semantic data awareness and NDP functions can be retroactively added to existing distributed storage, it is often complex and difficult to accomplish in practice. We propose sharing storage system layout information with data writers so they can adjust data layouts to prevent data alignment issues regardless of the underlying architectures. By doing so, we can simplify NDP implementation by reducing the need for data reassembly, and reduce the need for complex storage system or application extensions. We demonstrate a hinting mechanism on both HDFS with computational block storage and an erasure coded MinIO deployment, reducing data movement by up to 99% when querying CSV data with NDP co-located with the stored data. This was accomplished purely with client side data alignment, no modifications to the server side write paths, and no inter-node collation of data.

Optimizing power efficiency for 3D stacked GPU-in-memory architecture

Sparse activation is caused by the activation function (e.g., ReLU) in the feed-forward network (FFN) of large language models (LLMs), and recently emerges as a promising method for LLM inference acceleration in resource-constrained scenarios by effectively reducing computational workload and memory requirements with >80 % predicted dynamic sparsity. In this paper, we identify the heavy and dynamic data transfer is the primary reason for the significant synchronization and poor GPU utilization during decoding phase of LLM inference with sparse activation, and propose to apply the near-data-processing (NDP) architecture to handle the dynamic sparse activation, while addressing three critical challenges for further NDP-GPU collaboration optimization. (1) Under-utilization of DRAM bandwidth during memory access of NDP. (2) Workload imbalance across channels during computation of NDP. (3) Time-consuming parsing of the sparse predicted pattern during NDP-GPU collaboration. To tackle the above challenges, we present SpAct-NDP, the NDPGPU heterogeneous architecture for efficient LLM inference with sparse activation. (1) For the memory access during NDP, we design the the specific sparsity-aware weight mapping strategy considering the characteristics of sparse activation to improve DRAM bandwidth utilization by balancing the bank workload and eliminating redundant memory access. (2) For the computation during NDP, we propose two-level heuristic scheduling system to achieve channel-wise workload balance. (3) For the collaboration of NDP-GPU, we point out that the parsing of the predicted sparse pattern is more suitable for GPUs with high parallelism and propose the request-weight pair parsing mechanism according to the input requests and sparse pattern on GPU, reducing $\sim 3 \times$ execution time and $\sim 9 \times$ memory. Experiments show that SpActNDP achieves up to $2.17 \times$ and $1.92 \times$ end-to-end speedup and $1.53 \times$ and $1.45 \times$ energy efficiency compared with the SOTA software frameworks for LLM with sparse activation on NVIDIA RTX 3090 and NVIDIA Tesla A100.

High-performance applications running on current and future architectures are mostly performance-limited by the cost of data movement, vertically through the memory hierarchy of a node or between CPU host and accelerator, and horizontally across nodes. In this talk, we discuss the role of packed data layout in optimizing data movement across a variety of different architectures (CPU and GPU) and memory systems. Conceptually, a high performance application is defined as operations on a unit of data and parallelism that is designed for efficient vertical and horizontal data movement, and can be tailored to match the requirements of an application and target architecture. Integrating data layout abstractions and optimizations into programming systems provides critical technology for achieving performance portability across the diversity of high-performance architectures arising in the exascale era.

With the ability to help wildlife conservation, precise medical care, and disease understanding, genomics analysis is becoming more and moe important. Recently, with the development and wide adoption of the Next-Generation Sequencing (NGS) technology, bio-data grows exponentially, putting forward great challenges for k-mer counting - a widely used application in genomics analysis. Many hardware approaches have been explored to accelerate k-mer counting. Most of those approaches are compute-centric, i.e., based on CPU/GPU/FPGA. However, the space for performance improvement is limited for compute-centric accelerators, because k-mer counting is a memory-bound application. By integrating memory and computation close together and embracing higher memory bandwidth, Near-Data-Processing (NDP) is a good candidate to accelerate k-mer counting. Unfortunately, due to challenges of communication, bandwidth utilization, workload balance, and redundant memory accesses, previous NDP accelerators for k-mer counting cannot fully unleash the power of NDP. To build a practical, scalable, high-performance, and energy-efficient NDP accelerator for k-mer counting, we perform hardware/software co-design and propose the DIMM based Near-Data-Processing Accelerator for k-mer counting (NEST). To fully unleash the potential of NEST architecture, we modify the k-mer counting algorithm and propose a dedicated workflow to support efficient parallelism. Moreover, the proposed algorithm and workflow are able to reduce unnecessary inter-DIMM communication. To improve memory bandwidth utilization, we propose a novel address mapping scheme. The challenge of workload balance is addressed with the proposed task scheduling technique. In addition, scattered memory access and task switching are proposed to eliminate redundant memory access. Experimental results show that NEST provides 677.33x/27.24x/6.02x performance improvement and 1076.14x/62.26x/4.30x energy reduction, compared with a 48-thread CPU, a CPU/GPU hybrid approach, and a state-of-the-art NDP accelerator, respectively.

Data access costs dominate the execution times of most parallel applications and they are expected to be even more important in the future. To address this, recent research has focused on Near Data Processing (NDP) as a new paradigm that tries to bring computation to data, instead of bringing data to computation (which is the norm in conventional computing). This paper explores the potential of compiler support in exploiting NDP in the context of emerging many core systems. To that end, we propose a novel compiler algorithm that partitions the computations in a given loop nest into sub computations and schedules the resulting sub computations on different cores with the goal of reducing the distance-to-data on the on-chip network. An important characteristic of our approach is that it exploits NDP while taking advantage of data locality. Our experiments with 12 multi-threaded applications running on a state of-the-art commercial many core system indicate that the proposed compiler-based approach significantly reduces data movements on the on-chip network by taking advantage of NDP, and these benefits lead to an average execution time improvement of 18.4%. CCS CONCEPTS •Computer systems organization → Multi-core architectures;

Fargraph+: Excavating the parallelism of graph processing workload on RDMA-based far memory system

Convolutional neural networks (CNNs) are widely adopted in various applications. State-of-the-art CNN models deliver excellent classification performance, but they require a large amount of computation and data exchange because they typically employ many processing layers. Among these processing layers, convolution layers, which carry out many multiplications and additions, account for a major portion of computation and memory access. Therefore, reducing the amount of computation and memory access is the key for high-performance CNNs. In this study, we propose a cost-effective neural network accelerator, named CENNA, whose hardware cost is reduced by employing a cost-centric matrix multiplication that employs both Strassen’s multiplication and a naïve multiplication. Furthermore, the convolution method using the proposed matrix multiplication can minimize data movement by reusing both the feature map and the convolution kernel without any additional control logic. In terms of throughput, power consumption, and silicon area, the efficiency of CENNA is up to 88 times higher than that of conventional designs for the CNN inference.

Domain-specific accelerators for graph analytics leverage a large on-chip memory in order to tackle the intensive random memory accesses, offering higher performance and energy efficiency than conventional architectures. However, limited by the inefficient usage of on-chip memory, current accelerators suffer from energy and performance bottlenecks due to the large amount of off-chip memory accesses. In this work, we introduce an online preprocessing step for the vertex-centric programming model based on our observation of imbalanced memory bandwidth utilization between two execution phases. Our scheme improves energy efficiency and performance by significantly reducing off-chip accesses in two ways. First, we sequence random off-chip memory accesses to balance memory bandwidth demands and improve the utilization of on-chip memory. Second, we prune active leaf vertices to avoid redundant memory accesses. We evaluate our method on a state-of-the-art graph analytics accelerator and achieve 1.6× speedup while reducing energy consumption by 42% on average. © 2019 IEEE.

Despite the ability of modern processors to execute a variety of algorithms efficiently through instructions based on registers with ever-increasing widths, some applications present poor performance due to the limited interconnection bandwidth between main memory and processing units. Near-data processing has started to gain acceptance as an accelerator device due to the technology constraints and high costs associated with data transfer. However, previous approaches to near-data computing do not provide general-purpose processing, or require large amounts of logic and do not fully use the potential of the DRAM devices. These issues limited its wide adoption. In this paper, we present the Memory Vector Extensions (MVX), which implement vector instructions directly inside the DRAM devices, therefore avoiding data movement between memory and processing units, while requiring a lower amount of logic than previous approaches. MVX is able to obtain up to 211× increase in performance for application kernels with a high spatial locality and a low temporal locality. Comparing to an embedded processor with 8 cores and 2 memory channels that supports AVX-512 instructions, MVX performs 24× faster on average for three well known algorithms.

Recent years have seen an exploration of data volumes from a myriad of IoT devices, such as various sensors and ubiquitous cameras. The deluge of IoT data creates enormous opportunities for us to explore the physical world, especially with the help of deep learning techniques. Traditionally, the Cloud is the option for deploying deep learning based applications. However, the challenges of Cloud-centric IoT systems are increasing due to significant data movement overhead, escalating energy needs, and privacy issues. Rather than constantly moving a tremendous amount of raw data to the Cloud, it would be beneficial to leverage the emerging powerful IoT devices to perform the inference task. Nevertheless, the statically trained model could not efficiently handle the dynamic data in the real in-situ environments, which leads to low accuracy. Moreover, the big raw IoT data challenges the traditional supervised training method in the Cloud. To tackle the above challenges, we propose In-situ AI, the first Autonomous and Incremental computing framework and architecture for deep learning based IoT applications. We equip deep learning based IoT system with autonomous IoT data diagnosis (minimize data movement), and incremental and unsupervised training method (tackle the big raw IoT data generated in ever-changing in-situ environments). To provide efficient architectural support for this new computing paradigm, we first characterize the two In-situ AI tasks (i.e. inference and diagnosis tasks) on two popular IoT devices (i.e. mobile GPU and FPGA) and explore the design space and tradeoffs. Based on the characterization results, we propose two working modes for the In-situ AI tasks, including Single-running and Co-running modes. Moreover, we craft analytical models for these two modes to guide the best configuration selection. We also develop a novel two-level weight shared In-situ AI architecture to efficiently deploy In-situ tasks to IoT node. Compared with traditional IoT systems, our In-situ AI can reduce data movement by 28-71%, which further yields 1.4X-3.3X speedup on model update and contributes to 30-70% energy saving.

Chip designers have shown increasing interest in integrating specialized fixed-function coprocessors into multicore designs to improve energy efficiency. Recent work in academia [11, 37] and industry [16] has sought to enable more fine-grain offloading at the granularity of functions and loops. The sequential program now needs to migrate across the chip utilizing the appropriate accelerator for each program region. As the execution migrates, it has become increasingly challenging to retain the temporal and spatial locality of the original program as well as manage the data sharing. We show that with the increasing energy cost of wires and caches relative to compute operations, it is imperative to optimize data movement to retain the energy benefits of accelerators. We develop FUSION, a lightweight coherent cache hierarchy for accelerators and study the tradeoffs compared to a scratchpad based architecture. We find that coherency, both between the accelerators and with the CPU, can help minimize data movement and save energy. FUSION leverages temporal coherence [32] to optimize data movement within the accelerator tile. The accelerator tile includes small per-accelerator L0 caches to minimize hit energy and a per-tile shared cache to improve localized-sharing between accelerators and minimize data exchanges with the host LLC. We find that overall FUSION improves performance by 4.3× compared to an oracle DMA that pushes data into the scratchpad. In workloads with inter-accelerator sharing we save up to 10x the dynamic energy of the cache hierarchy by minimizing the host-accelerator data ping-ponging.