Dataflow Processing in Memory Achieves Significant Energy Efficiency

Processing-In-Memory (PIM) is an increasingly popular architecture aimed at addressing the 'memory wall' crisis by prioritizing the integration of processors within DRAM. It promotes low data access latency, high bandwidth, massive parallelism, and low power consumption. The skyline operator is a known primitive used to identify those multi-dimensional points offering optimal trade-offs within a given dataset. For large multidimensional dataset, calculating the skyline is extensively compute and data intensive. Although, PIM systems present opportunities to mitigate this cost, their execution model relies on all processors operating in isolation with minimal data exchange. This prohibits direct application of known skyline optimizations which are inherently sequential, creating dependencies and large intermediate results that limit the maximum parallelism, throughput, and require an expensive merging phase. In this work, we address these challenges by introducing the first skyline algorithm for PIM architectures, called DSky. It is designed to be massively parallel and throughput efficient by leveraging a novel work assignment strategy that emphasizes load balancing. Our experiments demonstrate that it outperforms the state-of-the-art algorithms for CPUs and GPUs, in most cases. DSky achieves 2× to 14× higher throughput compared to the state-of-the-art solutions on competing CPU and GPU architectures. Furthermore, we showcase DSky's good scaling properties which are intertwined with PIM's ability to allocate resources with minimal added cost. In addition, we showcase an order of magnitude better energy consumption compared to CPUs and GPUs.

With increasing performance mismatch between processor and memory, memory wall has become the bottleneck of the entire computing system. In order to bridge the gap, processing-in-memory (PIM) has been revisited as a viable option to overcome the challenge, with various researches from devices to system. In this paper we present a complete design framework for PIM based acceleration with energy efficiency and performance improvement. The framework covers system level design and prototype architecture and software stack support to enable hardware accelerator design and optimization. It is also featured with configurability, easy access and effective evaluating and profiling. In the experiments, we analyzed a convolutional neural network to identify the least energy-efficient operation and replaced that by PIM acceleration. The experimental results show that the proposed accelerator is able to achieve up 6-9X performance gain for matrix multiplication as well as 10-15X energy improvement compared to conventional CPU-only implementation.

While parallel architectures based on clusters of Processing Elements (PEs) sharing L1 memory are widespread, there is no consensus on how lean their PE should be. Architecting PEs as vector processors holds the promise to greatly reduce their instruction fetch bandwidth, mitigating the Von Neumann Bottleneck (VNB). However, due to their historical association with supercomputers, classical vector machines include micro-architectural tricks to improve the Instruction Level Parallelism (ILP), which increases their instruction fetch and decode energy overhead. In this paper, we explore for the first time vector processing as an option to build small and efficient PEs for large-scale shared-L1 clusters. We propose Spatz, a compact, modular 32-bit vector processing unit based on the integer embedded subset of the RISC-V Vector Extension version 1.0. A Spatz-based cluster with four Multiply-Accumulate Units (MACUs) needs only 7.9 pJ per 32-bit integer multiply-accumulate operation, 40% less energy than an equivalent cluster built with four Snitch scalar cores. We analyzed Spatz' performance by integrating it within MemPool, a large-scale many-core shared-L1 cluster. The Spatz-based MemPool system achieves up to 285 GOPS when running a 256x256 32-bit integer matrix multiplication, 70% more than the equivalent Snitch-based MemPool system. In terms of energy efficiency, the Spatz-based MemPool system achieves up to 266 GOPS/W when running the same kernel, more than twice the energy efficiency of the Snitch-based MemPool system, which reaches 128 GOPS/W. Those results show the viability of lean vector processors as high-performance and energy-efficient PEs for large-scale clusters with tightly-coupled L1 memory.

In order to deliver high performance under stringent power constraints, future systems may include die-stacked memories with processing-in-memory (PIM) cores. Because of their proximity to the memory, PIMs are expected to target applications which require high bandwidth, implying that PIMs do not need the same computational capabilities as traditional host processor and can therefore be implemented using slower, low-leakage transistors to increase energy efficiency. Such systems must carefully balance design-time choices, such as the circuits used to build the devices, and run-time choices, such as DVFS states and the preferred hardware platform on which to run the application. This paper explores these parameters in a GPGPU PIM system with a large compute-optimized host and a collection of bandwidth-optimized PIMs. We develop high-level performance and power models and use them to find optimal DVFS and kernel placement decisions for a series of GPGPU applications targeting maximum energy efficiency. We find, for instance, that the energy efficiency of PIM systems is greatly affected by DVFS; simply selecting the optimum hardware (host/PIM) results in 7\(\times \) higher ED\(^2\) than migrating work in conjunction with DVFS.

Graphs are used in a wide variety of applicationdomains, from social science to machine learning. Graphalgorithms present large numbers of irregular accesses with littledata reuse to amortize the high cost of memory accesses, requiring high memory bandwidth. Processing in memory (PIM) implemented through 3D die-stacking can deliver this highmemory bandwidth. In a system with multiple memory moduleswith PIM, the in-memory compute logic has low latency and highbandwidth access to its local memory, while accesses to remotememory introduce high latency and energy consumption. Ideally, in such a system, computation and data are partitioned amongthe PIM devices to maximize data locality. But the irregularmemory access patterns present in graph applications make itdifficult to guarantee that the computation in each PIM devicewill only access its local data. A large number of remote memoryaccesses can negate the benefits of using PIM. In this paper, we examine the feasibility and potential of finegrainedwork migration to reduce remote data accesses insystems with multiple PIM devices. First, we propose a datadrivenimplementation of our study algorithms: breadth-firstsearch (BFS), single source shortest path (SSSP) and betweennesscentrality (BC) where each PIM has a queue where the verticesthat it needs to process are held. New vertices that need to beprocessed are enqueued at the PIM device co-located with thememory that stores those vertices. Second, we propose hardwaresupport that takes advantage of PIM to implement highlyefficient queues that improve the performance of the queuingframework by up to 16.7%. Third, we develop a timing model forthe queueing framework to explore the benefits of workmigration vs. remote memory accesses. And, finally, our analysisusing the above framework shows that naive task migration canlead to performance degradations and identifies trade-offsamong data locality, redundant computation, and load balanceamong PIM devices that must be taken into account to realize thepotential benefits of fine-grain task migration.

To solve the scaling, memory wall and high power density issues, recently RRAM-based accelerators, which show a better energy and area efficiency compared with the CMOS-based counterparts, have been proposed for convolutional neural networks. However, the RRAM-based architectures still face several design challenges, including the high energy and timing overhead at the analog/digital (A/D) conversion and interfacing circuits. To address these issues, we propose several novel optimization schemes in this work. First an encoding scheme for the synaptic weights and the input feature maps is proposed to reduce the energy of the in-situ computation and the bit-resolution of the A/D conversion. Then the resolution of the A/D conversion is further optimized for a lower energy consumption. Moreover, a dynamic quantization scheme for the multiply-accumulate operations (MACs) is proposed to improve the throughput and the energy efficiency by reducing the number of partial products. Experimental results show that the throughput, the energy efficiency and the area efficiency are improved by 2 to 4 times when compared with the state-of-the-art RRAM-based accelerators.

Efficient data transfer between memory and photonic components is crucial for a wide range of applications. However, this necessity brings forth energy-efficient data movement challenges associated with the memory wall, underscoring the demand for a fast and low-energy electro-optic photonic memory solution. Here, we demonstrate a class of energy-efficient electro-optic devices, namely Pockels photonic memory, that combines low-field switchable ferroelectrics with lithium niobate’s Pockel’s effect. Among such devices, this article will describe in detail the integrated embodiment of a ferroelectric field-effect transistor with lithium niobate on insulator micro ring resonator. We achieve switchable and non-volatile multiple optical memory states (6 states per transistor) with ultra-low energy cost (femto Joule/state), while achieving robust 10 year data retention and read-write endurance exceeding 107 cycles. Furthermore, we demonstrate the possibility of linear memory state stacking. The Pockels photonic memory enables the scaling of reconfigurable photonic systems into the femto Joule/state energy efficiencies.

The kd-tree is a widely used data structure for managing multidimensional data. However, most existing kd-tree designs suffer from the memory wall---bottlenecked by off-chip memory latency and bandwidth limitations. Processing-in-memory (PIM), an emerging architectural paradigm, offers a promising solution to this issue by integrating processors (PIM cores) inside memory modules and offloading computational tasks to these PIM cores. This approach enables low-latency on-chip memory access and provides bandwidth that scales with the number of PIM modules, significantly reducing off-chip memory traffic.

5G network is an enhanced communication network, designed to converge the requirements of quality of service (QoS) parameters, data and capacity, by means of signals with high QoS and high-speed data rate. Several state-of-art technologies are involved to obtain the requirement of energy efficient communication system with increased number of users, devices, higher data rate with low latency. This paper presents a system which demonstrates the energy and spectral efficiency achieved for various number of nodes in a specific area. This study addresses the improvement in energy and spectral efficiency when the proposed algorithm is used. The proposed algorithm is a combination of swarm based artificial bee colony (ABC) algorithm with neural network. Experimental results have been carried out to observe the performance of QoS parameters such as bit error rate (BER), throughput, power consumption and mean square error (MSE). The maximum energy efficiency achieved is 34% and Spectral efficiency is 36%.

In the past decade, various processing-in-memory (PIM) accelerators based on various devices, micro-architectures, and interfaces have been proposed to accelerate deep neural networks (DNNs). How to deploy DNNs onto PIM-based accelerators is the key to explore PIM’s high performance and energy efficiency. The scale of DNN models, the diversity of PIM accelerators, and the complexity of deployment are far beyond the human deployment capability. Hence, an automatic deployment methodology is indispensable. In this work, we propose PIMCOMP, an end-to-end DNN compiler tailored for PIM accelerators, achieving efficient deployment of DNN models on PIM hardware. PIMCOMP can adapt to various PIM architectures by using an abstract configurable PIM accelerator template with a set of pseudo instructions, which is a high-level abstraction of the hardware’s fundamental functionalities. Through a generic multilevel optimization framework, PIMCOMP realizes an end-to-end conversion from a high-level DNN description to pseudo instructions, which can be further converted to specific hardware intrinsics/primitives. The compilation addresses two critical issues in PIM-accelerated inference from a system perspective: 1) resource utilization and 2) dataflow scheduling. PIMCOMP adopts a flexible unfolding format to reshape and partition convolutional layers, adopts a weight-layout guided computation-storage-mapping approach to enhance resource utilization, and balances the system’s computation, memory access, and communication characteristics. For dataflow scheduling, we design two scheduling algorithms with different interlayer pipeline granularities to support varying application scenarios while ensuring high-computational parallelism. Experiments demonstrate that PIMCOMP improves throughput, latency, and energy efficiency across various architectures. PIMCOMP is open-sourced at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/sunxt99/PIMCOMP-NN</uri>.

Modern computing systems are overwhelmingly designed to move data to computation. This design choice goes directly against at least three key trends in computing that cause performance, scalability and energy bottlenecks: (1) data access is a key bottleneck as many important applications are increasingly data-intensive, and memory bandwidth and energy do not scale well, (2) energy consumption is a key limiter in almost all computing platforms, especially server and mobile systems, (3) data movement, especially off-chip to on-chip, is very expensive in terms of bandwidth, energy and latency, much more so than computation. These trends are especially severely-felt in the data-intensive server and energy-constrained mobile systems of today. At the same time, conventional memory technology is facing many technology scaling challenges in terms of reliability, energy, and performance. As a result, memory system architects are open to organizing memory in different ways and making it more intelligent, at the expense of higher cost. The emergence of 3D-stacked memory plus logic, the adoption of error correcting codes inside the latest DRAM chips, proliferation of different main memory standards and chips, specialized for different purposes (e.g., graphics, low-power, high bandwidth, low latency), and the necessity of designing new solutions to serious reliability and security issues, such as the RowHammer phenomenon, are an evidence of this trend. This chapter discusses recent research that aims to practically enable computation close to data, an approach we call processing-in-memory (PIM). PIM places computation mechanisms in or near where the data is stored (i.e., inside the memory chips, in the logic layer of 3D-stacked memory, or in the memory controllers), so that data movement between the computation units and memory is reduced or eliminated. While the general idea of PIM is not new, we discuss motivating trends in applications as well as memory circuits/technology that greatly exacerbate the need for enabling it in modern computing systems. We examine at least two promising new approaches to designing PIM systems to accelerate important data-intensive applications: (1) processing using memory by exploiting analog operational properties of DRAM chips to perform massively-parallel operations in memory, with low-cost changes, (2) processing near memory by exploiting 3D-stacked memory technology design to provide high memory bandwidth and low memory latency to in-memory logic. In both approaches, we describe and tackle relevant cross-layer research, design, and adoption challenges in devices, architecture, systems, and programming models. Our focus is on the development of in-memory processing designs that can be adopted in real computing platforms at low cost. We conclude by discussing work on solving key challenges to the practical adoption of PIM.

Deep learning architectures are increasingly limited by the memory wall — the “von Neumann bottleneck” that forces frequent data transfers between memory and processing units, throttling throughput and energy efficiency. To alleviate this bottleneck, researchers have explored Compute-in-Memory (CiM) architectures to accelerate the core computing primitives such as the matrix–vector multiplications (MVMs), within memory arrays. Yet, large language models based on transformers also require many non-linear softmax operations, creating a new bottleneck. From a technology perspective, non-volatile STT-MRAMs are promising for CiM thanks to their endurance, nonvolatility, and area footprint. However, conventional 1T–1R STT-MRAM CiM arrays place bit-cells in parallel, yielding lower MVM energy efficiency than 8T-SRAM CiM due to low device resistance; line parasitics and low OFF/ON ratios of STT-MRAM devices further limit row parallelism and scalability. In this context, we propose ROSETTA, a CiM array based on 3T–2R STT-MRAM bit-cells that uses series-resistance sensing and time-to-digital conversion (TDC) for low-power MVMs. An added word line also stores a ROM bit, enabling in-array tables for fast softmax, without increasing bit-cell area. We introduce a novel palindromic encoding of inputs and weights to mitigate the data-dependent nonlinearity in STT-MRAM CiM arrays based on TDC sensing. This enables 8× higher row-level parallelism than standard 1T–1R MRAM CiM and facilitates scaling to larger CiM arrays. Our macro occupies ∼ 47% less area and consumes 3.1× less MVM energy than an equivalent ROM-overlaid 8T-SRAM macro, at similar latency and accuracy. At the system level, a spatial architecture comprising our macros exhibits 11.7−14.5× and 1.44 − 1.5× higher energy efficiency than equivalent architectures comprising 1T-1R MRAM and 8T-SRAM ROM-overlaid CiM macros, respectively.

Processing in Memory based on memristors is considered the most effective solution to overcome the Von Neumann bottleneck issue and has become a hot research topic. The execution efficiency of logical computation and in-memory data transmission is crucial for Processing in Memory. This paper presents a design scheme for data transmission and multi-bit multipliers within MAT (a data storage set in MPU) based on the memristive alternating crossbar array structure. Firstly, to improve the data transfer efficiency, we reserve the edge row and column of the array as assistant cells for OR AND (OA) and AND data transmission logic operations to reduce the data transfer steps. Furthermore, we convert the multipliers into multi-bit addition operations via Multiple Input Multiple Output (MIMO) logical operations, which effectively improves the execution efficiency of multipliers. PSpice simulation shows that the proposed data transmission and multi-bit multiplier solution has lower latency and power consumption and higher efficiency and flexibility.

The energy constraints due to the end of Dennard scaling, the popularity of in-memory analytics, and the advances in 3D integration technology have led to renewed interest in near-data processing (NDP) architectures that move processing closer to main memory. Due to the limited power and area budgets of the logic layer, the NDP compute units should be area and energy efficient while providing sufficient compute capability to match the high bandwidth of vertical memory channels. They should also be flexible to accommodate a wide range of applications. Towards this goal, NDP units based on fine-grained (FPGA) and coarse-grained (CGRA) reconfigurable logic have been proposed as a compromise between the efficiency of custom engines and the flexibility of programmable cores. Unfortunately, FPGAs incur significant area overheads for bit-level reconfiguration, while CGRAs consume significant power in the interconnect and are inefficient for irregular data layouts and control flows. This paper presents Heterogeneous Reconfigurable Logic (HRL), a reconfigurable array for NDP systems that improves on both FPGA and CGRA arrays. HRL combines both coarse-grained and fine-grained logic blocks, separates routing networks for data and control signals, and uses specialized units to effectively support branch operations and irregular data layouts in analytics workloads. HRL has the power efficiency of FPGA and the area efficiency of CGRA. It improves performance per Watt by 2.2x over FPGA and 1.7x over CGRA. For NDP systems running MapReduce, graph processing, and deep neural networks, HRL achieves 92% of the peak performance of an NDP system based on custom accelerators for each application.

Purpose Integrating Internet of Things (IoT) networks with distributed ledger technology (DLT) and artificial intelligence (AI) presents critical challenges, particularly related to latency, scalability, hardware constraints and data security. Efficient data ingestion and validation are essential to enable real-time AI processing. The main contribution of this paper is the proposal of the Energy consensus algorithm, designed to minimize both latency and energy consumption in such environments. Design/methodology/approach Energy is a consensus algorithm tailored for public directed acyclic graph-based DLTs in IoT contexts. It introduces a flexible transaction validation mechanism that reduces or bypasses Proof of Work requirements. The algorithm’s performance is experimentally compared with IOTA under varying payload conditions. Findings Results show that Energy significantly reduces latency and energy consumption, especially for small payloads, which are common in IoT applications. These findings demonstrate Energy’s ability to enhance transaction efficiency and support real-time AI model updates based on verified IoT data streams. Research limitations/implications Future work should investigate the scalability of Energy in larger and more heterogeneous IoT ecosystems, as well as its compatibility with different AI frameworks. Evaluating its performance under diverse network conditions and hardware setups would further strengthen the generalizability of the results. Practical implications The Energy algorithm enables continuous AI model updates while ensuring data integrity, traceability and low latency. Its adaptability makes it a suitable solution for large-scale IoT deployments requiring secure and efficient data processing. Originality/value This paper presents a novel consensus algorithm that bridges the requirements of IoT, DLT and AI, with a particular focus on improving latency and energy efficiency. Energy offers a robust approach for optimizing data flow and transaction processing in real-time, AI-driven IoT systems.