ROSETTA: ROM-Overlaid STT-MRAM for Efficient MVM and Softmax Operations Towards Accelerating Transformer Inference

The rapid progress in artificial intelligence (AI) is driving the need for more efficient computing architectures capable of performing the demanding, repetitive computations required for processing large datasets. Unlike traditional von Neumann architectures, which separate processing and memory units, new approaches aim to integrate computational capabilities directly into memory arrays, facilitating in-memory computing (IMC). This integration significantly improves both energy efficiency and processing speed, particularly for AI and similar applications. A crucial operation within IMC is vector-matrix multiplication (VMM), where input signal vectors interact with a memory state matrix to produce output signal vectors.One innovative approach to implementing VMM is through the use of a ferroelectric Hf₀.₅Zr₀.₅O₂ (HZO) layer in conventional field-effect transistor (FET) or metal-insulator-metal (MIM) devices. These HZO-based ferroelectric components show significant promise for low-power, IMC-compatible systems, especially when integrated into MIM capacitor structures. Their simple fabrication process supports scalable system integration, while offering benefits such as low power consumption and high reliability. However, current IMC techniques utilizing ferroelectric MIM capacitors rely on small-signal capacitance readout, which suffers from limited contrast between high and low capacitance states, making it difficult to distinguish memory states effectively for efficient IMC operations.In this study, we introduce two novel readout techniques to improve the scalability and robustness of IMC, focusing on enhanced memory differentiation by reading the ferroelectric remanent polarization (Pᵣ) instead of relying on small-signal capacitance. These methods exploit the switching behavior between two stable polarization states (Pₗₒʷ and Pᵤₚ) of the HZO material. To measure charge accumulation resulting from polarization switching, we connect the ferroelectric HZO capacitors in series with either a shunt resistor (Rₛₕᵤₙₜ) or an external capacitor (Cₑₓt) to produce current or voltage signals, respectively. Both readout methods show significantly improved memory differentiation and VMM performance compared to the traditional small-signal capacitance approach. For example, in the current readout method, the maximum displacement current in the high-current state (Iₛʷ + Iₙₒₙ-ˢʷ) is approximately 0.31 mA, while the low-current state (Iₙₒₙ-ˢʷ) is about 0.04 mA when no switching current is present. Notably, the static leakage current during the input pulse is nearly zero, underscoring the potential for low-power operation in capacitive systems. In the voltage readout method, an initial Pᵤₚ state leads to a high output voltage (Vₒᵤₜ) of approximately 0.13 V, while an initial Pₗₒʷ state produces a low Vₒᵤₜ of about 0 V. These polarization-based readout techniques provide substantially higher memory distinction and better VMM performance compared to traditional small-signal capacitance methods.We will demonstrate IMC systems by using voltage readout of the Pr change in HZO layers, employing charge compensation between the Cext and HZO capacitors. This method takes advantage of the superior memory distinction within the capacitive system, offering more reliable and lower-power performance compared to traditional Fe-FET approaches.The authors acknowledge Prof. Joseph S. Friedman and Dr. Peng Zhou for the technical discussion.

Low-power Al edge devices should provide short-latency <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$(\mathsf{T}_{\mathsf{WK-RP}})$</tex> and low-energy <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$(\mathsf{E}_{\mathsf{WK-RP}})$</tex> wakeup responses from power-off mode to handle event-triggered computing tasks with high inference accuracy (IA), which requires high-capacity nonvolatile memory (NVM) to store high-precision weight data in power-off and high bit-precision multiply and accumulate (MAC) operations with high energy efficiency. SRAM computing-in-memory (CIM) and digital processors suffer large <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mathsf{E}_{\mathsf{WK-RP}}$</tex> and long <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mathsf{T}_{\mathsf{WK-RP}}$</tex> due to the movement of weight data from off-chip NVM to the on-chip buffer and processing unit after wakeup. Thus, on-chip nonvolatile CIM (nvCIM) is preferred for Al-edge processors by combining NVM storage and computing tasks on the same macro. Among nvCIM structures, in-memory compute (IMC) [1] provides short computing latency and high energy efficiency, but suffers from low computing yield. Near-memory compute (NMC) [2–4] provides high computing yield, but suffers long computing latency and low energy efficiency.

A review on selective in-memory computing processors: Potential alternative to AI-driven applications

Computation-in-memory using memristive devices is a promising approach to overcome the performance limitations of conventional computing architectures introduced by the von Neumann bottleneck which are also known as memory wall and power wall. It has been shown that accelerators based on memristive devices can deliver higher energy efficiencies and data throughputs when compared with conventional architectures. In the vast multitude of memristive devices, bipolar resistive switches based on the valence change mechanism (VCM) are particularly interesting due to their low power operation, non-volatility, high integration density and their CMOS compatibility. While a wide range of possible applications is considered, many of them such as artificial neural networks heavily rely on vector-matrix-multiplications (VMMs) as a mathematical operation. These VMMs are made up of large numbers of multiplication and accumulation (MAC) operations. The MAC operation can be realised using memristive devices in an analog fashion using Ohm’s law and Kirchhoff’s law. However, VCM devices exhibit a range of non-idealities, affecting the VMM performance, which in turn impacts the overall accuracy of the application. Those non-idealities can be classified into time-independent (programming variability) and time-dependent (read disturb and read noise). Additionally, peripheral circuits such as analog to digital converters can introduce errors during the digitalization. In this work, we experimentally and theoretically investigate the impact of device- and circuit-level effects on the VMM in a VCM crossbars. Our analysis shows that the variability of the low resistive state plays a key role and that reading in the RESET direction should be favored to reading in the SET direction.

Performing data-intensive tasks in the von Neumann architecture is challenging to achieve both high performance and energy efficiency due to the memory wall bottleneck. Compute-in-memory (CiM) is a promising mitigation approach by enabling parallel and in-situ multiply-accumulate (MAC) operations within the memory array. Thanks to the good matching of capacitors, SRAM-based charge-domain CiM (Q-CiM) has shown its potential for higher row-wise parallelism. However, the peripheral circuits of Q-CiM, such as the input drivers and analog-digital converters (ADCs), limit further improvement of throughput and area efficiency. This paper proposes a single-ADC multi-bit accumulation CiM macro architecture SAMBA, which can perform multi-bit MAC operation with ReLU of two vectors in one CiM cycle by only a single A/D conversion to mitigate the ADC overhead. In addition, post-correction methods are proposed to compensate the non-linearity of sensitive circuit modules in SAMBA to recover the accuracy drop due to the capacitor mismatch. A proof-of-concept macro is fabricated in a 65nm process and achieves 51.2GOPS throughput and 10.3TOPS/W energy efficiency, while showing 88.6% accuracy on CIFAR-10 and 64.8% accuracy on the CIFAR-100 with VGG-8 model.

Computing-in-memory (CIM) relieves the Von Neumann bottleneck by storing the weights of neural networks in memory arrays. However, two challenges still exist, hindering the efficient acceleration of convolutional neural networks (CNN) in artificial intelligence (AI) edge devices. Firstly, the activations for sliding window (SW) operations in CNN still bring high memory access pressure. This can be alleviated by increasing the SW parallelism, but simple array replication suffers from poor array utilization and large peripheral circuits overhead. Secondly, the partial sums from individual CIM arrays, which are usually accumulated to obtain the final sum, introduce large latency due to enormous shift-and-add operations. Moreover, high-resolution ADCs are also needed to reduce the quantization error of partial sums, further increasing the hardware costs. In this paper, a hardware-efficient CIM accelerator, ARBiS, is proposed with improved activation reusability and bit-scalable matrix-vector-multiplication (MVM) for CNN acceleration in AI edge applications. The cyclic-shift weight duplication exploits a third dimension of receptive field (RF) depth for SW weight mapping to reduce the memory accesses of activations, improving the array utilization. The parasitic-capacitance charge sharing is employed to realize high-precision analog MVM in order to reduce the ADC cost. Compared with conventional architectures, ARBiS with parallel processing of 9 SW operations achieves 56.6%~58.8% alleviation of memory access pressure. Meanwhile, ARBiS configured with 8-bit ADCs saves 92.53%~94.53% ADC energy consumption. An ARBiS accelerator is evaluated to realize a computational efficiency (CE) of 10.28 (10.43) TOPS/mm2, an energy efficiency (EE) of 91.19 (112.36) TOPS/W with 8-bit (4-bit) ADCs, achieving <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$11.4\sim 11.7\times $ </tex-math></inline-formula> ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$11.6\sim 11.8\times $ </tex-math></inline-formula> ), <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$1.1\sim 3.3\times $ </tex-math></inline-formula> ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$1.4\sim 4\times $ </tex-math></inline-formula> ) improvements over state-of-the-art works, respectively.

In-memory computing (IMC) is gaining momentum for low-power acceleration of data intensive problems, particularly in the frame of artificial intelligence (AI). Analogue matrix-vector multiplication (MVM) in crosspoint arrays allows to dramatically improve energy efficiency in neural networks for inference and training [1]. Recently, it has been shown that a feedback configuration of MVM can accelerate the solution of generic linear matrix problems, such as linear systems, matrix inversion and eigenvector calculation, with a single computational step [2,3]. This is achieved by the high parallelism of MVM in the memory array, combined with the physical iteration executed naturally in the feedback loop. Eigenvector extraction allows to address general problems, such as PageRank of internet webpages, with complexity O(1), i.e., the time to solve a ranking problem remains fixed irrespective of the size of the coefficient matrix [4]. In this work, we address two specific problems of AI acceleration, namely regression and optimization.Linear regression can be viewed as one of the most basic problems of machine learning: given a sequence of points, one must predict the value of the next point in the sequence. This problem can be solved in just one computational step by using the closed-loop crosspoint array of Fig. 1, where the independent variables x_i are physically written within the two crosspoint arrays, while the dependent variables y_i are applied externally as input currents. The crosspoint arrays are connected within a closed-loop configuration by using operational amplifiers (OAs) in the rows and the columns of the arrays. The OA output yields the coefficients of the best fitting line along the sequence, i.e., the coefficients a and b in the line equation y= ax + b providing the least mean squares (LMS) regression of the sequence. This problem can be extended to any arbitrary dimension, i.e., not only data on x-y plane, but also data on x-y-z space and any hyperspace of n dimensions. The problem can also be extended to logistic regression, which enables classification in one computational step. In particular, the crosspoint regression circuit allows the training of a 2-layer perceptron in just one step, by assuming random weights in the first layer according to the extreme learning machine (ELM) approach. In-memory computing thus appears as a promising option for accelerating neural network training with complexity O(1).The continuous-time, analogue closed-loop approach can be extended to a discrete-time, digital recurrent neural network (RNN) for addressing optimization problems [5]. The RNN allows the efficient solution of constraint satisfaction problems (CSPs), such as Sudoku, by taking advantage of analogue MVM in the crosspoint array. In the RNN, stochastic iterations can allow the minimization of the cost function, hence the number of violated constraints, thus leading to the global minimum. Noise can be injected as spike jitter by phase change memory devices as integrate-and-fire neurons. The convergence to the solution can be efficiently accelerated by a 2-layer RNN, where the first layer is programmed to prevent the violation of constraints in the CSP. In-memory solution of CSPs can dramatically improve the energy efficiency and speed of data intensive optimization in transport, industry, and society in general.

The use of deep learning (DL) to real-world applications, such as computer vision, speech recognition, and robotics, has become ubiquitous. This can be largely attributed to a virtuous cycle between algorithms, data and computing, and storage capacity, which has driven rapid advances in all these dimensions. The ever-increasing demand for computation and memory from DL workloads presents challenges across the entire spectrum of computing platforms, from edge devices to the cloud. Hence, there is a need to explore new hardware paradigms that go well beyond the current mainstays such as graphical processing units (GPUs), tensor processing units (TPUs), and neural processing units (NPUs). A key bottleneck of current platforms is the so-called memory wall, which arises from the need to move large amounts of data between memory and compute units, expending considerable time and energy. One promising solution to this challenge is to move some computations either closer to memory, within the memory subsystem, or even within individual memory arrays. This approach, which is broadly referred to as compute-in-memory (CiM), has the potential to break the memory wall, and thereby greatly improve speed and power consumption. In this article, we provide an overview of CiM techniques used at different levels of the memory hierarchy and based on different memory technologies, including static random access memories (SRAMs), nonvolatile memories (NVMs), and DRAMs. We also discuss architectural approaches to designing CiM-based DL accelerators. Finally, we discuss the challenges associated with adopting CiM in future DL accelerators.

The rapid development of wearable electronics requires a revolution of power accessories regarding flexibility and energy density. The Li-CO2 battery was recently proposed as a novel and promising candidate for next-generation energy-storage systems. However, the current Li-CO2 batteries usually suffer from the difficulties of poor stability, low energy efficiency, and leakage of liquid electrolyte, and few flexible Li-CO2 batteries for wearable electronics have been reported so far. Herein, a quasi-solid-state flexible fiber-shaped Li-CO2 battery with low overpotential and high energy efficiency, by employing ultrafine Mo2 C nanoparticles anchored on a carbon nanotube (CNT) cloth freestanding hybrid film as the cathode, is demonstrated. Due to the synergistic effects of the CNT substrate and Mo2 C catalyst, it achieves a low charge potential below 3.4 V, a high energy efficiency of ≈80%, and can be reversibly discharged and charged for 40 cycles. Experimental results and theoretical simulation show that the intermediate discharge product Li2 C2 O4 stabilized by Mo2 C via coordinative electrons transfer should be responsible for the reduction of overpotential. The as-fabricated quasi-solid-state flexible fiber-shaped Li-CO2 battery can also keep working normally even under various deformation conditions, giving it great potential of becoming an advanced energy accessory for wearable electronics.

The von Neumann bottleneck is a major challenge in the development of energy-efficient processors capable of handling high-workload computations. Computing-in-memory (CiM) technique offers a promising solution to overcome the memory wall restrictions that limit performance. By embedding processing units directly into memory, CiM can mitigate issues of latency and energy consumption during memory access. In this study, we implemented a dual-port design method for a 10-Transistor (10T) SRAM bit-cell to perform Binarized Multiply-Accumulate operation using 45nm CMOS process technology. We use several functional block designs, including isolated read and write paths, to design the 1Kb CiM architecture using the Cadence Virtuoso EDA tool. The proposed 10T SRAM-CiM design supports fully parallel computing, allowing it to perform 32 Binarized MAC operations simultaneously. The design achieves a maximum operating frequency of 100MHz, a throughput of 204.8 GOPS and energy efficiency of 443.15 TOPS/W.

The memory wall presents a significant challenge in computing, highlighting the increasing gap between processor and memory speeds. This discrepancy has sparked extensive research efforts to bridge the divide between memory units and processors. An approach is to minimize latency by integrating memory and processing units, specifically aiming to enhance memory processing capabilities. In this context, in-memory computing has emerged as a solution offering swift performance and streamlined data retrieval. This study addresses the memory wall issue by employing Block RAM (BRAM), a common component in digital design and FPGA programming, as an in-memory logic. The focus is on utilizing BRAM to store partial products from a Wallace Tree Multiplier (WTM), thereby facilitating rapid and efficient data storage within the FPGA fabric. The effectiveness of this method was demonstrated through simulation using Xilinx Vivado, showcasing its potential in overcoming the challenges posed by the memory wall.

Deep Neural Networks (DNN) have emerged as a dominant algorithm for machine learning (ML). High performance and extreme energy efficiency are critical for deployments of DNN, especially in mobile platforms such as autonomous vehicles, cameras, and other devices of internet of things. However, DNNs lead to massive data movement and memory accesses, which prevents it from being integrated into always-on Internet-of-Things (IoT) devices. Recently, computing in-memory (CIM) architectures embeds analog computation circuits in/near the memory arrays. It significantly reduces data movement energy. This paper summarizes the most recent novel methods on the CIM architectures based on the time-domain computation. Compared with voltage-domain and frequency-domain analog computing method, time-domain computation provides more flexibility, higher accuracy and greater scalability for larger neural networks. Thereafter, the first in-memory binary weight network (BWN) processor based on pulse-width modulation in which the feature is stored in memory is also presented. This work significantly reduces memory accesses (4x), and achieves state-of-the-art peak energy efficiency of 119.7TOPS/W.

Deep learning and nonconvex optimization problems are well known data-intensive applications. Although graphic processing units (GPUs) have become the mainstream platform to accelerate the algorithms in the cloud, there is a growing interest to develop application-specific integrated-circuit (ASIC) chips for further improving the energy-efficiency for these data-intensive workloads. Digital multiply-and-accumulate (MAC) arrays are generally employed as ASIC solutions, and data flow is often optimized to increase the data reuse on-chip. Nevertheless, most of the inputs and outputs are moved across MAC arrays and from global buffers. Therefore, it is more attractive to embed the MAC computations into the memory array itself, namely compute-in-memory (CIM), to minimize the data transfer. In CIM, the vector–matrix multiplication is executed in parallel (with analog computation) where the input vectors activate multiple rows. The dot-product is obtained as the multiplication of input voltage and cell conductance, and the partial sum is added up by the column current. An analog-to-digital converter (ADC) at the edge of the array generally converts the partial sum to binary bits for further digital processing.

In-memory computing (IMC) has been proposed to overcome the von Neumann bottleneck in data-intensive applications. However, existing IMC solutions could not achieve both high parallelism and high flexibility, which limits their application in more general scenarios: As a highly parallel IMC design, the functionality of a MAC crossbar is limited to the matrix-vector multiplication; Another IMC method of logic-in-memory (LiM) is more flexible in supporting different logic functions, but has low parallelism. To improve the LiM parallelism, we are inspired by investigating how the single-instruction, multiple-data (SIMD) instruction set in conventional CPU could potentially help to expand the number of LiM operands in one cycle. The biggest challenge is the inefficiency in handling non-continuous data in parallel due to the SIMD limitation of (i) continuous address, (ii) limited cache bandwidth, and (iii) large full-resolution parallel computing overheads. This article presents GRAPHIC, the first reported in-memory SIMD architecture that solves the parallelism and irregular data access challenges in applying SIMD to LiM. GRAPHIC exploits content-addressable memory (CAM) and row-wise-accessible SRAM. By providing the in-situ, full-parallelism, and low-overhead operations of address search, cache read-compute-and-update, GRAPHIC accomplishes high-efficiency gather and aggregation with high parallelism, high energy efficiency, low latency, and low area overheads. Experiments in both continuous data access and irregular data pattern applications show an average speedup of 5x over iso-area AVX-like LiM, and 3-5x over the emerging CAM-based accelerators of CAPE and GaaS-X in advanced techniques.

As the demand for artificial intelligence continues to grow, there is a need for specialized hardware that can efficiently process and power AI algorithms. This is where VLSI (Very Large-Scale Integration) design comes into play. By integrating millions, or even billions, of transistors onto a single chip, VLSI design enables the creation of powerful AI-integrated circuits. However, designing hardware specifically for artificial intelligence poses its own set of challenges. The complexity of AI algorithms and the need for parallel processing capabilities demand innovative solutions. VLSI designers must carefully consider factors such as power consumption, heat dissipation, and the integration of specialized neural network accelerators. In addition to these considerations, VLSI designers must also address the scaling limitations of semiconductor technology. As AI algorithms become more complex, the demand for advanced VLSI architecture increases. Designers must find ways to enhance transistor density and performance while overcoming the limitations imposed by the physical laws governing semiconductor manufacturing. In-Memory Computing (IMC) is an emerging paradigm designed to overcome the limitations of the traditional Von Neumann architecture, particularly for energy-efficient AI applications. Traditional computing systems suffer from the "memory wall"—a bottleneck caused by the constant transfer of data between the CPU and memory. This leads to inefficiencies in latency, bandwidth, and energy consumption, especially for AI tasks which involve huge volumes of matrix operations and data movement. Keywords: Very Large-Scale Integration (VLSI), Artificial Intelligence (AI), In-memory computing (IMC), Neuromorphic, multiply-and-accumulate (MAC), RRAM (Resistive RAM), MRAM (Magnetoresistive RAM), PCM (Phase-Change Memory), FeFET (Ferroelectric FET), spiking neural networks (SNNs)