ISSA: Architecting CNN Accelerators Using Input-Skippable, Set-Associative Computing-in-Memory

The rapid progress in artificial intelligence (AI) is driving the need for more efficient computing architectures capable of performing the demanding, repetitive computations required for processing large datasets. Unlike traditional von Neumann architectures, which separate processing and memory units, new approaches aim to integrate computational capabilities directly into memory arrays, facilitating in-memory computing (IMC). This integration significantly improves both energy efficiency and processing speed, particularly for AI and similar applications. A crucial operation within IMC is vector-matrix multiplication (VMM), where input signal vectors interact with a memory state matrix to produce output signal vectors.One innovative approach to implementing VMM is through the use of a ferroelectric Hf₀.₅Zr₀.₅O₂ (HZO) layer in conventional field-effect transistor (FET) or metal-insulator-metal (MIM) devices. These HZO-based ferroelectric components show significant promise for low-power, IMC-compatible systems, especially when integrated into MIM capacitor structures. Their simple fabrication process supports scalable system integration, while offering benefits such as low power consumption and high reliability. However, current IMC techniques utilizing ferroelectric MIM capacitors rely on small-signal capacitance readout, which suffers from limited contrast between high and low capacitance states, making it difficult to distinguish memory states effectively for efficient IMC operations.In this study, we introduce two novel readout techniques to improve the scalability and robustness of IMC, focusing on enhanced memory differentiation by reading the ferroelectric remanent polarization (Pᵣ) instead of relying on small-signal capacitance. These methods exploit the switching behavior between two stable polarization states (Pₗₒʷ and Pᵤₚ) of the HZO material. To measure charge accumulation resulting from polarization switching, we connect the ferroelectric HZO capacitors in series with either a shunt resistor (Rₛₕᵤₙₜ) or an external capacitor (Cₑₓt) to produce current or voltage signals, respectively. Both readout methods show significantly improved memory differentiation and VMM performance compared to the traditional small-signal capacitance approach. For example, in the current readout method, the maximum displacement current in the high-current state (Iₛʷ + Iₙₒₙ-ˢʷ) is approximately 0.31 mA, while the low-current state (Iₙₒₙ-ˢʷ) is about 0.04 mA when no switching current is present. Notably, the static leakage current during the input pulse is nearly zero, underscoring the potential for low-power operation in capacitive systems. In the voltage readout method, an initial Pᵤₚ state leads to a high output voltage (Vₒᵤₜ) of approximately 0.13 V, while an initial Pₗₒʷ state produces a low Vₒᵤₜ of about 0 V. These polarization-based readout techniques provide substantially higher memory distinction and better VMM performance compared to traditional small-signal capacitance methods.We will demonstrate IMC systems by using voltage readout of the Pr change in HZO layers, employing charge compensation between the Cext and HZO capacitors. This method takes advantage of the superior memory distinction within the capacitive system, offering more reliable and lower-power performance compared to traditional Fe-FET approaches.The authors acknowledge Prof. Joseph S. Friedman and Dr. Peng Zhou for the technical discussion.

Memristor-based analogue computing in memory (CIM) offers revolutionary gains in energy efficiency and computing power for data-intensive applications such as artificial intelligence. However, it typically struggles with achieving high accuracy at the same time, owing to the noise-sensitive nature of analogue computing and the non-ideal characteristics at the device and circuit levels that inevitably result in computing errors. Although progress has been made in device engineering and hardware-algorithm co-optimization to mitigate the error and parasitic effects, many of these advances inadvertently incur a hardware or energy consumption overhead, undermining the core benefits of analogue CIM. This Review dissects the computing error sources across the CIM hierarchy from the memristor device and array to the system architecture and algorithm, and evaluates the strategies to minimize those errors. We highlight the material and device innovations, array-level techniques and algorithm-architecture co-design frameworks towards high-accuracy analogue CIM. By dissecting the trade-off between computing accuracy and implementation cost, this Review draws a roadmap for translating memristor-based analogue CIM technology from proof-of-concept prototypes to large-scale deployment for accelerating next-generation artificial intelligence.

Compute-in-memory (CIM) has recently emerged as a promising design paradigm to accelerate deep neural network (DNN) processing. Continuously better energy and area efficiency at the macrolevel had been reported through many testchips over the last few years. However, in those macro design-oriented studies, accelerator-level considerations, such as memory accesses and processing of entire DNN workloads have not been investigated in-depth. In this article, we aim to fill this gap starting with the characteristics of our latest CIM macro fabricated with cutting-edge FinFET CMOS technology at 4-nm node. We then study, through an accelerator simulator developed in-house, three key items that would determine the efficiency of our CIM macro in the accelerator context while running MLPerf Mobile suite: 1) dataflow optimization; 2) optimal selection of CIM macro dimensions to further improve macro utilization; and 3) optimal combination of multiple CIM macros. Although there is typically a stark contrast between macro-level peak and accelerator-level average throughput and energy efficiency, the aforementioned optimizations are shown to improve the macro utilization by <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$3.04\times $ </tex-math></inline-formula> and reduce the energy-delay product (EDP) to <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$0.34\times $ </tex-math></inline-formula> compared to the original macro on MLPerf Mobile inference workloads. While we exploit a digital CIM macro in this study, the findings and proposed methods remain valid for other types of CIM (such as analog CIM and analog–digital–hybrid CIM) as well.

Despite its promising potential for Artificial Intelligence (AI) applications, current In-Memory Computing (IMC) technology faces a variety of challenges before mass production. One of the major challenges we face is the absence of efficient toolchains for deploying canonical networks on IMC chips. To address this issue, we propose a co-designed framework that integrates circuit, toolchain, and system elements specifically for IMC. More specifically, our framework consists of several key techniques to improve the key performance including (a) an 8-bit hardware-friendly Quantization-Aware Training (QAT) approach to quantify the deep learning network from floating-point data to fixed-point data, (b) a novel operator optimization technique to increase the computing precision when running the algorithm models on the IMC chips, and (c) an efficient mapping strategy based on the Integer Linear Programming (ILP) approach to improve the computation resource utilization of the IMC array. We assess our method on our 40nm eFlash-based IMC SoC chip with voice recognition, speech noise reduction, and person detection tasks. Our experimental results show an accuracy over 94.60% in a quiet environment and 87.27% in a white noise environment and a false recognition rate below 1 time per 24 hours for voice recognition, a 21.53% improvement for the Perceptual Evaluation of Speech Quality (PESQ) for noise reduction, and a 97.80% accuracy in person detection.

Deep learning architectures are increasingly limited by the memory wall — the “von Neumann bottleneck” that forces frequent data transfers between memory and processing units, throttling throughput and energy efficiency. To alleviate this bottleneck, researchers have explored Compute-in-Memory (CiM) architectures to accelerate the core computing primitives such as the matrix–vector multiplications (MVMs), within memory arrays. Yet, large language models based on transformers also require many non-linear softmax operations, creating a new bottleneck. From a technology perspective, non-volatile STT-MRAMs are promising for CiM thanks to their endurance, nonvolatility, and area footprint. However, conventional 1T–1R STT-MRAM CiM arrays place bit-cells in parallel, yielding lower MVM energy efficiency than 8T-SRAM CiM due to low device resistance; line parasitics and low OFF/ON ratios of STT-MRAM devices further limit row parallelism and scalability. In this context, we propose ROSETTA, a CiM array based on 3T–2R STT-MRAM bit-cells that uses series-resistance sensing and time-to-digital conversion (TDC) for low-power MVMs. An added word line also stores a ROM bit, enabling in-array tables for fast softmax, without increasing bit-cell area. We introduce a novel palindromic encoding of inputs and weights to mitigate the data-dependent nonlinearity in STT-MRAM CiM arrays based on TDC sensing. This enables 8× higher row-level parallelism than standard 1T–1R MRAM CiM and facilitates scaling to larger CiM arrays. Our macro occupies ∼ 47% less area and consumes 3.1× less MVM energy than an equivalent ROM-overlaid 8T-SRAM macro, at similar latency and accuracy. At the system level, a spatial architecture comprising our macros exhibits 11.7−14.5× and 1.44 − 1.5× higher energy efficiency than equivalent architectures comprising 1T-1R MRAM and 8T-SRAM ROM-overlaid CiM macros, respectively.

Computing-in-memory (CIM) relieves the Von Neumann bottleneck by storing the weights of neural networks in memory arrays. However, two challenges still exist, hindering the efficient acceleration of convolutional neural networks (CNN) in artificial intelligence (AI) edge devices. Firstly, the activations for sliding window (SW) operations in CNN still bring high memory access pressure. This can be alleviated by increasing the SW parallelism, but simple array replication suffers from poor array utilization and large peripheral circuits overhead. Secondly, the partial sums from individual CIM arrays, which are usually accumulated to obtain the final sum, introduce large latency due to enormous shift-and-add operations. Moreover, high-resolution ADCs are also needed to reduce the quantization error of partial sums, further increasing the hardware costs. In this paper, a hardware-efficient CIM accelerator, ARBiS, is proposed with improved activation reusability and bit-scalable matrix-vector-multiplication (MVM) for CNN acceleration in AI edge applications. The cyclic-shift weight duplication exploits a third dimension of receptive field (RF) depth for SW weight mapping to reduce the memory accesses of activations, improving the array utilization. The parasitic-capacitance charge sharing is employed to realize high-precision analog MVM in order to reduce the ADC cost. Compared with conventional architectures, ARBiS with parallel processing of 9 SW operations achieves 56.6%~58.8% alleviation of memory access pressure. Meanwhile, ARBiS configured with 8-bit ADCs saves 92.53%~94.53% ADC energy consumption. An ARBiS accelerator is evaluated to realize a computational efficiency (CE) of 10.28 (10.43) TOPS/mm2, an energy efficiency (EE) of 91.19 (112.36) TOPS/W with 8-bit (4-bit) ADCs, achieving <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$11.4\sim 11.7\times $ </tex-math></inline-formula> ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$11.6\sim 11.8\times $ </tex-math></inline-formula> ), <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$1.1\sim 3.3\times $ </tex-math></inline-formula> ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$1.4\sim 4\times $ </tex-math></inline-formula> ) improvements over state-of-the-art works, respectively.

The proliferation of advanced analytics and artificial intelligence has been driven by huge volumes of data that are mostly generated at the edge. Simultaneously, there is a rising demand to perform analytics on edge platforms (i.e., near-sensor data analytics). However, conventional architectures of such platforms may not execute the targeted applications in an energy-efficient manner. Emerging near and in-memory computing paradigms can increase the energy efficiency of edge platforms by relying on emerging logic and memory devices. More importantly, these paradigms enable the possibility of performing computations on unconventional platforms, namely flexible computing systems. In this paper, we explore the benefits of in-memory computing at the edge on a flexible substrate enabled by thin-film transistors (TFTs) and resistive RAM (RRAM). As a case study, we consider bio-signal processing application workloads, i.e., compressive sensing and anomaly detection. We model the device, circuit, and architecture of our targeted platform and evaluate the corresponding system-level performance. Preliminary results indicate that in-memory computing enabled by flexible electronic devices enables a new class of edge platforms with lower power consumption, compared to that of rigid TFT devices.

The dramatic rise of data-intensive workloads has revived special-purpose hardware and architectures for continuing improvements in computational speed and energy efficiency. While traditional CMOS ASICs deliver some performance gains, typically by limiting data movement or implementing “in-memory computation”, such approaches still suffer from low power efficiency. Therefore, new proposals leveraging non-volatile memristive devices for in-memory computation are highly attractive in a variety of application domains. While originally developed for as digital (binary) high density non-volatile memories, memristive devices have demonstrated a wide range of behaviors and properties – such as a wide range of tunable analog resistance and non-linear dynamics – which motivate their use in novel functions and new computational models. Many recent in-memory compute studies have focused on crossbar circuit architectures, demonstrating their application for neural networks, scientific computing and signal processing. However, other circuit primitives – such as content addressable memories (CAMs) and combined systems such as crossbar arrays and non-linear elements– have shown further promise for mapping a diverse range of complimentary computational models such as finite state machines, pattern matching, hashing algorithms and Hopfield networks for tackling optimization problems. In this talk, I will review the different exciting opportunities for in-memory computational primitives levering non-volatile memristive devices and their circuits and architectures for enabling low power, high-throughput computation in a variety of application domains. I will also highlight the different materials and device requirements which steer material stack design and fabrication for these diverse circuits and application domains. Finally, I will show recent lab demonstrations of various applications mapped to these in-memory computational circuit primitives based on memristor devices, and give an outlook on performance.

As the demand for artificial intelligence continues to grow, there is a need for specialized hardware that can efficiently process and power AI algorithms. This is where VLSI (Very Large-Scale Integration) design comes into play. By integrating millions, or even billions, of transistors onto a single chip, VLSI design enables the creation of powerful AI-integrated circuits. However, designing hardware specifically for artificial intelligence poses its own set of challenges. The complexity of AI algorithms and the need for parallel processing capabilities demand innovative solutions. VLSI designers must carefully consider factors such as power consumption, heat dissipation, and the integration of specialized neural network accelerators. In addition to these considerations, VLSI designers must also address the scaling limitations of semiconductor technology. As AI algorithms become more complex, the demand for advanced VLSI architecture increases. Designers must find ways to enhance transistor density and performance while overcoming the limitations imposed by the physical laws governing semiconductor manufacturing. In-Memory Computing (IMC) is an emerging paradigm designed to overcome the limitations of the traditional Von Neumann architecture, particularly for energy-efficient AI applications. Traditional computing systems suffer from the "memory wall"—a bottleneck caused by the constant transfer of data between the CPU and memory. This leads to inefficiencies in latency, bandwidth, and energy consumption, especially for AI tasks which involve huge volumes of matrix operations and data movement. Keywords: Very Large-Scale Integration (VLSI), Artificial Intelligence (AI), In-memory computing (IMC), Neuromorphic, multiply-and-accumulate (MAC), RRAM (Resistive RAM), MRAM (Magnetoresistive RAM), PCM (Phase-Change Memory), FeFET (Ferroelectric FET), spiking neural networks (SNNs)

A review on selective in-memory computing processors: Potential alternative to AI-driven applications

Compute-in-memory (CIM) is a promising technique that reduces data movement in neural network (NN) acceleration. To achieve higher efficiency, some recent CIM accelerators exploit NN sparsity based on CIM’s small-grained operation unit (OU) feature. However, new problems arise in a practical multi-macro accelerator: The mismatch between workload parallelism and CIM macro organization causes spatial under-utilization; The multiple macros’ different computation time leads to temporal under-utilization. To solve the under-utilization problems, we propose a Sparsity-balanced Practical CIM accelerator (SPCIM), including optimized dataflow and hardware architecture design. For the CIM dataflow design, we first propose a reconfigurable cluster topology for CIM macro organization. Then we regularize weight sparsity in the OU-height pattern and reorder the weight matrix based on the sparsity ratio. The cluster topology can be reshaped to match workload parallelism for higher spatial utilization. Each CIM cluster’s workload is dynamically rebalanced for higher temporal utilization. Our hardware architecture supports the proposed dataflow with a spatial input dispatcher and a temporal workload allocator. Experimental results show that, compared with the baseline sparse CIM accelerator that suffers from spatial and temporal under-utilization, SPCIM achieves <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$2.94\times $ </tex-math></inline-formula> speedup and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$2.86\times $ </tex-math></inline-formula> energy saving. The proposed sparsity-balanced dataflow and architecture are generic and scalable, which can be applied to other CIM accelerators. We strengthen two state-of-the-art CIM accelerators with the SPCIM techniques, improving their energy efficiency by <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$1.92\times $ </tex-math></inline-formula> and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$5.59\times $ </tex-math></inline-formula> , respectively.

Artificial intelligence (AI) has successfully made its way into contemporary industrial sectors such as automobiles, defense, industrial automation 4.0, healthcare technologies, agriculture, and many other domains because of its ability to act autonomously without continuous human interventions. However, this capability requires processing huge amounts of learning data to extract useful information in real time. The buzz around AI is not new, as this term has been widely known for the past half century. In the 1960s, scientists began to think about machines acting more like humans, which resulted in the development of the first natural language processing computers. It laid the foundation of AI, but there were only a handful of applications until the 1990s due to limitations in processing speed, memory, and computational power available. Since the 1990s, advancements in computer architecture and memory organization have enabled microprocessors to deliver much higher performance. Simultaneously, improvements in the understanding and mathematical representation of AI gave birth to its subset, referred to as machine learning (ML). ML includes different algorithms for independent learning, and the most promising ones are based on brain-inspired techniques classified as artificial neural networks (ANNs). ANNs have subsequently evolved to have deeper and larger structures and are often characterized as deep neural networks (DNN) and convolution neural networks (CNN). In tandem with the emergence of multicore processors, ML techniques started to be embedded in a range of scenarios and applications. Recently, application-specific instruction-set architecture for AI applications has also been supported in different microprocessors. Thus, continuous improvement in microprocessor capabilities has reached a stage where it is now possible to implement complex real-time intelligent applications like computer vision, object identification, speech recognition, data security, spectrum sensing, etc. This paper presents an overview on the evolution of AI and how the increasing capabilities of microprocessors have fueled the adoption of AI in a plethora of application domains. The paper also discusses the upcoming trends in microprocessor architectures and how they will further propel the assimilation of AI in our daily lives.

The Ph.D. dissertation focuses on mixed-signal-based SRAM computation-in-memory (CIM) for artificial intelligence (AI) edge devices. CIM is an emerging approach that integrates computation capabilities into memory units, reducing data movement and enhancing performance for memory-intensive tasks like deep neural networks (DNNs). CIM mainly deals with multiply-and-accumulate (MAC) operations which consist most of the computing burden in DNN. Additionally, mixed-signal-based CIM has many advantages over digital-based CIM in terms of area, throughput, and power in low-precision DNN (1-4b) as able to accumulate multiplication results at once with analog domain (voltage, current and charge, etc.) Therefore, mixed-signal-based CIM is an attractive option as low-precision DNNs have achieved competitive precision so far.

In-memory computing (IMC) is gaining momentum for low-power acceleration of data intensive problems, particularly in the frame of artificial intelligence (AI). Analogue matrix-vector multiplication (MVM) in crosspoint arrays allows to dramatically improve energy efficiency in neural networks for inference and training [1]. Recently, it has been shown that a feedback configuration of MVM can accelerate the solution of generic linear matrix problems, such as linear systems, matrix inversion and eigenvector calculation, with a single computational step [2,3]. This is achieved by the high parallelism of MVM in the memory array, combined with the physical iteration executed naturally in the feedback loop. Eigenvector extraction allows to address general problems, such as PageRank of internet webpages, with complexity O(1), i.e., the time to solve a ranking problem remains fixed irrespective of the size of the coefficient matrix [4]. In this work, we address two specific problems of AI acceleration, namely regression and optimization.Linear regression can be viewed as one of the most basic problems of machine learning: given a sequence of points, one must predict the value of the next point in the sequence. This problem can be solved in just one computational step by using the closed-loop crosspoint array of Fig. 1, where the independent variables x_i are physically written within the two crosspoint arrays, while the dependent variables y_i are applied externally as input currents. The crosspoint arrays are connected within a closed-loop configuration by using operational amplifiers (OAs) in the rows and the columns of the arrays. The OA output yields the coefficients of the best fitting line along the sequence, i.e., the coefficients a and b in the line equation y= ax + b providing the least mean squares (LMS) regression of the sequence. This problem can be extended to any arbitrary dimension, i.e., not only data on x-y plane, but also data on x-y-z space and any hyperspace of n dimensions. The problem can also be extended to logistic regression, which enables classification in one computational step. In particular, the crosspoint regression circuit allows the training of a 2-layer perceptron in just one step, by assuming random weights in the first layer according to the extreme learning machine (ELM) approach. In-memory computing thus appears as a promising option for accelerating neural network training with complexity O(1).The continuous-time, analogue closed-loop approach can be extended to a discrete-time, digital recurrent neural network (RNN) for addressing optimization problems [5]. The RNN allows the efficient solution of constraint satisfaction problems (CSPs), such as Sudoku, by taking advantage of analogue MVM in the crosspoint array. In the RNN, stochastic iterations can allow the minimization of the cost function, hence the number of violated constraints, thus leading to the global minimum. Noise can be injected as spike jitter by phase change memory devices as integrate-and-fire neurons. The convergence to the solution can be efficiently accelerated by a 2-layer RNN, where the first layer is programmed to prevent the violation of constraints in the CSP. In-memory solution of CSPs can dramatically improve the energy efficiency and speed of data intensive optimization in transport, industry, and society in general.

Compute-in-memory (CiM) is one promising solution to address the memory bottleneck existing in traditional computing architectures. However, the tradeoff between energy efficiency and computing precision plagues most CiM implementations, and the low precision imposes a major limitation on CiM’s ability to support practical computational workloads. In this article, a static random access memory (SRAM)-based analog CiM macro is presented with the Intel 22FFL process. By introducing a 1-to-2 ratioed capacitor ladder (C-2C)-based charge domain computing scheme, the proposed CiM prototype chip demonstrates a maximum of 2k multiply-accumulate (MAC) operations in one clock cycle and achieves 32.2-TOPS/W peak power efficiency with 8-bit precision in both input activation and weight while ensuring accurate on-chip matrix–vector multiplications (MVMs) with a computation error less than 0.5%. A 4.0-TOPS/mm2 peak area efficiency is attained by adopting a local weight multiplexing scheme with a 9T SRAM cell, which improves the memory density and reduces the need to refresh the weight stored in the SRAM array. A variety of analog impairment factors, including parasitics, mismatch, and noise, were analyzed to guarantee a sufficiently high multibit linearity. The proposed passive analog computing mechanism ensures the computation accuracy over process–voltage–temperature (PVT), with the measured MVM error deviation of less than 1% over PVT variations.