Computational Failure Analysis of In-Memory RRAM Architecture for Pattern Classification CNN Circuits

Resistive random access memory (ReRAM) demonstrates the great potential of in-memory processing for neural network (NN) acceleration. However, since the convolutional neural network (CNN) is widely known as compute-bound, current ReRAM-based accelerators are not able to support CNN efficiently. In this paper, we for the first time propose the CNN accelerator with Winograd's convolution on ReRAM (CNNWire), which minimizes the multiplications to enable fast and efficient CNN inference. We realize the convolution with Winograd Processing Element (WPE) based on convolutional tiles. Interconnections between WPEs are designed aiming to improve the data reuse. Finally, we introduce the full mapping flow to implement the Winograd convolution The results show that CNMWire gains 3.85x energy efficiency boosting and 3.24x speedup on average among different CNN benchmarks, compared with traditional GEMM based mapping. © 2019 ACM.

Convolutional Neural Networks (CNNs) have achieved excellent performance on various artificial intelligence (AI) applications, while a higher demand on energy efficiency is required for future AI. Resistive Random-Access Memory (RRAM)-based computing system provides a promising solution to energy-efficient neural network training. However, it's difficult to support high-precision CNN in RRAM-based hardware systems. Firstly, multi-bit digital-analog interfaces will take up most energy overhead of the whole system. Secondly, it's difficult to write the RRAM to expected resistance states accurately; only low-precision numbers can be represented. To enable CNN training based on RRAM, we propose a low-bitwidth CNN training method, using low-bitwidth convolution outputs (CO), activations (A), weights (W) and gradients (G) to train CNN models based on RRAM. Furthermore, we design a system to implement the training algorithms. We explore the accuracy under different bitwidth combinations of (A, CO, W, G), and propose a practical tradeoff between accuracy and energy overhead. Our experiments demonstrate that the proposed system perform well on low-bitwidth CNN training tasks. For example, training LeNet-5 with 4-bit convolution outputs, 4-bit weights, 4-bit activations and 4-bit gradients on MNIST can still achieve 97.67% accuracy. Moreover, the proposed system can achieve 23.0X higher energy efficiency than GPU when processing the training task of LeNet-5, and 4.4X higher energy efficiency when processing the training task of ResNet-20.

Convolutional Neural Networks (CNNs) play a vital role in machine learning. CNNs are typically both computing and memory intensive. Emerging resistive random-access memories (RRAMs) and RRAM crossbars have demonstrated great potentials in boosting the performance and energy efficiency of CNNs. Compared with small crossbars, large crossbars show better energy efficiency with less interface overhead. However, conventional workload mapping methods for small crossbars cannot make full use of the computation ability of large crossbars. In this paper, we propose an Overlapped Mapping Method (OMM) and MIxed Size Crossbar based RRAM CNN Accelerator (MISCA) to solve this problem. MISCA with OMM can reduce the energy consumption caused by the interface circuits, and improve the parallelism of computation by leveraging the idle RRAM cells in crossbars. The simulation results show that MISCA with OMM can achieve 2.7× speedup, 30% utilization rate improvement, and 1.2× energy efficiency improvement on average compared with fixed size crossbars based accelerator using the conventional mapping method. In comparison with GPU platform, MISCA with OMM can perform 490.4× higher on average in energy efficiency and 20× higher on average in speedup. Compared with PRIME, an existing RRAM based accelerator, MISCA has 26.4× speedup and 1.65× energy efficiency improvement.

Objects and processes classification is a common experimental problem. Its solution, first of all, is needed in automatic diagnosis systems, for example, to determine the equipment operation state through the diagnostic signal or to identify abnormalities with medical images. With the development of convolutional neural networks, new prospects for solving such problems have risen up. However, the classification accuracy that can be achieved on these networks is not sufficient enough for all diagnosis issues. It is subject to, for example, timely diagnosis of the onset of transient phenomena. At the same time, another type of neural network, Kohonen self-organizing maps, has a conceptual property for training on unclassified set of classes, that is, giving the opportunity to solve such an issue. Therefore, the accuracy enhancement of the classification on the basis of Kohonen networks implementation in the architecture of convolutional networks is a relevant objective and has practical significance. The article analyzes the means of improving the accuracy of convolutional neural networks and arising problems solution. The ways of increasing the proportion of correct clustering on Kohonen networks due to the ‘growth’ of its grid shape in determining new classes in the learning process are also given. It is this property that makes it possible to recognize transient phenomena. It is determined that the existing solutions of the combination of Kohonen networks and convolutional networks are aimed at improving the efficiency of only self-organizing maps, with the purpose to improve the accuracy of classification by convolutional networks, it became necessary to develop a new architecture. The paper provides a description of this issue. Since the initial information of the Kohonen networks is the weighting matrix values of the grid shape neurons, it was necessary to associate it with the representation of the images in order to process the diagnostic images. The paper proposes the concept of a built-in associative array block based on Kohonen networks. According to the proposed method, a software implementation of a hybrid neural network is developed. The formulation and results of computational experiments are presented. The efficiency of the proposed method is experimentally proved. Keywords: convolutional neural networks, CNN, Kohonen self-organizing maps, SOM, signal classification, image classification.

Hyperspectral Image (HSI) classification is one of the fundamental tasks in the field of remote sensing data analysis. CNN (Convolutional Neural Network) has been proven to be an effective deep learning model, which can extract high-level features directly from the raw data and thereby utilize rich information contained in HSI data. However, labor cost to label enough HIS data for training model is usually expensive, so that it is a strong demand of utilizing limited training data to get a satisfied classification accuracy. In this paper, we put forward a deep cube CNN model – DCNR, which is composed of a cube neighbor HSI pixels strategy, a deep CNN and a random forest classifier. In DCNR model, cubic samples, containing spectral-spatial information, are generated by putting each target pixel and its neighbors together. Then features with high representative ability, extracted by applying a specially designed cube CNN model on each cubic sample, are fed into the random forest classifier for the classification of the target pixel. Results show that DCNR model can achieve classification accuracy of 96.78%, 96.08% and 94.85% on KSC, IP and SA datasets respectively with 20% samples as training set, and 85.03%, 83.45 and 62.17% on KSC, IP and SA datasets respectively with only 1% samples as training set, significantly outperforming random forest and cube CNN models.

Resistive random access memory (ReRAM) has been proven capable to efficiently perform in-situ matrix-vector computations in convolutional neural network (CNN) processing. The computations are often conducted on multi-level cell (MLC) that have limited precision and hence, show significant vulnerability to noises. The binarized neural network (BNN) is a hardware-friendly model that can dramatically reduce the computation and storage overheads. However, XNOR, which is the key operation in BNNs, cannot be directly computed in-situ in ReRAM because of its nonlinear behavior. To enable real in-situ processing of BNNs in ReRAM, we modified the BNN algorithm to enable direct computation of XNOR, POPCOUNT and POOL based on ReRAM cells. We also proposed the complementary resistive cell (CRC) design to efficiently conduct XNOR operations and optimized the pipeline design with decoupled buffer and computation stages. Our results show our scheme, namely, ReBNN, improves the system performance by $$25.36\times$$ and the energy efficiency by $$4.26\times$$ compared to conventional ReRAM based accelerator, and ensures a throughput higher than state-of-the-art BNN accelerators. The correctness of the modified algorithm is also validated.

Deep neural networks (DNNs) are tremendously applied in artificial intelligence field. While the performance of DNNs is continuously improved by more complicated and deeper structures, the feasibility of deployment on embedded system remains as a critical problem. Neuromorphic system designs based on resistive random-access memory (ReRAM) provide an opportunity for power-efficient DNN employment. However, it encounters the challenge of limited programming resolution. A quantized training method is proposed in this paper to enhance the performance of neuromorphic systems based on ReRAM. Different from previous methods in which a dedicated regularization term is used in loss function to constrain parameter distribution, our quantized training method deals with training and quantization at the same time to alleviate the impact of limited parameter precision. Models with discrete parameters obtained after training can be directly mapped onto ReRAM devices. We implement experiments on image recognition tasks using multi-layer perceptron (MLP) and convolution neural network (CNN). The results verify that the quantized training method can approximate the accuracy of full-precision training, e.g., a two-layer MLP based on binary ReRAM only decreases the classification accuracy by 0.25% for MNIST dataset. In addition, we carefully investigate and present the importance of layer size under the ReRAM's low programming resolution, the different parameter resolution demands for convolution layer and fully connected layer, and system robustness to ReRAM variations after quantized training. The codes are available at https://github.com/qingyangqing/quantized-rram-net.

Training Convolutional Neural Networks(CNNs) is both memory-and computation-intensive. The resistive random access memory (ReRAM) has shown its advantage to accelerate such tasks with high energy-efficiency. However, the ReRAM-based pipeline architecture suffers from the low utilization of computing resource, caused by the imbalanced data throughput in different pipeline stages because of the inherent down-sampling effect in CNNs and the inflexible usage of ReRAM cells. In this paper, we propose a novel ReRAM-based bidirectional pipeline architecture, named HUBPA, to accelerate the training with higher utilization of the computing resource. Two stages of the CNN training, forward and backward propagations, are scheduled in HUBPA dynamically to share the computing resource. We design an accessory control scheme for the context switch of these two tasks. We also propose an efficient algorithm to allocate computing resource for each neural network layer. Our experiment results show that, compared with state-of-the-art ReRAM pipeline architecture, HUBPA improves the performance by 1.7X and reduces the energy consumption by 1.5X, based on the current benchmarks.

The report presents an algorithm for constructing a convolutional hierarchical neural network classifier, which is a modification of the algorithm for constructing hierarchical neural network classifiers suggested before. The original algorithm was designed to exploit intrinsic class hierarchy to build a class tree with a neural network in each node classifying groups of initial classes (in a non-terminal node) or a subset of original classes (in a terminal node). The convolutional modification utilizes convolutional neural networks instead of regular fully connected networks in order to apply the model to image classification tasks. Use of class hierarchy for image classification should reduce the number of adjusted neural network parameters compared to deep convolutional neural networks, and therefore it should reduce training and inference time. In this context the algorithm may be compared with some pruning techniques. The convolutional hierarchical neural network classifier inherits some hyperparameters of a conventional hierarchical neural network classifier, like the activation threshold and the threshold by the share of voting patterns. The goal of this study was to explore different strategies of choosing these hyperparameters. To test these strategies, we used the CIFAR-10 dataset. Also, for demonstration purposes we apply the convolutional hierarchical neural network classifier to the CIFAR-100 dataset.

Deep learning has been successfully applied in classification of white blood cells (WBCs), however, accuracy and processing time are found to be less than optimal hindering it from getting its full potential. This is due to imbalanced dataset, intra-class compactness, inter-class separability and overfitting problems. The main research idea is to enhance the classification and prediction accuracy of blood images while lowering processing time through the use of deep convolutional neural network (DCNN) architecture by using the modified loss function. The proposed system consists of a deep neural convolution network (DCNN) that will improve the classification accuracy by using modified loss function along with regularization. Firstly, images are pre-processed and fed through DCNN that contains different layers with different activation function for the feature extraction and classification. Along with modified loss function with regularization, weight function aids in the classification of WBCs by considering weights of samples belonging to each class for compensating the error arising due to imbalanced dataset. The processing time will be counted by each image to check the time enhancement. The classification accuracy and processing time are achieved using the dataset-master. Our proposed solution obtains better classification performance in the given dataset comparing with other previous methods. The proposed system enhanced the classification accuracy of 98.92% from 96.1% and a decrease in processing time from 0.354 to 0.216 s. Less time will be required by our proposed solution for achieving the model convergence with 9 epochs against the current convergence time of 13.5 epochs on average, epoch is the formation white blood cells (WBCs) and the development of granular cells. The proposed solution modified loss function to solve the adverse effect caused due to imbalance dataset by considering weight and use regularization technique for overfitting problem.

Realization of spiking neural network (SNN) hardware with high energy efficiency and high integration may provide a promising solution to data processing challenges in future internet of things (IoT) and artificial intelligence (AI). Recently, design of multi-core reconfigurable SNN chip based on resistive random-access memory (RRAM) is drawing great attention, owing to the unique properties of RRAM, e.g., high integration density, low power consumption, and processing-in-memory (PIM). Therefore, RRAM-based SNN chip may have further improvements in integration and energy efficiency. The design of such a chip will face the following problems: significant delay in pulse transmission due to complex logic control and inter-core communication; high risk of digital, analog, and RRAM hybrid design; and non-ideal characteristics of analog circuit and RRAM. In order to effectively bridge the gap between device, circuit, algorithm, and architecture, this paper proposes a simulation model—FangTianSim, which covers analog neuron circuit, RRAM model and multi-core architecture and its accuracy is at the clock level. This model can be used to verify the functionalities, delay, and power consumption of SNN chip. This information cannot only be used to verify the rationality of the architecture but also guide the chip design. In order to map different network topologies on the chip, SNN representation format, interpreter, and instruction generator are designed. Finally, the function of FangTianSim is verified on liquid state machine (LSM), fully connected neural network (FCNN), and convolutional neural network (CNN).

The resistive random-access memory (ReRAM) has widely been used to accelerate convolutional neural networks (CNNs) thanks to its analog in-memory computing capability. ReRAM crossbars not only store layers’ weights, but also perform in-situ matrix-vector multiplications which are core operations of CNNs. To boost the performance of ReRAM-based CNN accelerators, crossbars can be duplicated to explore more intra-layer parallelism. The crossbar allocation scheme can significantly influence both the computing throughput and bandwidth requirements of ReRAM-based CNN accelerators. Under the resource constraints (i.e., crossbars and memory bandwidths), how to find the optimal number of crossbars for each layer to maximize the inference performance for an entire CNN is an unsolved problem. In this work, we find the optimal crossbar allocation scheme by mathematically modeling the problem as a constrained optimization problem and solving it with a dynamic programming based solver. Experiments demonstrate that our model for CNN inference time is almost precise, and the proposed framework can obtain solutions with near-optimal inference time. We also emphasize that communication (i.e., data access) is an important factor and must also be considered when determining the optimal crossbar allocation scheme.

FitCNN: A cloud-assisted and low-cost framework for updating CNNs on IoT devices

Resource-constrained computing devices such as those used in IoTs require low-power, high performance, and small size to be enabled to operate efficiently. Resistive random access memory (ReRAM) is a promising technology for building novel in-memory computing architectures, due to its ability to perform storage and computation using the same physical element with low energy and high density. ReRAMbased search engines and neural network (NN) accelerators have grown significantly especially for IoT devices. In this paper, we propose a memristor-based voltage-resistance xnor (VR-XNOR) cell. The advantages of this cell are demonstrated through building a reconfigurable content-addressable memory (CAM) architecture that can support binary-CAM (BCAM) and ternary-CAM (TCAM) and enable approximate search operations. Moreover, the memristor-based VR-XNOR cell is utilized for binarized convolutional neural networks (CNN) with focus on the convolution operation. This is achieved by replacing the convolution module with XNOR-based filter banks. Simulations of the proposed architectures for search engine and feature extraction were carried out using VTEAM model in Cadence Virtuoso Analog Design Environment. The proposed filter bank architecture achieves a 1-ns extraction cycle time over N filters and produces multiple output feature maps in a single processing cycle. The filter uses two memristor devices to realize each XNOR gate and has shown a significant reduction in the number of multiply-add operations.

A genetic algorithm (GA), is used to optimize the many parameters of a convolutional neural network (CNN) that control the structure of the network. CNNs are used in image classification problems where it is necessary to generate feature descriptors to discern between image classes. Because of the deep representation of image data that CNNs are capable of generating, they are increasingly popular in research and industry applications. With the increasing number of use cases for CNNs, more and more time is being spent to come up with optimal CNN structures for different applications. Where one CNN might succeed at classification, another can fail. As a result, it is desirable to more easily find an optimal CNN structure to increase classification accuracy. In the proposed method, a GA is used to evolve the parameters that influence the structure of a CNN. The GA compares CNNs by training them on images of concrete containing cracks. The best CNN after several generations of the GA is then compared to the state-of-the-art CNN for crack detection. This work shows that it is possible to generalize the process of optimizing a CNN for image classification through the use of a GA.