Capacitor-less RRAM-based stochastic neuron for event-based unsupervised learning

Processing-in-memory (PIM) is a promising solution to address the "memory wall" challenges for future computer systems. Prior proposed PIM architectures put additional computation logic in or near memory. The emerging metal-oxide resistive random access memory (ReRAM) has showed its potential to be used for main memory. Moreover, with its crossbar array structure, ReRAM can perform matrix-vector multiplication efficiently, and has been widely studied to accelerate neural network (NN) applications. In this work, we propose a novel PIM architecture, called PRIME, to accelerate NN applications in ReRAM based main memory. In PRIME, a portion of ReRAM crossbar arrays can be configured as accelerators for NN applications or as normal memory for a larger memory space. We provide microarchitecture and circuit designs to enable the morphable functions with an insignificant area overhead. We also design a software/hardware interface for software developers to implement various NNs on PRIME. Benefiting from both the PIM architecture and the efficiency of using ReRAM for NN computation, PRIME distinguishes itself from prior work on NN acceleration, with significant performance improvement and energy saving. Our experimental results show that, compared with a state-of-the-art neural processing unit design, PRIME improves the performance by ~2360× and the energy consumption by ~895×, across the evaluated machine learning benchmarks.

Processing-in-memory (PIM) is a promising solution to address the "memory wall" challenges for future computer systems. Prior proposed PIM architectures put additional computation logic in or near memory. The emerging metal-oxide resistive random access memory (ReRAM) has showed its potential to be used for main memory. Moreover, with its crossbar array structure, ReRAM can perform matrix-vector multiplication efficiently, and has been widely studied to accelerate neural network (NN) applications. In this work, we propose a novel PIM architecture, called PRIME, to accelerate NN applications in ReRAM based main memory. In PRIME, a portion of ReRAM crossbar arrays can be configured as accelerators for NN applications or as normal memory for a larger memory space. We provide microarchitecture and circuit designs to enable the morphable functions with an insignificant area overhead. We also design a software/hardware interface for software developers to implement various NNs on PRIME. Benefiting from both the PIM architecture and the efficiency of using ReRAM for NN computation, PRIME distinguishes itself from prior work on NN acceleration, with significant performance improvement and energy saving. Our experimental results show that, compared with a state-of-the-art neural processing unit design, PRIME improves the performance by ~2360x and the energy consumption by ~895x, across the evaluated machine learning benchmarks.

Hyperspectral data is not linearly separable, and it has a high characteristic dimension. This paper proposes a new algorithm that combines a deep belief network based on the Boltzmann machine with a self-organizing neural network. The primary features of the hyperspectral image are extracted with a deep belief network. The weights of the network are fine-tuned using the labeled sample. Feature vectors extracted by the deep belief network are classified by a self-organizing neural network. The method reduces the spectral dimension of the data while preserving the large amount of original information in the data. The method overcomes the long training time required when using self-organizing neural networks for clustering, as well as the training difficulties of Deep Belief Networks (DBN) when the labeled sample size is small, thereby improving the accuracy and robustness of the semi-supervised classification. Simulation results show that the structure of the network can achieve higher classification accuracy when the labeled sample is deficient.

The training of neural network (NN) is usually time-consuming and resource intensive. Memristor has shown its potential in computation of NN. Especially for the metal-oxide resistive random access memory (RRAM), its crossbar structure and multi-bit characteristic can perform the matrix-vector product in high precision, which is the most common operation of NN. However, there exist two challenges on realizing the training of NN. Firstly, the current architecture can only support the inference phase of training and cannot perform the backpropagation (BP), the weights update of NN. Secondly, the training of NN requires enormous iterations and constantly updates the weights to reach the convergence, which leads to large energy consumption because of lots of write and read operations. In this work, we propose a novel architecture, TIME, and peripheral circuit designs to enable the training of NN in RRAM. TIME supports the BP and the weights update while maximizing the reuse of peripheral circuits for the inference operation on RRAM. Meanwhile, a variability-free tuning scheme and gradually-write circuits are designed to reduce the cost of tuning RRAM. We explore the performance of both SL (supervised learning) and DRL (deep reinforcement learning) in TIME, and a specific mapping method of DRL is also introduced to further improve the energy efficiency. Experimental results show that, in SL, TIME can achieve 5.3× higher energy efficiency on average compared with the most powerful application-specific integrated circuits (ASIC) in the literature. In DRL, TIME can perform averagely 126× higher than GPU in energy efficiency. If the cost of tuning RRAM can be further reduced, TIME have the potential of boosting the energy efficiency by 2 orders of magnitude compared with ASIC.

Processing-in-memory (PIM) exploits massive parallelism with high energy efficiency and becomes a promising solution to the von Neumann bottleneck. Recently, the emerging metal-oxide resistive random access memory (RRAM) shows its potential to construct a PIM architecture, because several stateful logic operations, e.g., IMP and NOR, can be executed in an RRAM crossbar in parallel. Previous synthesis flows focus on improving latency with stateful logic operations, but they ignore that the memory should be used primarily for storage. i.e., most of the area in the crossbar is used for computation but not storage. In this situation, storage and computation still have to be separated into different crossbars, which leads to considerable data transfer overhead and limited parallelism. In this work, we define the ratio of storage in a crossbar as area utilization. We aim to improve the area utilization without throughput loss by proposing STAR, a novel synthesis flow for the stateful logic. We present two optimization strategies to reduce the computation area in STAR. First, we reduce the area for redundant inputs. For the shared constants among different rows (or columns), we encode them as immediate values into the control signals without writing them into the crossbar at runtime. For the other inputs, we only store one copy of them in the crossbar. Second, we reduce the area for intermediate variables by reusing invalid cells. And we design a scheduling algorithm to find a computation sequence with the minimal variable erasing cycles. Invalid primary inputs can also be erased in this algorithm. Furthermore, we present a case study of the image convolution to demonstrate the effectiveness of STAR. Experimental evaluation shows that STAR achieves 33.03% more area utilization and a 1.43x throughput compared to SIMPLER, the state-of-the-art stateful logic synthesis flow. Our image convolution implementation also provides 78.36% more area utilization and a 1.48x throughput compared with IMAGING, the state-of-the-art stateful logic-based image processing accelerator.

2 - Metal-oxide resistive random access memory (RRAM) technology: Material and operation details and ramifications

A low-energy hardware implementation of deep belief network (DBN) architecture is developed using near-zero energy barrier probabilistic spin logic devices (p-bits), which are modeled to realize an intrinsic sigmoidal activation function. A CMOS/spin based weighted array structure is designed to implement a restricted Boltzmann machine (RBM). Device-level simulations based on precise physics relations are used to validate the sigmoidal relation between the output probability of a p-bit and its input currents. Characteristics of the resistive networks and p-bits are modeled in SPICE to perform a circuit-level simulation investigating the performance, area, and power consumption tradeoffs of the weighted array. In the application-level simulation, a DBN is implemented in MATLAB for digit recognition using the extracted device and circuit behavioral models. The MNIST data set is used to assess the accuracy of the DBN using 5,000 training images for five distinct network topologies. The results indicate that a baseline error rate of 36.8% for a 784x10 DBN trained by 100 samples can be reduced to only 3.7% using a 784x800x800x10 DBN trained by 5,000 input samples. Finally, Power dissipation and accuracy tradeoffs for probabilistic computing mechanisms using resistive devices are identified.

PurposeUp to date development in sentiment analysis has resulted in a symbolic growth in the volume of study, especially on more subjective text types, namely, product or movie reviews. The key difference between these texts with news articles is that their target is defined and unique across the text. Hence, the reviews on newspaper articles can deal with three subtasks: correctly spotting the target, splitting the good and bad content from the reviews on the concerned target and evaluating different opinions provided in a detailed manner. On defining these tasks, this paper aims to implement a new sentiment analysis model for article reviews from the newspaper.Design/methodology/approachHere, tweets from various newspaper articles are taken and the sentiment analysis process is done with pre-processing, semantic word extraction, feature extraction and classification. Initially, the pre-processing phase is performed, in which different steps such as stop word removal, stemming, blank space removal are carried out and it results in producing the keywords that speak about positive, negative or neutral. Further, semantic words (similar) are extracted from the available dictionary by matching the keywords. Next, the feature extraction is done for the extracted keywords and semantic words using holoentropy to attain information statistics, which results in the attainment of maximum related information. Here, two categories of holoentropy features are extracted: joint holoentropy and cross holoentropy. These extracted features of entire keywords are finally subjected to a hybrid classifier, which merges the beneficial concepts of neural network (NN), and deep belief network (DBN). For improving the performance of sentiment classification, modification is done by inducing the idea of a modified rider optimization algorithm (ROA), so-called new steering updated ROA (NSU-ROA) into NN and DBN for weight update. Hence, the average of both improved classifiers will provide the classified sentiment as positive, negative or neutral from the reviews of newspaper articles effectively.FindingsThree data sets were considered for experimentation. The results have shown that the developed NSU-ROA + DBN + NN attained high accuracy, which was 2.6% superior to particle swarm optimization, 3% superior to FireFly, 3.8% superior to grey wolf optimization, 5.5% superior to whale optimization algorithm and 3.2% superior to ROA-based DBN + NN from data set 1. The classification analysis has shown that the accuracy of the proposed NSU − DBN + NN was 3.4% enhanced than DBN + NN, 25% enhanced than DBN and 28.5% enhanced than NN and 32.3% enhanced than support vector machine from data set 2. Thus, the effective performance of the proposed NSU − ROA + DBN + NN on sentiment analysis of newspaper articles has been proved.Originality/valueThis paper adopts the latest optimization algorithm called the NSU-ROA to effectively recognize the sentiments of the newspapers with NN and DBN. This is the first work that uses NSU-ROA-based optimization for accurate identification of sentiments from newspaper articles.

Precision mechanism is widely used for various industry applications. Quality inspection for precision mechanism is essential for manufacturers to assure the product leaving factory with expected quality. In this paper, we propose a novel automated fault detection method, named Tilear, based on a Deep Belief Network (DBN) auto-encoder. DBN is a probabilistic generative model, composed by stacked Restricted Boltzmann Machines. With its RBM-layer-wise training methods, DBN can perform fast inference and extract high level feature of the inputs. By unfolding the stacked RBMs symmetrically, a DBN auto-encoder is constructed to reconstruct the inputs as closely as possible. Based on the DBN auto-encoder, Tilear is structured in two parts: training and decision-making. During training, Tilear is trained with the signals only from good samples, which enables the trained DBN auto-encoder only know how to reconstruct signals of good samples. In the decision-making part, comparing the recorded signal from test sample and the Tilear reconstructed signal, allows to measure how well a recording from a test sample matches the DBN auto-encoder model learned from good samples. A reliable decision could be made. We perform experiments on two different precision mechanisms: precision electromotors and greasing control units. The feasibility of Tilear was demonstrated first. Additionally, performance of Tilear on the acquired electromotor dataset was compared with the state-of-the-art machine learning based fault detection technique, support vector machine (SVM). First result indicates that Tilear excels the SVM in terms of the Area Under the Curve (AUC) obtained from the Receiver Operating Characteristics (ROC) curve plot: 0.960 achieved by Tilear, while 0.941 by SVM.

Deep belief network (DBN) is an effective learning model based on deep learning. It can hierarchically transform the input data via stacked feature detectors. As a predictive model, DBN has shown a promising prospect in model predictive control (MPC). However, its successful application relies seriously on the suitable structure size (the numbers of hidden layers and neurons), which is challenging to determine. In this work, we present a theoretical bound on its minimum structure size in order to accurately approximate a desired control law of MPC, called DBN-MPC. First, according to the Markov assumption, a controlled system is equivalent to a quadratic program, which only depends on the current system state. Second, a universal theorem is proposed to give a bound on the minimum structure size of DBN from the perspective of piecewise affine function analysis. Third, a partial least square regression is used to fine-tune DBN to overcome the problems of local-minimum and time-consuming training process. Finally, we demonstrate the effectiveness of the proposed method through two classical experiments: 1) tracking control of a benchmark dynamical system and 2) temperature control of a practical second-order continuous stirred tank reactor (CSTR) system. The experimental results generally give an answer to the question that how deep is deep enough for DBN to approximate an MPC law. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —The proposed deep belief network (DBN)-model predictive control (MPC) scheme presents an answer to the question that how deep is deep enough for DBN, from the perspective of approximating an MPC law. Given an industrial system, the proposed DBN-MPC scheme can give the best tracking control performance with minimal computational complexity through an effective self-growing DBN and optimal approximation theorem of MPC law. In practical applications, it is better for practitioners to obtain accurate historical input and output data. During the operation process, DBN-MPC will be more practical if more accurate initial structure size of DBN is provided in advance because the initial structure size of DBN is the starting point of the dynamic optimization.

Deep belief networks (DBNs) are stochastic neural networks that can extract rich internal representations of the environment from the sensory data. DBNs had a catalytic effect in triggering the deep learning revolution, demonstrating for the very first time the feasibility of unsupervised learning in networks with many layers of hidden neurons. These hierarchical architectures incorporate plausible biological and cognitive properties, making them particularly appealing as computational models of human perception and cognition. However, learning in DBNs is usually carried out in a greedy, layer-wise fashion, which does not allow to simulate the holistic maturation of cortical circuits and prevents from modeling cognitive development. Here we present iDBN, an iterative learning algorithm for DBNs that allows to jointly update the connection weights across all layers of the model. We evaluate the proposed iterative algorithm on two different sets of visual stimuli, measuring the generative capabilities of the learned model and its potential to support supervised downstream tasks. We also track network development in terms of graph theoretical properties and investigate the potential extension of iDBN to continual learning scenarios. DBNs trained using our iterative approach achieve a final performance comparable to that of the greedy counterparts, at the same time allowing to accurately analyze the gradual development of internal representations in the deep network and the progressive improvement in task performance. Our work paves the way to the use of iDBN for modeling neurocognitive development.

Graph algorithms such as graph traversal have been gaining ever-increasing importance in the era of big data. However, graph processing on traditional architectures issues many random and irregular memory accesses, leading to a huge number of data movements and the consumption of very large amounts of energy. To minimize the waste of memory bandwidth, we investigate utilizing processing-in-memory (PIM), combined with non-volatile metal-oxide resistive random access memory (ReRAM), to improve both computation and I/O performance. We propose a new ReRAM-based processing-in-memory architecture called RPBFS, in which graph data can be persistently stored and processed in place. We study the problem of graph traversal, and we design an efficient graph traversal algorithm in RPBFS. Benefiting from low data movement overhead and high bank-level parallel computation, RPBFS shows a significant performance improvement compared with both the CPU-based and the GPU-based BFS implementations. On a suite of real-world graphs, our architecture yields a speedup in graph traversal performance of up to 33.8×, and achieves a reduction in energy over conventional systems of up to 142.8×.

With the increased development of information technology, almost all the sectors have been developed. Age, educational qualifications, gender, and other factors have no bearing on acquiring knowledge in information technology.Most humans use mobile phones and other gadgets to make their lives easier. Machine Learning techniques are used to analyse the given data and aid in the classification or prediction of the dataset depending on the problem statement. It is significant to determine human behaviour analysis in the context of sports. In this research, the Deep Learning-Deep Belief Network (DL-DBN) algorithm is implemented with probability to analyse human behaviour in sports and implement a distributed probability model for classifying the behavior. The classification results have shown that the accuracy for strength training is both the maximum and the smallest, reaching 99% and 71%, respectively.

Supply chain flexibility is the processing ability of the enterprize to deal with the uncertain environment of supply and demand. In this paper, we consider the supply side (node interrupt) and demand side (demand fluctuations) under uncertain environment. By using the deep belief network (DBN), which is composed of multilayer Restricted Boltzmann Machine (RBM), it establishes the supply chain flexibility network with optimization of the transfer node and flow. The deep belief network is trained by the data of large manufacturing enterprize, compared with the traditional neural network (MLR, BP and GA). The results show that the deep belief network model overcomes the shortcomings of the traditional neural networks, such as easy to fall into local optimum, long training time and low function fitting degree, and it has higher prediction accuracy. This network model based on the deep belief network can promote the supply chain flexibility more, when supply and demand fluctuations occur.

With the development of Industry 4.0, not only the equipment but also the operational conditions in industrial manufacturing are becoming more and more complex. It is necessary to diagnose failures, whose probability is now increasing violently. As a typical deep learning model, the Deep Belief Network (DBN) can be employed to extract features from the original data directly. Compared with traditional fault diagnosis methods, the DBN can get rid of the dependence on signal processing technology and diagnosis experience. In this paper, the fault diagnosis approach based on DBN is studied to identify the bearing failure. First of all, the basic principles of DBN and the steps of fault diagnosis are described. Then some key parameters of DBN which affect the fault identification performance are analyzed and determined according to the simulation experiments. The practicability of this method is verified by comparing with Support Vector Machine (SVM) and Back Propagation Neural Network (BPNN) at last.