Optimizing Artificial Neural Networks to Minimize Arithmetic Errors in Stochastic Computing Implementations

As the energy and hardware investments necessary for conventional high-precision digital computing continue to explode in the era of artificial intelligence (AI), a change in paradigm that can trade precision for energy and resource efficiency is being sought for many computing applications. Stochastic computing (SC) is an attractive alternative since, unlike digital computers, which require many logic gates and a high transistor volume to perform basic arithmetic operations such as addition, subtraction, multiplication, sorting, etc., SC can implement the same using simple logic gates. While it is possible to accelerate SC using traditional silicon complementary metal-oxide-semiconductor (CMOS) technology, the need for extensive hardware investment to generate stochastic bits (s-bits), the fundamental computing primitive for SC, makes it less attractive. Memristor and spin-based devices offer natural randomness but depend on hybrid designs involving CMOS peripherals for accelerating SC, which increases area and energy burden. Here, the limitations of existing and emerging technologies are overcome, and a standalone SC architecture embedded in memory and based on 2Dmemtransistors is experimentally demonstrated. The monolithic and non-von-Neumann SC architecture occupies a small hardware footprint and consumes a miniscule amount of energy (<1nJ) for both s-bit generation and arithmetic operations, highlighting the benefits of SC.

5G network is an enhanced communication network, designed to converge the requirements of quality of service (QoS) parameters, data and capacity, by means of signals with high QoS and high-speed data rate. Several state-of-art technologies are involved to obtain the requirement of energy efficient communication system with increased number of users, devices, higher data rate with low latency. This paper presents a system which demonstrates the energy and spectral efficiency achieved for various number of nodes in a specific area. This study addresses the improvement in energy and spectral efficiency when the proposed algorithm is used. The proposed algorithm is a combination of swarm based artificial bee colony (ABC) algorithm with neural network. Experimental results have been carried out to observe the performance of QoS parameters such as bit error rate (BER), throughput, power consumption and mean square error (MSE). The maximum energy efficiency achieved is 34% and Spectral efficiency is 36%.

The demand for cost-effective digital filters in near-sensor computing has aroused interest in stochastic computing (SC) as a promising alternative to traditional binary approaches. This study investigates in detail the error tolerance and hardware cost of SC-based digital filters of both the finite and infinite impulse response (FIR and IIR) types, focusing on their stochastic number generation and addition methods. We demonstrate that SC filters need only about half the area and power of binary designs under error-free conditions, though with a slight reduction of accuracy. Furthermore, these filters exhibit strong error resilience, with only 4X and 6X performance degradation under a 3% bit-flip rate, significantly outperforming their binary (non-SC) counterparts. We also compare the error tolerance of SC filters against binary filters equipped with triple modular redundancy (TMR). Our results indicate that for FIR filters, SC-based designs remain far more error-tolerant than binary FIR filters with TMR. However, in the IIR case, the binary designs with TMR outperform SC-based IIR filters at higher error rates. These findings highlight the possible trade-offs between SC and binary implementations, with SC offering more robust error tolerance, significant area and power savings and manageable accuracy trade-offs in most cases. This work also underscores SC’s potential as an efficient, resilient approach for low-cost digital filtering in edge devices and near-sensor computing applications.

Stochastic neuromorphic computation (SNC) has the potential to enable a low power, error tolerant and scalable computing platform in comparison to its deterministic counterparts. However, the hardware implementation of complementary metal oxide semiconductor (CMOS)-based stochastic circuits involves conversion blocks that cost more than the actual processing circuits. The realization of the activation function for SNCs also requires a complicated circuit that results in a significant amount of power dissipation and area overhead. The inherent probabilistic switching behavior of nanomagnets provides an advantage to overcome these complexity issues for the realization of low power and area efficient SNC systems. This paper presents magnetic tunnel junction (MTJ)-based stochastic computing methodology for the implementation of a neural network. The stochastic switching behavior of the MTJ has been exploited to design a binary to stochastic converter to mitigate the complexity of the CMOS-based design. The paper also presents the technique for realizing stochastic sigmoid activation function using an MTJ. Such circuits are simpler than existing ones and use considerably less power. An image classification system employing the proposed circuits has been implemented to verify the effectiveness of the technique. The MTJ-based SNC system shows area and energy reduction by a factor of 13.5 and 2.5, respectively, while the prediction accuracy is 86.66%. Furthermore, this paper investigates how crucial parameters, such as stochastic bitstream length, number of hidden layers and number of nodes in a hidden layer, need to be set precisely to realize an efficient MTJ-based stochastic neural network (SNN). The proposed methodology can prove a promising alternative for highly efficient digital stochastic computing applications.

Deep neural networks (DNNs) have been revolutionizing machine learning research and various applications by achieving better accuracy than human perception. However, a large amount of computational resources are needed due to their complex topologies, restricting the usage of DNNs in embedded systems and portable devices with limited areas and power budgets. This chapter considers stochastic computing (SC) as an alternative to binarized computing. SC uses a random bit sequence where the signal value is computed by counting the number of ones in the bit sequence. This allows the implementation of major arithmetic units like multiplication and addition by a simple logic. Consequently, SC can implement DNNs with essentially low hardware overhead and also offers good scalability. First, the chapter presents a detailed study of processing elements used in neural networks. Then, combining several building blocks, such as stochastic approximation of activation functions in SC, saturation arithmetic, and scaling, a new scheme is proposed for DNN inference and training in SC. At last, the chapter shows that SC-compatible DNNs can be effectively deployed on low-cost hardware. Deep neural network, optimization, accelerator, custom computing, stochastic computing

Both binarized and stochastic computing-based neural networks exploit bit-wise operations to replace expensive full-precision multiplications with simple XNOR gates and thus, offer low-cost hardware implementation. In stochastic computing, arithmetic computations are performed on sequences of random bits which can approximate any real values. Stochastic computing-based neural networks benefit from approximate computing and promote fault-tolerant architectures against soft errors in noisy environments. On the other hand, in binarized neural networks, real values are deterministically binarized using the sign function. As a result, any bit-flip in the binarized values dramatically changes the outcome of arithmetic computations and makes binarized neural networks more vulnerable against soft errors. In this paper, we compare these two neural networks against each other in terms of fault-tolerance and hardware complexity (i.e., area and energy efficiency).

Stochastic Number Generators (SNG) plays a significant role in designing a stochastic computing system. SNGs make the stochastic system comfortable for computing in the stochastic domain. The challenges in developing the stochastic computing system are correlation and hardware area occupancy. By considering these phenomena, we have considered Linear Feedback Shift Register (LFSR) based SNG and S-box based SNG in this work. Our contributions to this paper are stochastic computation for activation functions using the SNGs mentioned above and stochastic computation for arithmetic components in the stochastic domain. By considering the two SNG methods, the difference in the computation for accuracy has been analyzed for stochastic activation function and stochastic arithmetic computation. The better SNG method will be used as SNG for stochastic convolutional neural network design using this analysis.

Managing energy efficiency is essential for ensuring national energy security and facilitating a transition to a green economy. Outer circulation creates favorable external conditions. Using data from listed firms information in China between 2007 and 2016, this paper examines the effect of cross-border mergers and acquisitions (M&amp;A) on energy efficiency through a PSM-DID model. The research shows that: (1) M&amp;A can effectively manage and improve energy efficiency of domestic enterprises, with the positive impact showing a clear trend of increasing annually. (2) The mechanism analysis suggests that the enhancement of energy efficiency is driven by technological and structural effects. (3) Furthermore, the rise in energy efficiency benefits more from M&amp;A in regions along the Belt and Road Initiative and countries with stricter environmental regulations. The findings of this paper provide important insights into the management of international capital flows, energy efficiency improvement, and sustainable development.

Enhancement of energy efficiency (EE) can help bridge the gap between supply and demand for energy. This paper assesses the energy efficiency and conservation (EE and C) potential of sixteen EE end-use technologies and subsectors (for both primary energy (oil, gas and coal) and electricity) in Bangladesh vis a vis “business-as-usual”. Further, it prioritizes among them on the bases of their potential for generating energy savings, their costs, and the benefits of deployment on a large scale. The end-use EE improvement technologies/measures analyzed range from lights, fans, and refrigerators to motors, boilers, and chillers. Sectors covered include garments, textile dyeing and weaving, steel and cement. They were chosen for analysis because they represent the most promising prospective candidates for demand side energy efficiency improvement in the country. The analysis indicates that a total of 400 petajoule (PJ) can be saved in the year 2030 when the projected total primary energy requirement is likely to be approximately 2800 PJ, i.e., a savings of 14.3 percent of the total primary energy requirement in 2030 can be achieved by implementing EE measures alone. A back-of-the-envelope calculation of the relative cost effectiveness of the different options analyzed provides a measure of the costs and benefits of these EE options. Despite domestic electricity and gas prices being fairly low in comparison with international market prices, the cost of saved energy for most options is either low or negative: in other words, these EE investments have a favorable rate of return and should be a key part of energy sector development in Bangladesh. Higher electricity or gas prices would increase the cost-effectiveness of all options because the monetized value of energy saved would be higher.

This paper presents a Spiking Neural Network (SNN) processor specifically designed to overcome the limitations of existing parallel architectures in maintaining high energy efficiency and model adaptability in a compact area footprint for Artificial Intelligence of Things (AIoT). This is achieved through two key design features: a Temporal-Spatial Post-Neuron Processing (PoNP) scheme that efficiently reuses membrane potential, maximizes parallelism, and reduces memory bank requirements; and a Model-Adaptive Crossbar design with pre-configured parameters and a dynamic switching mechanism enables processing of various SNN models through operation orchestration without efficiency degradation. Using an 8-way parallel pipeline design, the processor achieves a throughput of 128 Synaptic Operations (SOPs) per cycle, resulting in a 2.8$\boldsymbol{\times}$ enhancement in energy efficiency. Fabricated in a 40-nm CMOS process, the chip occupies a compact core area of 0.66 mm${}^{2}$. It achieves a power consumption of 6.26 mW, an energy efficiency of 0.49 pJ/SOP, and a throughput of 12.8 GSOPS at 0.75 V, 100 MHz. The chip is evaluated using typical spatial, temporal, and temporal-spatial datasets, including MIT-BIH, MNIST, N-MNIST, NavGesture, and SHD. These results demonstrate that our chip achieves best-in-class in terms of energy efficiency and latency compared to state-of-the-art architectures.

We consider the design of stochastic computing (SC) hardware based on spintronic devices. SC offers low-cost implementations of arithmetic operations and high degrees of error tolerance. When compared to charge-based devices, spin-based devices could be lower energy, nonvolatile, etc. However, spin-based devices can be fundamentally more error prone than charge-based devices. The marriage of SC architectures and spin-based devices has the potential to produce information processing systems that are robust, low energy, and nonvolatile. In this paper, we investigate SC hardware comprised of nanomagnetic logic (NML) devices. NML is a “beyond-CMOS” technology that uses bistable magnets to store, process, and move binary information. We introduce new NML circuit structures that exploit unique features of the technology to efficiently realize hardware components required for an SC system (e.g., random number generation). We also benchmark NML-based SC circuits against other implementations, and illustrate how features that are unique to NML (e.g., inherent pipelining) can lead to improved performance. Our results indicate NML SC implementations achieve smaller area footprints with reduced energy and delay when compared to CMOS equivalents.

As we approach the limits of traditional Moore’s-Law scaling, alternative computing techniques that consume energy more efficiently become attractive. Stochastic computing (SC), as a re-emerging computing technique, is a low-cost and error-tolerant alternative to conventional binary circuits in several important applications such as image processing and communications. SC allows a natural accuracy-energy tradeoff that has been exploited in the past. This article presents an accuracy-energy tradeoff technique for SC circuits that reduces their energy consumption with virtually no accuracy loss. To this end, we employ voltage or frequency scaling, which normally reduce energy consumption at the cost of timing errors. Then we show that due to their inherent error tolerance, SC circuits operate satisfactorily without significant accuracy loss even with aggressive scaling. This significantly improves their energy efficiency. In contrast, conventional binary circuits quickly fail as the supply voltage decreases. To find the most energy-efficient operating point of an SC circuit, we propose an error estimation method that allows us to quickly explore the circuit’s design space. The error estimation method is based on Markov chain and least-squares regression. Furthermore, we investigate opportunities to optimize SC circuits under such aggressive scaling. We find that logical and physical design techniques can be combined to significantly expand the already-powerful accuracy-energy tradeoff possibilities of SC. In particular, we demonstrate that careful adjustment of path delays can lead to significant error reduction under voltage and frequency scaling. We perform buffer insertion and route detouring to achieve more balanced path delays. These techniques differ from conventional path-balancing techniques whose goal is to minimize power consumption by resizing the non-critical paths. The goal of our path-balancing approach is to increase error cancellation chances in voltage-/frequency-scaled SC circuits. Our circuit optimization comprehends the tradeoff between power overheads due to inserted buffers and wires versus the energy reduction from supply voltage downscaling enabled by more balanced path delays. Simulation results show that our optimized SC circuits can tolerate aggressive voltage scaling with no significant signal-to-noise ratio (SNR) degradation. In one example, a 40% supply voltage reduction (1V to 0.6V) on the SC circuit leads to 66% energy saving (20.7pJ to 6.9pJ) and makes it more efficient than its conventional binary counterpart. In the same example, a 100% frequency boosting (400ps to 200ps) of the optimized circuits leads to no significant SNR degradation. We also show that process variation and temperature variation have limited impact on optimized SC circuits. The error change is less than 5% when temperature changes by 100°C or process condition changes from worst case to best case.

The Long Term Evolution (LTE) standards are generated by the Third Generation Partnership Project (3GPP) for the Fourth Generation (4G) wireless communication systems. The Fourth Generation (4G) wireless communication standards use new physical layer signaling techniques in their physical layer. Single Carrier Frequency Division Multiple Access (SC-FDMA) and Orthogonal Frequency Division Multiple Access (OFDMA) are the two major signaling techniques used in LTE. OFDMA is used in the downlink of the Fourth Generation (4G) wireless communication systems for its high spectral efficiency and SC-FDMA is used in the uplink of the Fourth Generation (4G) wireless communication systems for its high power efficiency. The main objective of this paper is to design and implement the reliable and energy efficient architecture for LTE uplink system using FPGA. IFDMA is the most energy efficient subcarrier mapping scheme of SC-FDMA multiple access scheme. In IFDMA, the data symbols subcarrier mapping is not one to one instead each subcarrier carries a part from each data symbol transmitted at that time instant. The reason for the selection of IFDMA subcarrier mapping along with SC-FDMA multiple access scheme for LTE uplink comes mainly from its advantage to provide a low peak to average power ratio (PAPR) for the transmitted waveform. This results in less power consumption in the mobile station during uplink transmission. The reliability of the system is improved using interleaving process by avoiding packet loss. Convolution codes are less complex and are used to provide error free transmission. The enhancement of energy efficiency is done using the convolution coded IFDMA. The chosen energy efficient architecture must be more power efficient and support high data rates, which is the main requirement for the mobile user. The proposed reliable and energy efficient architecture for LTE uplink system is implemented using Xilinx system generator tool.

Transient bit flips may cause unexpected behaviors in digital arithmetic circuits, and thus computation errors. It is generally believed that stochastic computing (SC) can tolerate bit flips due to its unique number representation format. In SC, a value is encoded by a random bit stream using the probability of 1s in the bit stream. Therefore, one bit flip changes the probability by a small number given a long enough bit stream, and the computation results might not change much. This is verified in previous works mostly through empirical studies. However, we found that bit flips actually change the probability of a stochastic bit stream by a predictable value. This is formulated by considering the bit flips as Bernoulli events. The distorted probability of the bit stream considering bit flips is derived and the SC bit flip models for various stochastic circuits are proposed. The error recovery based on the formulation is also proposed. The SC bit flip model and error recovery scheme are verified on basic SC elements such as the SC multipliers and an FSM-based stochastic circuit. They are then applied to a complex SC system computing a neural network. The results show that the accuracy of MNIST digit recognition can be recovered even with a high bit flip rate of 10%.

Computing on time-based data is a recent evolution of research in stochastic computing. As with stochastic computing, complex functions can be computed with remarkably low area cost. Unlike stochastic computing, the latency and energy efficiency are very favorable compared to computations on conventional binary radix. In this article, the authors review and evaluate the design and implementation of arithmetic operations on time-encoded signals, with a particular focus on low power. The advantages, challenges, and potential applications are discussed.