FAT: An In-Memory Accelerator With Fast Addition for Ternary Weight Neural Networks

Computing-in-memory (CIM) relieves the Von Neumann bottleneck by storing the weights of neural networks in memory arrays. However, two challenges still exist, hindering the efficient acceleration of convolutional neural networks (CNN) in artificial intelligence (AI) edge devices. Firstly, the activations for sliding window (SW) operations in CNN still bring high memory access pressure. This can be alleviated by increasing the SW parallelism, but simple array replication suffers from poor array utilization and large peripheral circuits overhead. Secondly, the partial sums from individual CIM arrays, which are usually accumulated to obtain the final sum, introduce large latency due to enormous shift-and-add operations. Moreover, high-resolution ADCs are also needed to reduce the quantization error of partial sums, further increasing the hardware costs. In this paper, a hardware-efficient CIM accelerator, ARBiS, is proposed with improved activation reusability and bit-scalable matrix-vector-multiplication (MVM) for CNN acceleration in AI edge applications. The cyclic-shift weight duplication exploits a third dimension of receptive field (RF) depth for SW weight mapping to reduce the memory accesses of activations, improving the array utilization. The parasitic-capacitance charge sharing is employed to realize high-precision analog MVM in order to reduce the ADC cost. Compared with conventional architectures, ARBiS with parallel processing of 9 SW operations achieves 56.6%~58.8% alleviation of memory access pressure. Meanwhile, ARBiS configured with 8-bit ADCs saves 92.53%~94.53% ADC energy consumption. An ARBiS accelerator is evaluated to realize a computational efficiency (CE) of 10.28 (10.43) TOPS/mm2, an energy efficiency (EE) of 91.19 (112.36) TOPS/W with 8-bit (4-bit) ADCs, achieving <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$11.4\sim 11.7\times $ </tex-math></inline-formula> ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$11.6\sim 11.8\times $ </tex-math></inline-formula> ), <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$1.1\sim 3.3\times $ </tex-math></inline-formula> ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$1.4\sim 4\times $ </tex-math></inline-formula> ) improvements over state-of-the-art works, respectively.

Convolutional neural networks (CNN) is a widely known deep learning architecture and achieves higher accuracies in speech processing and computer vision applications. CNN has arisen as best option in the fields such as video surveillance, mobile robot vision, smart factories, and medical diagnostics. CNN has shown its tremendous potential for image understanding in Cardio-Vascular (CV) systems. But, the tremendous potential of CNNs came up with additional computational burden. To get an outstanding performance of CNN-based algorithms, it requires large amount of enormous applications and memory resources. Matrix multiplication (MM) is the most fundamental computational operation taking place in CNN. Researchers in this field found that multiplication operation is the most resource-intensive and power-hungry operation in CNN. This work aims to design an FPGA-based CNN using systolic array which can improve both the accuracy and hardware efficiency of convolutional neural network. Here, we made design optimisation in multiplier unit which brings a significant impact on overall performance of convolutional neural networks.

Hyperspectral sensors provide an opportunity to capture the intensity of high spatial/spectral information and enable applications for high-level earth observation missions, such as accurate land cover mapping and target/object detection. Currently, convolutional neural networks (CNNs) are good at coping with hyperspectral image processing tasks because of the strong spatial and spectral feature extraction ability brought by hierarchical structures, but the convolution operation in CNNs is limited to local feature extraction in both dimensions. In the meanwhile, the introduction of the Transformer structure has provided an opportunity to capture long-distance dependencies between tokens from a global perspective; however, Transformer-based methods have a restricted ability to extract local information because they have no inductive bias, as CNNs do. To make full use of these two methods’ advantages in hyperspectral image processing, a dual-flow architecture named Hyper-LGNet to couple local and global features is firstly proposed by integrating CNN and Transformer branches to deal with HSI spatial-spectral information. In particular, a spatial-spectral feature fusion module (SSFFM) is designed to maximally integrate spectral and spatial information. Three mainstream hyperspectral datasets (Indian Pines, Pavia University and Houston 2013) are utilized to evaluate the proposed method’s performance. Comparative results show that the proposed Hyper-LGNet achieves state-of-the-art performance in comparison with the other nine approaches concerning overall accuracy (OA), average accuracy (AA) and kappa index. Consequently, it is anticipated that, by coupling CNN and Transformer structures, this study can provide novel insights into hyperspectral image analysis.

Achieving energy efficiency is of paramount importance in facilitating the deployment of artificial intelligence (AI) applications, particularly for edge devices. The computational complexity and memory requirement of AI workloads have made energy efficiency in circuits the primary design constraint, prompting a need for architectural reconsideration. However, operations in Convolutional Neural Networks (CNN) relying on matrix multiplication offer opportunities for algorithm-level optimization. The dynamic power consumption in CMOS technology exhibits a direct correlation with the switching activities (SA) associated with transitions of data bits from one cycle to the next. Minimizing SA constitutes an effective approach to curbing power consumption, resulting in prolonged battery life. This work introduces a cross-layer end-to-end optimization technique, which involves a one-time sorting of pre-trained model parameters to reduce switching activity during matrix multiplication or convolution operations while eliminating the indexing overhead during inference. While upto 40% reduction in switching activities through sorting has been achieved, the actual saving in dynamic power achievable for the input stationary systolic array is approximately 10% in total power consumption and around 15% in switching power. The proposed method has been validated using various pre-trained networks, including GoogeLeNet, MobileNet, AlexNet, and SqueezeNet, among others.

We propose Absum, which is a regularization method for improving adversarial robustness of convolutional neural networks (CNNs). Although CNNs can accurately recognize images, recent studies have shown that the convolution operations in CNNs commonly have structural sensitivity to specific noise composed of Fourier basis functions. By exploiting this sensitivity, they proposed a simple black-box adversarial attack: Single Fourier attack. To reduce structural sensitivity, we can use regularization of convolution filter weights since the sensitivity of linear transform can be assessed by the norm of the weights. However, standard regularization methods can prevent minimization of the loss function because they impose a tight constraint for obtaining high robustness. To solve this problem, Absum imposes a loose constraint; it penalizes the absolute values of the summation of the parameters in the convolution layers. Absum can improve robustness against single Fourier attack while being as simple and efficient as standard regularization methods (e.g., weight decay and L1 regularization). Our experiments demonstrate that Absum improves robustness against single Fourier attack more than standard regularization methods. Furthermore, we reveal that robust CNNs with Absum are more robust against transferred attacks due to decreasing the common sensitivity and against high-frequency noise than standard regularization methods. We also reveal that Absum can improve robustness against gradient-based attacks (projected gradient descent) when used with adversarial training.

In image recognition, one of the main challenges is preserving discriminative feature information while ensuring spatial invariance. Conventional pooling operations in Convolutional Neural Networks (CNNs), such as max pooling and average pooling, often discard fine-grained details, leading to information loss. To address this, we propose an Autoencoder-like Multilayer Perceptron (MLP) architecture that replaces traditional pooling by projecting feature maps into a compact latent space and reconstructing them before classification. Unlike existing pooling strategies such as Expansion Downsampling learnable-Scaling (EDS) and threshold-based T-Max-Avg pooling, our approach maintains simplicity without requiring additional heuristic parameters or complex mechanisms. Experiments on the MNIST dataset demonstrate that the proposed model achieves an accuracy of 0.982, outperforming both conventional pooling and state-of-the-art pooling methods (EDS: 0.981, Threshold: 0.876), while also reducing training time by 2–5 seconds. Furthermore, when compared to modern CNN architectures, naturally, ResNet-18 and MobileNetV2 achieve very high accuracy due to their deep architectures and residual connections. However, our Autoencoder-like MLP achieves strong performance with Precision: 0.983, Recall: 0.979, F1 Score: 0.980, and an inference time of 7.33 seconds, strengthening latent feature representations while maintaining efficiency, making it well-suited for integration into modern CNNs. Latent representation analysis further confirms that the encoder–decoder design fosters robust and separable feature spaces, effectively retaining critical details for classification. These findings highlight the potential of Autoencoder-like MLPs as an efficient and learnable alternative to pooling, with promising directions for integration into deeper and hybrid CNN architectures in future work.

Recurrent reverse attention guided residual learning for saliency object detection

Capsule Networks (CapsNets) have demonstrated to be a promising alternative to Convolutional Neural Networks (CNNs). However, they often fall short of state-of-the-art accuracies on large-scale high-dimensional datasets. We propose a Detail-Oriented Capsule Network (DECAPS) that combines the strength of CapsNets with several novel techniques to boost its classification accuracies. First, DECAPS uses an Inverted Dynamic Routing (IDR) mechanism to group lower-level capsules into heads before sending them to higher-level capsules. This strategy enables capsules to selectively attend to small but informative details within the data which may be lost during pooling operations in CNNs. Second, DECAPS employs a Peekaboo training procedure, which encourages the network to focus on fine-grained information through a second-level attention scheme. Finally, the distillation process improves the robustness of DECAPS by averaging over the original and attended image region predictions. We provide extensive experiments on the CheXpert and RSNA Pneumonia datasets to validate the effectiveness of DECAPS. Our networks achieve state-of-the-art accuracies not only in classification (increasing the average area under ROC curves from 87.24% to 92.82% on the CheXpert dataset) but also in the weakly-supervised localization of diseased areas (increasing average precision from 41.7% to 80% for the RSNA Pneumonia detection dataset).

Analysis of functional connectivity networks (FCNs) derived from resting-state functional magnetic resonance imaging (rs-fMRI) has greatly advanced our understanding of brain diseases, including Alzheimer's disease (AD) and attention deficit hyperactivity disorder (ADHD). Advanced machine learning techniques, such as convolutional neural networks (CNNs), have been used to learn high-level feature representations of FCNs for automated brain disease classification. Even though convolution operations in CNNs are good at extracting local properties of FCNs, they generally cannot well capture global temporal representations of FCNs. Recently, the transformer technique has demonstrated remarkable performance in various tasks, which is attributed to its effective self-attention mechanism in capturing the global temporal feature representations. However, it cannot effectively model the local network characteristics of FCNs. To this end, in this paper, we propose a novel network structure for Local sequential feature Coupling Global representation learning (LCGNet) to take advantage of convolutional operations and self-attention mechanisms for enhanced FCN representation learning. Specifically, we first build a dynamic FCN for each subject using an overlapped sliding window approach. We then construct three sequential components (i.e., edge-to-vertex layer, vertex-to-network layer, and network-to-temporality layer) with a dual backbone branch of CNN and transformer to extract and couple from local to global topological information of brain networks. Experimental results on two real datasets (i.e., ADNI and ADHD-200) with rs-fMRI data show the superiority of our LCGNet.

Due to sparsity, a significant percentage of the operations carried out in Convolutional Neural Networks (CNNs) contains a zero in at least one of their operands. Different approaches try to take advantage of sparsity in two different ways. On the one hand, sparse matrices can be easily compressed, saving space and memory bandwidth. On the other hand, multiplications with zero in their operands can be avoided.&#x0D; We propose the implementation in an FPGA of an architecture for CNNs capable of taking advantage of both, sparsity and filter compression.&#x0D; &#x0D;

The COVID-19 pandemic in the world has given rise to a lot of research done in the field of medical imaging using deep learning and artificial intelligence methods for the detection and prognosis of the disease. Capsule Networks (CapsNet) perform image classification by identifying the spatial location and orientation of features within the images. In this paper, a Multi-lane Capsule Neural Networks (MLCN) model is introduced that performs dynamic routing networks with dimensionally distinct parallel lanes, replacing the traditional pooling operations in Convolutional Neural Networks (CNN). With the use of parallel capsules, feature orientation identification at any given part of the image is improved. The MLCN model has been studied in this paper using the X-ray images collected from patients tested for COVID-19 and its performance is evaluated using a number of metrics. It has been observed that the performance of the constructed CapsNet model achieved a testing accuracy of 96.8% with the F-1 score 97.19% performing better than the existing state-of-the-art models.

Graph convolutional neural networks (GCNNs) have emerged as a promising tool in the deep learning community to learn complex hidden relationships of data generated from non-Euclidean domains and represented as graphs. GCNNs are formed by a cascade of layers of graph filters, which replace the classical convolution operation in convolutional neural networks. These graph filters, when operated over real networks, can be subject to random perturbations due to link losses that can be caused by noise, interference and adversarial attacks. In addition, these graph filters are executed by finite-precision processors, which generate numerical quantization errors that may affect their performance. Despite the research works studying the effect of either graph perturbations or quantization in GCNNs, their robustness against both of these problems jointly is still not well investigated and understood. In this paper, we propose a quantized GCNN architecture based on neighborhood graph filters under random graph perturbations. We investigate the stability of such architecture to both random graph perturbations and quantization errors. We prove that the expected error due to quantization and random graph perturbations at the GCNN output is upper-bounded and we show how this bound can be controlled. Numerical experiments are conducted to corroborate our theoretical findings.

Deep convolutional neural networks (CNNs) have shown appealing performance on various computer vision tasks in recent years. This motivates people to deploy CNNs to real-world applications. However, most of state-of-art CNNs require large memory and computational resources, which hinders the deployment on mobile devices. Recent studies show that low-bit weight representation can reduce much storage and memory demand, and also can achieve efficient network inference. To achieve this goal, we propose a novel approach named BWNH to train Binary Weight Networks via Hashing. In this paper, we first reveal the strong connection between inner-product preserving hashing and binary weight networks, and show that training binary weight networks can be intrinsically regarded as a hashing problem. Based on this perspective, we propose an alternating optimization method to learn the hash codes instead of directly learning binary weights. Extensive experiments on CIFAR10, CIFAR100 and ImageNet demonstrate that our proposed BWNH outperforms current state-of-art by a large margin.

Manual detection of eye diseases using retina Optical Coherence Tomography (OCT) images by Ophthalmologists is time consuming, prone to errors and tedious. Previous researchers have developed a computer aided system using deep learning-based convolutional neural networks (CNNs) to aid in faster detection of the retina diseases. However, these methods find it difficult to achieve better classification performance due to noise in the OCT image. Moreover, the pooling operations in CNN reduce resolution of the image that limits the performance of the model. The contributions of the paper are in two folds. Firstly, this paper makes a comprehensive literature review to establish current-state-of-act methods successfully implemented in retina OCT image classifications. Additionally, this paper proposes a capsule network coupled with contrast limited adaptive histogram equalization (CLAHE-CapsNet) for retina OCT image classification. The CLAHE was implemented as layers to minimize the noise in the retina image for better performance of the model. A three-layer convolutional capsule network was designed with carefully chosen hyperparameters. The dataset used for this study was presented by University of California San Diego (UCSD). The dataset consists of 84,495 X-Ray images (JPEG) and 4 categories (NORMAL, CNV, DME, and DRUSEN). The images went through a grading system consisting of multiple layers of trained graders of expertise for verification and correction of image labels. Evaluation experiments were conducted and comparison of results was done with state-of-the-art models to find out the best performing model. The evaluation metrics; accuracy, sensitivity, precision, specificity, and AUC are used to determine the performance of the models. The evaluation results show that the proposed model achieves the best performing model of accuracies of 97.7%, 99.5%, and 99.3% on overall accuracy (OA), overall sensitivity (OS), and overall precision (OP), respectively. The results obtained indicate that the proposed model can be adopted and implemented to help ophthalmologists in detecting retina OCT diseases.

Convolutional neural networks (CNNs) have achieved outstanding performance in many domains. However, the state-of-the-art CNN models also introduce massive computation and huge memory footprint. To facilitate the deployment of CNN on embedded platforms, many existing studies focus on designing dedicated hardware accelerators. But there still exists many legacy DSP-based platforms which can also be exploited to accelerate the inference of CNN. In this work, we study the computation of CNN on MaPU, which is a customized VLIW DSP. MaPU is empowered with a multi-granularity parallel memory system and a flexible program model, which is very suitable for compute-intensive tasks. Through an in-depth analysis of CNN's parallelism and the hardware architecture, we propose a kernel-expanded scheduling scheme, which can handle different kernel size uniformly. Based on our experiment on a face recognition network, MaPU achieves great performance and power efficiency.