NeuroFabric: Hardware and ML Model Co-Design for A Priori Sparse Neural Network Training

Usage of wireless gadgets is increasing day by day in both metropolitan and countryside localities. The network infrastructure does not reach all the localities; especially in rural areas, where the internet availability is inadequate to meet the network requirements. Devices in such rural locations will be out of the network for some period. This non-persistent connectivity problem can be solved by using the moving nodes as a carrier for data transmission. In this work, we classify the vehicular network into sparse, semi-sparse and dense network based on the number of static and moving nodes present in a location. In sparse and semi-sparse network, there won't be any continuous path between the source and destination nodes. In such networks, we can use Delay Tolerant Network (DTN) protocols for data transmission. In our work, we try to find the suitable DTN protocol for the sparse and semi-sparse network topology. The protocols considered in this work are Epidemic, Spray and Wait, and Maxprop and the parameters we analyzed are delivery ratio, average latency and overhead ratio. We simulated these protocols in a vehicular network with heterogeneous characteristics, for the dense, semi-sparse and sparse network topology. The routing protocols performance is analyzed by delivery ratio, average latency and overhead ratio using Opportunistic Network Environment (ONE) simulator for varying the message lifetime and node density.

The usage of Deep Neural Networks (DNN) on resource-constrained edge devices has been limited due to their high computation and large memory requirement. In this work, we propose an algorithm to compress DNNs by jointly optimizing structured sparsity and quantization constraints in a single DNN training framework. The proposed algorithm has been extensively validated on high/low capacity DNNs and wide/deep sparse DNNs. Further, we perform Pareto-optimal analysis to extract optimal DNN models from a large set of trained DNN models. The optimal structurally-compressed DNN model achieves ~50X weight memory reduction without test accuracy degradation, compared to floating-point uncompressed DNN.

The ever-increasing complexity of both Deep Neural Networks (DNN) and hardware accelerators has made the co-optimization of these domains extremely complex. Previous works typically focus on optimizing DNNs given a fixed hardware configuration or optimizing a specific hardware architecture given a fixed DNN model. Recently, the importance of the joint exploration of the two spaces drew more and more attention. Our work targets the co-optimization of DNN and hardware configurations on edge GPU accelerators. We propose an evolutionary-based co-optimization strategy by considering three metrics: DNN accuracy, execution latency, and power consumption. By combining the two search spaces, a larger number of configurations can be explored in a short time interval. In addition, a better tradeoff between DNN accuracy and hardware efficiency can be obtained. Experimental results show that the co-optimization outperforms the optimization of DNN for fixed hardware configuration with up to 53% hardware efficiency gains with the same accuracy and inference time.

The transition from analog to digital television has availed new spectrum called white space, which can be used to boost the capacity of wireless networks on an opportunistic basis. One sector in which there is a need to use white space frequencies is the healthcare sector because of existent protocols which are using it and the white space frequency is not as crowded as Wi-Fi. However, design simulations of wireless communication networks in white space frequencies have revealed dense network topology because of better signal propagation and penetration properties of white space frequencies. Consequently, communication networks designed in white space frequencies will require topology reduction for better communication and routing. Therefore, this paper proposes a link-based topology reduction algorithm to reduce a dense mesh network topology designed in white space frequencies into a sparse mesh network topology. The paper also proposes a network optimization function to introduce a hierarchical backbone-based network topology from the sparse network topology for better scalability. Performance evaluation on the proposed designs show that the designs can guide network engineers to select the most relevant performance metrics during a network feasibility study in white space frequencies, aimed at guiding the implementation process.

This paper presents GPU performance optimization and scaling results for inference models of the Sparse Deep Neural Network Challenge 2020. Demands for network quality have increased rapidly, pushing the size and thus the memory requirements of many neural networks beyond the capacity of available accelerators. Sparse deep neural networks (SpDNN) have shown promise for reining in the memory footprint of large neural networks. However, there is room for improvement in implementing SpDNN operations on GPUs. This work presents optimized sparse matrix multiplication kernels fused with the ReLU function. The optimized kernels reuse input feature maps from the shared memory and sparse weights from registers. For multi-GPU parallelism, our SpDNN implementation duplicates weights and statically partition the feature maps across GPUs. Results for the challenge benchmarks show that the proposed kernel design and multi-GPU parallelization achieve up to 180 tera-edges per second inference throughput. These results are up to 4.3x faster for a single GPU and an order of magnitude faster at full scale than those of the champion of the 2019 Sparse Deep Neural Network Graph Challenge for the same generation of NVIDIA V100 GPUs. Using the same implementation, we also show single-GPU throughput on NVIDIA A100 is 2.37$\times$ faster than V100.

Deep neural network is widely used in many intelligent computing scenarios, but the storage overhead caused by its huge number of parameters limits its application in embedded systems. The neural network pruning technology can remove a large number of redundant parameters in the network, theoretically, which reduces the amount of parameter storage and calculation. However, there are two problems in deploying sparse neural network algorithms on existing embedded hardware platforms: one is that using the existing coding to store sparse neural network parameters will increase additional memory access overhead; the other is the decoding process of sparse model is highly complex, which increases the amount of calculation. To solve the two problems, this paper proposes Dynamic ELL encoding format to represent the sparse weight matrix for pruning neural network. This new coding combines the advantages of the ELL encoding and relative indexing, which compromises the storage overhead and decoding overhead. The sparse MLP that uses dynamic ELL encoding storage is deployed on a SoC built on a RISC-V processor. Experiments show that compared to dense networks that use Dynamic ELL encoding storage, the sparse network storage space is reduced by 43%, and the system running time is reduced by 37%.

In this paper, we use graphics processing units (GPU) to accelerate sparse and arbitrary structured neural networks. Sparse networks have nodes in the network that are not fully connected with nodes in preceding and following layers, and arbitrary structure neural networks have different number of nodes in each layers. Sparse Neural networks with arbitrary structures are generally created in the processes like neural network pruning and evolutionary machine learning strategies. We show that we can gain significant speedup for full activation of such neural networks using graphical processing units. We do a prepossessing step to determine dependency groups for all the nodes in a network, and use that information to guide the progression of activation in the neural network. Then we compute activation for each nodes in its own separate thread in the GPU, which allows for massive parallelization. We use CUDA framework to implement our approach and compare the results of sequential and GPU implementations. Our results show that the activation of sparse neural networks lends very well to GPU acceleration and can help speed up machine learning strategies which generate such networks or other processes that have similar structure.

Deep Neural Networks (DNNs) have made significant improvements to reach the desired accuracy to be employed in a wide variety of Machine Learning (ML) applications. Recently the Google Brain's team demonstrated the ability of Capsule Networks (CapsNets) to encode and learn spatial correlations between different input features, thereby obtaining superior learning capabilities compared to traditional (i.e., non-capsule based) DNNs. However, designing CapsNets using conventional methods is a tedious job and incurs significant training effort. Recent studies have shown that powerful methods to automatically select the best/optimal DNN model configuration for a given set of applications and a training dataset are based on the Neural Architecture Search (NAS) algorithms. Moreover, due to their extreme computational and memory requirements, DNNs are employed using the specialized hardware accelerators in IoT-Edge/CPS devices. In this paper, we propose NASCaps, an automated framework for the hardware-aware NAS of different types of DNNs, covering both traditional convolutional DNNs and CapsNets. We study the efficacy of deploying a multi-objective Genetic Algorithm (e.g., based on the NSGA-II algorithm). The proposed framework can jointly optimize the network accuracy and the corresponding hardware efficiency, expressed in terms of energy, memory, and latency of a given hardware accelerator executing the DNN inference. Besides supporting the traditional DNN layers, our framework is the first to model and supports the specialized capsule layers and dynamic routing in the NAS-flow. We evaluate our framework on different datasets, generating different network configurations, and demonstrate the tradeoffs between the different output metrics. We will open-source the complete framework and configurations of the Pareto-optimal architectures at https://github.com/ehw-fit/nascaps.

In principle, sparse neural networks should be significantly more efficient than traditional dense networks. Neurons in the brain exhibit two types of sparsity; they are sparsely interconnected and sparsely active. These two types of sparsity, called weight sparsity and activation sparsity, when combined, offer the potential to reduce the computational cost of neural networks by two orders of magnitude. Despite this potential, today’s neural networks deliver only modest performance benefits using just weight sparsity, because traditional computing hardware cannot efficiently process sparse networks. In this article we introduce Complementary Sparsity, a novel technique that significantly improves the performance of dual sparse networks on existing hardware. We demonstrate that we can achieve high performance running weight-sparse networks, and we can multiply those speedups by incorporating activation sparsity. Using Complementary Sparsity, we show up to 100× improvement in throughput and energy efficiency performing inference on FPGAs. We analyze scalability and resource tradeoffs for a variety of kernels typical of commercial convolutional networks such as ResNet-50 and MobileNetV2. Our results with Complementary Sparsity suggest that weight plus activation sparsity can be a potent combination for efficiently scaling future AI models.

Neurons linked to spatial navigation and toroidal dynamics in the mouse medial entorhinal cortex (MEC) show unexpected minute‐scale (< 0.01 Hz) oscillatory sequences without neural organization or clear relation to behavior. However, the conditions sustaining these’ ultraslow' equences remain uncertain. Since dopaminergic modulation of spike‐timing‐dependent plasticity (STDP) enables infraslow (< 0.1 Hz) oscillations in sparse neural networks (SNN), we hypothesize that SNN might sustain minute‐scale (ultraslow) oscillatory sequences when bypassing the modulation. Using computational simulations through detailed numerical investigations, the conditions that enable the MEC‐like ultraslow rhythms are characterized in an Izhikevich's SNN with dopaminergic STDP modulation. To induce the ultraslow sequences, a few active neurons are defined at each simulation step following a toroid‐like trajectory. The results indicate that even when disrupting the dopamine‐based STDP learning, the ultraslow oscillations require a second‐scale resetting of the membrane potential to keep the sequential firing. Interestingly, separate oscillations on synapses that do not contribute to the firing rate at a specific time step (silent synaptic connections) are observed in the presence of the sequences. Since the mechanisms underlying the experimental finding are unknown, the present manuscript generates hypotheses on the conditions that sustain minute‐scale sequences, which will be relevant for the community studying population dynamics in the MEC.

3D reconstruction from videos has become increasingly popular for various applications, including navigation for autonomous driving of robots and drones, augmented reality (AR), and 3D modeling. This task often combines traditional image/video processing algorithms and deep neural networks (DNNs). Although recent developments in deep learning have improved the accuracy of the task, the large number of calculations involved results in low computation speed and high power consumption. Although there are various domain-specific hardware accelerators for DNNs, it is not easy to accelerate the entire process of applications that alternate between traditional image/video processing algorithms and DNNs. Thus, FPGA-based end-to-end acceleration is required for such complicated applications in low-power embedded environments. This paper proposes a novel FPGA-based accelerator for DeepVideoMVS, a DNN-based depth estimation method for 3D reconstruction. We employ HW/SW co-design to appropriately utilize heterogeneous components in modern SoC FPGAs, such as programmable logic (PL) and CPU, according to the inherent characteristics of the method. As some operations are unsuitable for hardware implementation, we determine the operations to be implemented in software through analyzing the number of times each operation is performed and its memory access pattern, and then considering comprehensive aspects: the ease of hardware implementation and degree of expected acceleration by hardware. The hardware and software implementations are executed in parallel on the PL and CPU to hide their execution latencies. The proposed accelerator was developed on a Xilinx ZCU104 board by using NNgen, an open-source high-level synthesis (HLS) tool. Experiments showed that the proposed accelerator operates 60.2 times faster than the software-only implementation on the same FPGA board with minimal accuracy degradation.

Advanced Artificial Intelligence (AI) systems that process vast feature sets can be designed using Deep Neural Networks (DNN). The popularity and prevalence of DNN based tasks are mainly due to their ability to achieve state of the art accuracy. However, DNN based systems are computationally intensive with heavy on-chip and off-chip data movement demands. Efficient hardware accelerators with systematized interconnection platform support handling this insistent computation and fast communication requirements. With rise in accelerator complexity, it is important to identify and solve communication overheads imposed due to heavy data movement in the system. Need for increased data transfers come as a consequence of complex computations and structure of data storage hierarchy in different DNN models. Multi-core system with underlying Network on Chip (NoC) framework is a promising design choice for implementing DNN based systems with diverse layers. Data movement plays a vital role in deciding performance of such systems. Analysing metrics that affect latency and energy/power efficiency of system is thus crucial in deciding performance enhancement of these DNN inference accelerators. This paper quantifies the communication overheads associated with accelerating different DNN models which use NoC as its basic communication platform.

Deep Neural Networks (DNNs) have been successfully applied to various real-world machine learning applications. However, performing large DNN inference tasks in real-time remains a challenge due to its substantial computational costs. Recently, Spiking Neural Networks (SNNs) have emerged as an alternative way of processing DNN'fs task. Due to its eventbased, data-driven computation, SNN reduces both inference latency and complexity. With efficient conversion methods from traditional DNN, SNN exhibits similar accuracy, while leveraging many state-of-the-art network models and training methods. In this work, an efficient neuromorphic hardware architecture for image recognition task is presented. To preserve accuracy, the analog-to-spiking conversion algorithm is adopted. The system aims to minimize hardware area cost and power consumption, enabling neuromorphic hardware processing in edge devices. Simulation results have shown that, with the MNIST digit recognition task, the system has achieved $\times 20$ reduction in terms of core area cost compared to the state-of-the-art works, with an accuracy of 94.4%, core area of 15 $\mu m^{2}$ at a maximum frequency of 250 MHz.

The multi-session bridge

In delay tolerant networks (DTNs), nodes have intermittent connectivity patterns due to various factors such as mobility, sparse network topology, and unpredictable movement patterns. In such networks, nodes store messages, carry them along as they move and forward them opportunistically whenever an encounter occurs. In many DTN applications, such as in disaster situations and remote regions with no communications infrastructure, we can envisage a network topology formed by the mobility and contact pattern of people carrying their mobile devices. The social behavior of people (nodes) in Spatio-temporal dimensions governs the formation of such a topology. To improve data dissemination in such networks, several works have proposed to utilize social-aware metrics for the selection of relays. However, existing routing protocols for DTNs may perform poorly in the scenarios in which nodes are localized in multiple small regions, and the topology is sparse such as in co-located villages in remote regions with no communication infrastructure. This work proposes a Cooperative mobile throwbox-based routing protocol (CMTR) for social-aware DTNs. In CMTR, static and mobile ThrowBoxes (TBs) are deployed to improve the efficiency of a DTN. The static and mobile TBs cooperatively relay data to enhance network efficiency. Performance evaluation via simulations in ONE simulator shows that CMTR improves the data delivery ratio, delays, and energy efficiency as compared to Epidemic routing and Bubble Rap routing schemes.