SCALENet

Field programmable gate array (FPGA) is widely considered as a promising platform for convolutional neural network (CNN) acceleration. However, the large numbers of parameters of CNNs cause heavy computing and memory burdens for FPGA-based CNN implementation. To solve this problem, this paper proposes an optimized compression strategy, and realizes an accelerator based on FPGA for CNNs. Firstly, a reversed-pruning strategy is proposed which reduces the number of parameters of AlexNet by a factor of 13× without accuracy loss on the ImageNet dataset. Peak-pruning is further introduced to achieve better compressibility. Moreover, quantization gives another 4× with negligible loss of accuracy. Secondly, an efficient storage technique, which aims for the reduction of the whole overhead cache of the convolutional layer and the fully connected layer, is presented respectively. Finally, the effectiveness of the proposed strategy is verified by an accelerator implemented on a Xilinx ZCU104 evaluation board. By improving existing pruning techniques and the storage format of sparse data, we significantly reduce the size of AlexNet by 28×, from 243 MB to 8.7 MB. In addition, the overall performance of our accelerator achieves 9.73 fps for the compressed AlexNet. Compared with the central processing unit (CPU) and graphics processing unit (GPU) platforms, our implementation achieves 182.3× and 1.1× improvements in latency and throughput, respectively, on the convolutional (CONV) layers of AlexNet, with an 822.0× and 15.8× improvement for energy efficiency, separately. This novel compression strategy provides a reference for other neural network applications, including CNNs, long short-term memory (LSTM), and recurrent neural networks (RNNs).

Deep convolutional neural networks (DCNNs) have achieved state-of-the-art performance in classification, natural language processing (NLP), and regression tasks. However, there is still a great gap between DCNNs and the human brain in terms of computation efficiency. Inspired by neural synaptic plasticity and stochastic computing (SC), we propose neural synaptic plasticity-inspired computing (NSPC) to simulate the human brain's neural network activity for inference tasks with simple logic gates. The multiplication and accumulation (MAC) is transformed by the wire connectivity in NSPC, which only requires bundles of wires and small width adders. To this end, the NSPC imitates the structure of neural synaptic plasticity from a circuit wires connection perspective. Furthermore, from the principle of NSPC, we use a data mapping method to convert the convolution operations to matrix multiplications. Based on the methodology of NSPC, fully-pipelined and low latency architecture is designed. The proposed NSPC accelerator exhibits high hardware efficiency while maintaining a comparable network accuracy level. The NSPC based DCNN accelerator (NSPC-CNN) processes DCNN at 1.5625M images/s with a power dissipation of 15.42 W and an area of 36.4 mm <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> . The NSPC based deep neural network (DNN) accelerator (NSPC-DNN) that implements three fully connected layers DNN consumes only 6.6 mm2 area and 2.93 W power, and achieves a throughput of 400M images/s. Compared with conventional fixed-point implementations, the NSPC-CNN achieves 2.77× area efficiency, 2.25× power efficiency; the proposed NSPC-DNN exhibits 2.31× area efficiency and 2.09× power efficiency.

This paper investigates the connections between two state of the art classifiers: decision forests (DFs, including decision jungles) and convolutional neural networks (CNNs). Decision forests are computationally efficient thanks to their conditional computation property (computation is confined to only a small region of the tree, the nodes along a single branch). CNNs achieve state of the art accuracy, thanks to their representation learning capabilities. We present a systematic analysis of how to fuse conditional computation with representation learning and achieve a continuum of hybrid models with different ratios of accuracy vs. efficiency. We call this new family of hybrid models conditional networks. Conditional networks can be thought of as: i) decision trees augmented with data transformation operators, or ii) CNNs, with block-diagonal sparse weight matrices, and explicit data routing functions. Experimental validation is performed on the common task of image classification on both the CIFAR and Imagenet datasets. Compared to state of the art CNNs, our hybrid models yield the same accuracy with a fraction of the compute cost and much smaller number of parameters.

The study presents the results of the development and testing of deep learning neural network architectures, which demonstrate high accuracy rates in classifying neurophysiological data, in particular hemodynamic brain activation patterns obtained by functional near-infrared spectroscopy, during solving mathematical problems on spatial-numerical associations. The analyzed signal represents a multidimensional time series of oxyhemoglobin and deoxyhemoglobin dynamics. Taking the specificity of the fNIRS signal into account, a comparative analysis of 2 types of neural network architectures was carried out: (1) architectures based on recurrent neural networks: recurrent neural network with long short-term memory, recurrent neural network with long short-term memory with fully connected layers, bidirectional recurrent neural network with long short-term memory, convolutional recurrent neural network with long short-term memory; (2) architectures based on convolutional neural networks with 1D convolutions: convolutional neural network, fully convolutional neural network, residual neural network. Trained long short-term memory recurrent neural network architectures showed worse results in accuracy in comparison with 1D convolutional neural network architectures. Residual neural network (model_Resnet) demonstrated the highest accuracy rates in three experimental conditions more than 88% in detecting age-related differences in brain activation during spatial-numerical association tasks considering the individual characteristics of the respondents’ signal.

Activation functions facilitate deep neural networks by introducing non-linearity to the learning process. The non-linearity feature gives the neural network the ability to learn complex patterns. Recently, the most widely used activation function is the Rectified Linear Unit (ReLU). Though, other various existing activation including hand-designed alternatives to ReLU have been proposed. However, none has succeeded in replacing ReLU due to their existing inconsistencies. In this work, activation function called ReLUMemristor-like Activation Function (RMAF) is proposed to leverage benefits of negative values in neural networks. RMAF introduces a constant parameter (α) and a threshold parameter (p) making the function smooth, non-monotonous, and introduces non-linearity in the network. Our experiments show that, the RMAF works better than ReLU and other activation functions on deeper models and across number of challenging datasets. Firstly, experiments are performed by training and classifying on multi-layer perceptron (MLP) over benchmark data such as the Wisconsin breast cancer, MNIST, Iris and Car evaluation. RMAF achieves high performance of 98.74%, 99.67%, 98.81% and 99.42% respectively, compared to Sigmoid, Tanh and ReLU. Secondly, experiments were performed on convolution neural network (ResNet) over MNIST, CIFAR-10 and CIFAR-100 data and observed the proposed activation function achieves higher performance accuracy of 99.73%, 98.77% and 79.82% respectively than Tanh, ReLU and Swish. Additionally, we experimented our work on deep networks i.e. squeeze network (SqueezeNet), Dense connected neural network (DenseNet121) and ImageNet dataset, which RMAF produced the best performance. We note that, the RMAF converges faster than the other functions and can replace ReLU in any neural network due to the efficiency, scalability and its similarity to both ReLU and Swish.

paper, we focus on exploring the relationship between weave patterns and their mechanical properties in woven fiber composites through Machine Learning. Specifically, we explore the interactions between woven architectures and in-plane stiffness properties through Deep Convolutional Neural Network (DCNN) and Generative Adversarial Network (GAN). Our research is important for exploring how woven composite’s pattern is related to its mechanical properties and accelerating woven composite design as well as optimization. We focus on two tasks: (1) Stiffness prediction: Predicting in-plane stiffness properties for given weave patterns. Our DCNN extracts high-level features through several convolutional and fully connected layers to determine the final predictions. (2) Weave pattern prediction: Predicting weave patterns for target stiffness properties, which can be treated as the reverse task of the first one. Due to many-to-one mapping between weave patterns and the composite properties, we utilize a Decoder Neural Network as our baseline model and compare its performance with GAN and Genetic Algorithm. We represent the weave patterns as 2D checkerboard models and use finite element analysis (FEA) to determine in-plane stiffness properties, which serve as input data for our ML framework. We show that: (1) for stiffness prediction, DCNN can predict stiffness values for a given weave pattern with relatively high accuracy (above 93%); (2) for weave pattern prediction, the GAN model gives the best prediction accuracy (above 92%) while Decoder Neural Network has the best time efficiency. HAOTIAN FENG

The Sixth International Symposium on Neural Networks (ISNN 2009)

Early detection and treatment are crucial for recovery from cancer. Due to rapid advancement in medical imaging techniques, today doctors can scan various body parts from different perspectives to diagnose cancer or see the progress during a course of treatment. Such techniques are less painful and require less preparation and recovery. The recent evolution of AI, particularly deep learning algorithms have shown potential in computer-aided diagnosis of cancer. These include analysis of cancer images for detection of tumor region, prognosis, skin tissue inspection, etc. Convolutional Neural Network (CNN) based image processing techniques are showing promising results in the diagnosis of breast cancer, skin cancer, prostate cancer, lung cancer, etc. Literature shows many other CNN-based models such as feed-forward neural network, Deep CNN, Multi-Crop CNN, fully convolutional networks (FCNs), Deep Fully Convolutional Network (DFCNet), Recurrent neural networks, etc. The CNN focus solely on the image-specific features and hence require a lesser number of input parameters. A CNN model reduces the size of the input image vector without losing the features critical for making an accurate prediction. This makes it easier to process and reduces computational complexity. CNNs are also not sensitive to the object positioning in an image since the approximate position of image motifs relative to others is known. CNN deals with the variations in images through augmentation and other pre-processing techniques. This makes them robust to various straining conditions in images. This chapter presents an overview of a generic CNN architecture followed by a discussion on major advances in CNN models and their applications for cancer image analysis.

As a popular deep learning technique, convolutional neural network has been widely used in many tasks such as image classification and object recognition. Convolutional neural network exploits spatial correlations in the images by performing convolution operations in local receptive fields. Convolutional neural networks are preferred over fully connected neural networks because they have fewer weights and are easier to train. Many research works have been conducted to reduce the computational complexity and memory requirements of convolutional neural network, to make it applicable to the low-power embedded applications with limited memories. This paper presents the architecture design of convolutional neural network with binary weights and activations, also known as binary neural network, on an FPGA platform. Weights and input activations are binarized with only two values, +1 and -1. This reduces all the fixed point multiplication operations in convolutional layers and fully connected layers to 1-bit XNOR operations. The proposed design uses only on-chip memories. Furthermore, an efficient implementation of batch normalization operation is introduced. When evaluating the CIFAR-10 benchmark, the proposed FPGA design can achieve a processing rate of 332,158 images per second with with accuracy of 86.06% using 1-bit quantized weights and activations.

This paper presents an hardware- and bandwidth-efficient high-performance 2D convolution accelerator for convolutional neural networks (CNNs) in IoT applications. The 2D convolution accelerator is made up of three main components including a dedicated Loader, a Circle Buffer, and a MAC (Multiplier-Accumulator) unit. The 2D convolution accelerator has a 2-stage pipeline structure that reduces latency, increases processing throughput, and reduces power consumption. The architecture proposed in the paper eliminates the reloading of both the weights as well as the input image data. The 2D convolution unit is configurable to support 2D convolution operations with different sizes of input image matrix and kernel filter. The architecture can reduce memory access time and power as well as execution time thanks to the efficient reuse of the preloaded input data, while simplifying hardware implementation. The 2D convolution unit has been simulated and implemented on Xilinx's FPGA platform to evaluate its superiority. Experimental results show that our design is 1.54× and 13.6× faster in performance than the design in [7] and [8], respectively, at lower hardware cost without using any FPGA's dedicated hardware blocks. By reusing preloaded data, our design achieves a bandwidth reduction ratio between 66.4% and 90.5%.

Deep Convolutional Neural Networks (DCNNs) contain a high level of complexity and nonlinearity, so it is not clear based on what features DCNN models make decisions and how they can reach such promising results. There are two types of visualization techniques to interpret and explain the deep models: Backpropagation-based and Perturbation-based algorithms. The most notable drawback of the backpropagation-based visualization is that they cannot be applied for all architectures, whereas Perturbation-based visualizations are totally independent of the architectures. These methods, however, take a lot of computation and memory resources which make them slow and expensive, thereby unsuitable for many real-world applications. To cope with these problems, in this paper, a perturbation-based visualization method called Fast Multi-resolution Occlusion (FMO) are presented which is efficient in terms of time and resource consumption and can be considered in real-world applications. In order to compare the FMO with five well-known Perturbation-based visualizations methods such as Occlusion Test, Super-pixel perturbation (LIME), Randomized Input Sampling (RISE), Meaningful Perturbation and Extremal Perturbation, different experiments are designed in terms of time-consumption, visualization quality and localization accuracy. All methods are applied on 5 well-known DCNNs DenseNet121, InceptionV3, InceptionResnetV2, MobileNet and ResNet50 using common benchmark datasets ImageNet, PASCAL VOC07 and COCO14. According to the experimental results, FMO is averagely 2.32 times faster than LIME on five models DenseNet121, InceptionResnetV2, InceptionV3, MobileNet and ResNet50 with images of ILSVRC2012 dataset as well as 24.84 times faster than Occlusion Test, 11.87 times faster than RISE, 8.72 times faster than Meaningful Perturbation and 10.03 times faster than Extremal Perturbation on all of the five used models with images of common dataset ImageNet without scarifying visualization quality. Moreover, the methods are evaluated in terms of localization accuracy on two hard common datasets of PASCAL VOC07 and COCO14. The results show that FMO outperforms the compared relevant methods in terms of localization accuracy. Also, FMO extends the superimposing process of the Occlusion Test method, which yields a heatmap with more visualization quality than the Occlusion Test on many colorful images.

Brain tumors are formed due to the abnormal growth of the cells that get multiplied and become in uncontrollable perspective. Tumors can damage brain cells by pressing down the skull, which consociate begins in, which negatively affects human health. In advanced stages, a brain tumor is a more dangerous infection that cannot be relieved. Brain tumor detection and early prevention are necessary in today's world. An extreme learning machine (ELM) is a widely adopted algorithm in machine learning. It is proposed to use classification models in brain tumor imaging. This classification is based on the techniques implemented: Convolutional Neural Networks(CNN) and Generative Adversarial Networks(GAN). CNN efficiently solves the convex optimization problem and is faster, requiring less human effort. The algorithmic architecture of GAN uses two neural networks, pocking one against the other. These networks are implemented in various fields to classify brain Tumor images. The present study mainly aims to introduce a new proposed classification system of preschool children brain imaging with Hybrid Convolutional Neural Networks and with the techniques of GAN. The proposed technique is compared with the existing hybrid-CNN and hybrid-GAN techniques. The outcomes are encouraging because the loss is deduced, and the accuracy facet increases. The proposed system achieved a training accuracy of 97.8% and a validation accuracy of 89%. The outcomes of the studies show that ELM in the platform of GAN for preschool children brain imaging classification has achieved higher predictive performance than the traditional classification mechanisms in increasingly complex scenarios. Time elapsed for training brain images samples finds inference value for training samples and time elapsed value increased by 28.9855%. Approximation ratio for cost based on probability, finding Approximation ratio for low probability range is increased by 88.1%. The combination of CNN, GAN and hybrid-CNN, hybrid-GAN, and hybrid CNN + GAN, compared with the proposed hybrid system, increased Detection latency for low range learning rate by 3.31%.

The report presents an algorithm for constructing a convolutional hierarchical neural network classifier, which is a modification of the algorithm for constructing hierarchical neural network classifiers suggested before. The original algorithm was designed to exploit intrinsic class hierarchy to build a class tree with a neural network in each node classifying groups of initial classes (in a non-terminal node) or a subset of original classes (in a terminal node). The convolutional modification utilizes convolutional neural networks instead of regular fully connected networks in order to apply the model to image classification tasks. Use of class hierarchy for image classification should reduce the number of adjusted neural network parameters compared to deep convolutional neural networks, and therefore it should reduce training and inference time. In this context the algorithm may be compared with some pruning techniques. The convolutional hierarchical neural network classifier inherits some hyperparameters of a conventional hierarchical neural network classifier, like the activation threshold and the threshold by the share of voting patterns. The goal of this study was to explore different strategies of choosing these hyperparameters. To test these strategies, we used the CIFAR-10 dataset. Also, for demonstration purposes we apply the convolutional hierarchical neural network classifier to the CIFAR-100 dataset.

In this paper, alternative direction finding methods have been proposed using deep learning techniques. Firstly, Regeression and Classification models have created by using Convolutional Neural Networks (CNNs). In the second Convolutional Neural Networks and Recurrent Neural Networks (RNNs) have been utilized in the proposed methods. Despite having vast amount of direction finding studies, utilization of neural networks is scarce in literature and past works mostly only includes usage of CNNs. In this study, direction finding is performed by learning signals reaching multiple antenna arrays by networks. Created neural networks have been fed with different data formats and their performances against noisy and no-noise data have been shown. In addition, comparative analysis of the developed models were made in the similar Signal-to-Noise Ratio (SNR) range with the subspace based MUSIC algorithm, which is frequently used in direction finding.

Objectives: The paper presents application of convolution neural network and artificial neural network for image classification problem for clothing dataset along with their performance comparison against different optimizers. The major objective of this paper is to perform image classification on fashion-mnist clothing dataset images. Methods: The methods used here are, the traditional ANN and CNN. Here image classification is performed on Fashion-mnist, clothing dataset using CNN and ANN with different optimizers. The performance of the working of ANN and CNN in classifying images from fashion-mnist dataset is compared against different optimizers namely stochastic gradient Descent, Adagrad, RMS prop and Adam optimizer. Findings: The study found that CNN worked better than ANN yielding training accuracy of 95%, 93% and testing accuracy of 91%, 89% when used with Adam and RmsProp respectively. Novelty: The novelty of this work is to present a comparative study of image classification using CNN, ANN using different optimizers, since not many studies or research articles showed the performance comparison of traditional and convolution neural networks in image classification along with different optimizers. Since the real-world scenarios of today require enormous data to be processed, CNN can fit well to diversify applications since they highly reduce the number of parameters to be trained that speeds up the training process. Moreover, to be specific on image classification problems they require the best and most prominent features to be detected and uncovered; this can be achieved using CNN since it has the concept of convolution using filters at its Core. Hence, CNN is highly recommended for such image classification applications than the traditional artificial-neural-networks because of the aforementioned reasons. Keywords: ANN (Artificial NeuralNetworks); CNN (ConvolutionNeural Networks); Optimization; SGD (Stochastic Gradient Descent); RmsProp (Root mean Square propagation); Adam (Adaptive moment estimation); AdaGrad (Adaptive Gradient)