Robust learning of parsimonious deep neural networks

As deep learning networks mature and improve classification performance, a significant challenge is their deployment in embedded settings. Modern network typologies, such as convolutional neural networks, can be very deep and impose considerable complexity that is often not feasible in resource bound, real-time systems. Processing of these networks requires high levels of parallelization, maximizing data throughput, and support for different network types, while minimizing power and resource consumption. In response to these requirements, in this paper, we present a low power FPGA based neural network accelerator named SCALENet: a SCalable Low power AccELerator for real-time deep neural Networks. Key features include optimization for power with coarse and fine grain scheduler, implementation flexibility with hardware only or hardware/software co-design, and acceleration for both fully connected and convolutional layers. The experimental results evaluate SCALENet against two different neural network applications: image processing, and biomedical seizure detection. The image processing networks, implemented on SCALENet, trained on the CIFAR-10 and ImageNet datasets with eight different networks, are implemented on an Arty A7 and Zedboard#8482; FPGA platforms. The highest improvement came with the Inception network on an ImageNet dataset with a throughput of 22x and decrease in energy consumption of 13x compared to the ARM processor implementation. We then implement SCALENet for time series EEG seizure detection using both a Direct Convolution and FFT Convolution method to show its design versatility with a 99.7% reduction in execution time and a 97.9% improvement in energy consumption compared to the ARM. Finally, we demonstrate the ability to achieve parity with or exceed the energy efficiency of NVIDIA GPUs when evaluated against Jetson TK1 with embedded GPU System on Chip (SoC) and with a 4x power savings in a power envelope of 2.07 Watts.

Activation functions facilitate deep neural networks by introducing non-linearity to the learning process. The non-linearity feature gives the neural network the ability to learn complex patterns. Recently, the most widely used activation function is the Rectified Linear Unit (ReLU). Though, other various existing activation including hand-designed alternatives to ReLU have been proposed. However, none has succeeded in replacing ReLU due to their existing inconsistencies. In this work, activation function called ReLUMemristor-like Activation Function (RMAF) is proposed to leverage benefits of negative values in neural networks. RMAF introduces a constant parameter (α) and a threshold parameter (p) making the function smooth, non-monotonous, and introduces non-linearity in the network. Our experiments show that, the RMAF works better than ReLU and other activation functions on deeper models and across number of challenging datasets. Firstly, experiments are performed by training and classifying on multi-layer perceptron (MLP) over benchmark data such as the Wisconsin breast cancer, MNIST, Iris and Car evaluation. RMAF achieves high performance of 98.74%, 99.67%, 98.81% and 99.42% respectively, compared to Sigmoid, Tanh and ReLU. Secondly, experiments were performed on convolution neural network (ResNet) over MNIST, CIFAR-10 and CIFAR-100 data and observed the proposed activation function achieves higher performance accuracy of 99.73%, 98.77% and 79.82% respectively than Tanh, ReLU and Swish. Additionally, we experimented our work on deep networks i.e. squeeze network (SqueezeNet), Dense connected neural network (DenseNet121) and ImageNet dataset, which RMAF produced the best performance. We note that, the RMAF converges faster than the other functions and can replace ReLU in any neural network due to the efficiency, scalability and its similarity to both ReLU and Swish.

At present, deep learning is becoming one of the most popular approach to creation of the artificial intelligences systems such as speech recognition, natural language processing, computer vision and so on. The paper presents a historical overview of deep learning in neural networks. The model of the artificial neural network is described as well as the learning algorithms for neural networks including the error backpropagation algorithm, which is used to train deep neural networks. The development of neural networks architectures is presented including neocognitron, autoencoders, convolutional neural networks, restricted Boltzmann machine, deep belief networks, long short-term memory, gated recurrent neural networks, and residual networks. Training deep neural networks with many hidden layers is impeded by the vanishing gradient problem. The paper describes the approaches to solve this problem that provide the ability to train neural networks with more than hundred layers. An overview of popular deep learning libraries is presented. Nowadays, for computer vision tasks convolutional neural networks are utilized, while for sequence processing, including natural language processing, recurrent networks are preferred solution, primarily long short-term memory networks and gated recurrent neural networks.

In recent years, deep neural networks (DNNs) have achieved great successes in many areas, such as cognitive computation, pattern recognition, and computer vision. Although many hand-crafted deep networks have been proposed in the literature, designing a well-behaved neural network for a specific application requires high-level expertise yet. Hence, the automatic architecture design of DNNs has become a challenging and important problem. In this paper, we propose a new reinforcement learning method, whose action policy is to select neural blocks and construct deep networks. We define the action search space with three types of neural blocks, i.e., dense block, residual block, and inception-like block. Additionally, we have also designed several variants for the residual and inception-like blocks. The optimal network is automatically learned by a Q-learning agent, which is iteratively trained to generate well-performed deep networks. To evaluate the proposed method, we have conducted experiments on three datasets, MNIST, SVHN, and CIFAR-10, for image classification applications. Compared with existing hand-crafted and auto-generated neural networks, our auto-designed neural network delivers promising results. Moreover, the proposed reinforcement learning algorithm for deep networks design only runs on one GPU, demonstrating much higher efficiency than most of the previous deep network search approaches.

One of the key aspects of a successful drilling is effective optimization of ROP (Rate of Penetration). Because of the complexity and heterogeneity of formation permeability, the traditional ROP analysis method are limited by drilling prediction. With the accumulation of geological data and drilling records, new methods such as artificial neural network and particle swarm optimization have become powerful tools for obtaining optimization parameters. A ROP optimization method based on deep learning neural network and particle swarm optimization is proposed. Firstly, the prediction model of target wells is established from well logging data by using deep learning neural network. Secondly, the optimized wellbore operation parameters are obtained by using PSO algorithm. At last, the RNN learning algorithm is updated by introducing recovery factor. And also, for the sake of the realization of constraints, a penalty function is introduced. After analyzed logging data of a group of wells in Shunbei area, the experimental results showed that this method can effectively use engineering data to predict drilling rate and optimize drilling parameters.

Machine learning models are vulnerable to adversarial attacks, where a small, invisible, malicious perturbation of the input changes the predicted label. A large area of research is concerned with verification techniques that attempt to decide whether a given model has adversarial inputs close to a given benign input. Here, we show that current approaches to verification have a key vulnerability: we construct a model that is not robust but passes current verifiers. The idea is to insert artificial adversarial perturbations by adding a backdoor to a robust neural network model. In our construction, the adversarial input subspace that triggers the backdoor has a very small volume, and outside this subspace the gradient of the model is identical to that of the clean model. In other words, we seek to create a "needle in a haystack" search problem. For practical purposes, we also require that the adversarial samples be robust to JPEG compression. Large "needle in the haystack" problems are practically impossible to solve with any search algorithm. Formal verifiers can handle this in principle, but they do not scale up to real-world networks at the moment, and achieving this is a challenge because the verification problem is NP-complete. Our construction is based on training a hiding and a revealing network using deep steganography. Using the revealing network, we create a separate backdoor network and integrate it into the target network. We train our deep steganography networks over the CIFAR-10 dataset. We then evaluate our construction using state-of-the-art adversarial attacks and backdoor detectors over the CIFAR-10 and the ImageNet datasets. We made the code and models publicly available at https://github.com/szegedai/hiding-needles-in-a-haystack.

We propose self-adaptive training-a unified training algorithm that dynamically calibrates and enhances training processes by model predictions without incurring an extra computational cost-to advance both supervised and self-supervised learning of deep neural networks. We analyze the training dynamics of deep networks on training data that are corrupted by, e.g., random noise and adversarial examples. Our analysis shows that model predictions are able to magnify useful underlying information in data and this phenomenon occurs broadly even in the absence of any label information, highlighting that model predictions could substantially benefit the training processes: self-adaptive training improves the generalization of deep networks under noise and enhances the self-supervised representation learning. The analysis also sheds light on understanding deep learning, e.g., a potential explanation of the recently-discovered double-descent phenomenon in empirical risk minimization and the collapsing issue of the state-of-the-art self-supervised learning algorithms. Experiments on the CIFAR, STL, and ImageNet datasets verify the effectiveness of our approach in three applications: classification with label noise, selective classification, and linear evaluation. To facilitate future research, the code has been made publicly available at https://github.com/LayneH/self-adaptive-training.

Learning portable neural networks is very essential for computer vision for the purpose that pre-trained heavy deep models can be well applied on edge devices such as mobile phones and micro sensors. Most existing deep neural network compression and speed-up methods are very effective for training compact deep models, when we can directly access the training dataset. However, training data for the given deep network are often unavailable due to some practice problems (\eg privacy, legal issue, and transmission), and the architecture of the given network are also unknown except some interfaces. To this end, we propose a novel framework for training efficient deep neural networks by exploiting generative adversarial networks (GANs). To be specific, the pre-trained teacher networks are regarded as a fixed discriminator and the generator is utilized for derivating training samples which can obtain the maximum response on the discriminator. Then, an efficient network with smaller model size and computational complexity is trained using the generated data and the teacher network, simultaneously. Efficient student networks learned using the proposed Data-Free Learning (DFL) method achieve 92.22% and 74.47% accuracies without any training data on the CIFAR-10 and CIFAR-100 datasets, respectively. Meanwhile, our student network obtains an 80.56% accuracy on the CelebA benchmark.

We study the eigenvalue distribution of random matrices pertinent to the analysis of deep neural networks. The matrices resemble the product of the sample covariance matrices, however, an important difference is that the analog of the population covariance matrix is now a function of random data matrices (synaptic weight matrices in the deep neural network terminology). The problem has been treated in recent work [J. Pennington, S. Schoenholz and S. Ganguli, The emergence of spectral universality in deep networks, Proc. Mach. Learn. Res. 84 (2018) 1924–1932, arXiv:1802.09979] by using the techniques of free probability theory. Since, however, free probability theory deals with population covariance matrices which are independent of the data matrices, its applicability in this case has to be justified. The justification has been given in [L. Pastur, On random matrices arising in deep neural networks: Gaussian case, Pure Appl. Funct. Anal. (2020), in press, arXiv:2001.06188] for Gaussian data matrices with independent entries, a standard analytical model of free probability, by using a version of the techniques of random matrix theory. In this paper, we use another version of the techniques to extend the results of [L. Pastur, On random matrices arising in deep neural networks: Gaussian case, Pure Appl. Funct. Anal. (2020), in press, arXiv:2001.06188] to the case where the entries of the data matrices are just independent identically distributed random variables with zero mean and finite fourth moment. This, in particular, justifies the mean field approximation in the infinite width limit for the deep untrained neural networks and the property of the macroscopic universality of random matrix theory in this case.

Domain-invariant representation learning using an unsupervised domain adversarial adaptation deep neural network

DBMiP: A pre-training method for information propagation over deep networks

Layer-by-layer training is an alternative end-to-end backpropagation approach for training deep convolutional neural networks. Layer-by-layer training on specific architectures can yield very competitive results. In the ImageNet database (www.imagenet. org), layer-by-layer trained networks can perform comparable to many modern end-to-end trained networks. This article compares the performance gap between the two training procedures over a wide range of network architectures and further analyzes the potential limitations of layer-by-layer training. The results show that layer-by-layer learning quickly saturates after a certain critical level due to overfitting of early levels in neural networks. Several approaches that have been used to solve this problem are discussed, and a methodology for improving layer-by-layer learning in various neural network architectures is discussed. Fundamentally, this research highlights the need to open up the black box that represents modern deep neural networks and explore the layer-by-layer interactions between intermediate hidden layers within deep networks through the lens of layer-by-layer learning.

Short-term power load prediction is not only a critical part of power system dispatching but also an essential task for power marketing, grid planning, and other management departments; deep learning is an artificial intelligence method that has gained extensive attention in recent years; this paper selected three models with the most typical signs of deep learning recurrent neural networks and studied their model performance for short-term power load predictions, including power load data preprocessing, feature selection of prediction models, determination of model parameters, and comparison of prediction characteristics between shallow and deep networks, to explore and prove the applicability of deep learning neural networks in short-term power load predictions, ultimately improving the accuracy of short-term load predictions.KeywordsPower loadDeep learningPrediction

The early detection of autism spectrum disorder acts as a risk in the infants and toddlers as per the increase over the early invention awareness. Hence, this paper has made an effort to introduce a new autism detection technique in young children, which poses three major phases called weighted logarithmic transformation, optimal feature selection and classification. Initially, weighted transformation in the input data is carried out that correctly distinguishes the interclass labels, and it is determined to be the specified features. Because of the presence of numerous amounts of features, the ‘prediction’ becomes a serious issue, and therefore the optimal selection of features is important. Here, for optimal feature selection process, a new Levi Flight Cub Update-based lion algorithm (LFCU-LA) is introduced that can be a modification in lion algorithm. Once the optimal features are selected, they are given to the classification process that exploits a hybrid classifier: deep belief network (DBN) and neural network (NN). Additionally, the most important contributions in the hidden neurons of DBN and NN were optimally selected that paves way for exact detection.

<p xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" dir="auto" id="d612827e85"> <b>Objectives:</b> The utility and feasibility of Neural network (NN) and Bayesian in PK-PD analysis were investigated. Recently, Hierarchical Deep Compartment Modeling (HDCM) workflow was shown to be able to quantify uncertainty of NN model predicted parameters and was successfully tested on synthetic data from 10 subjects [ <a class="xref-link" href="#r1">1</a>]. While conceptually appealing, this approach incurs prohibitive run time for typical clinical datasets due to the complexity of the high-dimensional Bayesian posterior distribution. <p xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" dir="auto" id="d612827e93">To enable the full covariate modeling in PK-PD analyses with practical run time, this research utilized flow-based generative deep learning [ <a class="xref-link" href="#r2">2</a>] in addition to NN and Bayesian inference. A novel approach was developed, which trains normalizing flows (NFs) [ <a class="xref-link" href="#r2">2</a>] to generate both samples and their associated probabilities from the posterior distribution. This approach allows breaking down Bayesian inference from a large dataset into manageable subtasks that can be completed sequentially while accumulating all information from the whole dataset. <p xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" dir="auto" id="d612827e101"> <b>Methods:</b> The workflow was implemented in Julia to perform PopPK/PD analysis on simulated dataset for a hypothetical small molecule drug with pharmacodynamic activity following stimulatory response function. The data included 302 subjects from phase 1 and phase 2 studies, with formulation, subject type, weight, gender, age as PK covariates and 3 PD covariates (PDcov1-3). <p xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" dir="auto" id="d612827e106">The covariate-PK/PD parameter relationships were modeled by NN. Loose priors and standard normal distributions were set for fixed, random effect and NN parameters. The initial analysis step processed one sub-dataset (first subtask). The No-U-Turn Sampler (NUTS) [ <a class="xref-link" href="#r3">3</a>] was used to draw samples from the intermediate posterior distribution. A NF [ <a class="xref-link" href="#r2">2</a>] was trained to establish a probability distribution from these samples. The NF-induced distribution became the prior distribution input for the second analysis step, with the likelihood evaluated only for the next sub-dataset. This iterative procedure was repeated until the whole dataset was analyzed. <p xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" dir="auto" id="d612827e114"> <b>Results:</b> A six-step RealNVP[ <a class="xref-link" href="#r4">4</a>] architecture using NN conditioners of 2 hidden layers, width 100, was adequate for carrying posterior information forward across 30 sub-datasets. NUTS produced well-mixing chains (R^ &lt; 1.01). Full covariate analysis testing was completed within 3 days on a 64-core compute node. Posterior predictive checks confirmed that predicted concentrations/PD responses based on the posterior distributions captured observed drug concentrations/responses. Forest plots generated from the NN-predicted covariate effects correctly identified covariate-parameter relationships consistent with NONMEM-based Stepwise Covariate Modeling approach [ <a class="xref-link" href="#r5">5</a>]. <p xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" dir="auto" id="d612827e125"> <b>Conclusions:</b> Bayesian updates enabled by normalizing flows was shown to be a promising direction for improving scalability of NN-based PopPK/PD analyses. The newly developed method adds flexibility to analysis planning, allowing the posterior distribution to be provisionally generated from existing data while waiting for additional clinical studies, from which new information can be readily incorporated.