Exploiting Near-Memory Processing Architectures for Bayesian Neural Networks Acceleration

Context. In this second paper in our publication series, we present the open-source ZINGULARITY framework for parameter inference with deep Bayesian artificial neural networks. We carried out supervised learning with synthetic millimeter very long baseline interferometry observations of the Event Horizon Telescope (EHT). Our ground-truth models are based on general relativistic magnetohydrodynamic simulations of Sgr A* and M87* on horizon scales. The models predict the synchrotron emission produced by these accreting supermassive black hole systems. Aims. We investigated how well ZINGULARITY neural networks are able to infer key model parameters from EHT observations, such as the black hole spin and the magnetic state of the accretion disk, when uncertainties in the data are accurately taken into account. Methods. ZINGULARITY makes use of the TENSORFLOW PROBABILITY library and is able to handle large amounts of data with a combination of the efficient TFRecord data format plus the HOROVOD framework for distributed deep learning. Our approach is the first analysis of EHT data with Bayesian neural networks, where an unprecedented training data size, under consideration of a closely modeled EHT signal path, and the full information content of the observational data are used. ZINGULARITY infers parameters based on salient features in the data and is containerized for scientific reproducibility. Results. Through parameter surveys and dedicated validation tests, we identified neural network architectures, that are robust against internal stochastic processes and unaffected by noise in the observational and model data. We give examples of how different data properties affect the network training. We show how the Bayesian nature of our networks gives trustworthy uncertainties and uncovers failure modes for uncharacterizable data. Conclusions. It is easy to achieve low validation errors during training on synthetic data with neural networks, particularly when the forward modeling is too simplified. Through careful studies, we demonstrate that our trained networks can generalize well so that reliable results can be obtained from observational data.

Uncertainty propagation for dropout-based Bayesian neural networks

Stochastic variational inference for Bayesian deep neural networks (DNNs) requires specifying priors and approximate posterior distributions for neural network weights. Specifying meaningful weight priors is a challenging problem, particularly for scaling variational inference to deeper architectures involving high dimensional weight space. Based on empirical Bayes approach, we propose Bayesian MOdel Priors Extracted from Deterministic DNN (MOPED) method to choose meaningful prior distributions over weight space using deterministic weights derived from the pretrained DNNs of equivalent architecture. We empirically evaluate the proposed approach on real-world applications including image classification, video activity recognition and audio classification tasks with varying complex neural network architectures. The proposed method enables scalable variational inference with faster training convergence and provides reliable uncertainty quantification.

The intermittence and fluctuation of renewable energy bring significant uncertainty to the power system, which enormously increases the operational risks of the power system. The development of efficient interval prediction models can provide data support for decision making and help improve the economy and reliability of energy interconnection operation. The performance of Bayesian deep learning models and Bayesian shallow neural networks in short-term interval prediction of photovoltaic power is compared in this study. Specifically, an LSTM Approximate Bayesian Neural Network model (ABNN-I) is built on the basis of the deep learning and Monte Carlo Dropout method. Meanwhile, a Feedforward Bayesian Neural Network (ABNN-II) model is introduced by Feedforward Neural Network and the Markov Chain Monte Carlo method. To better compare and verify the interval prediction capability of the ABNN models, a novel clustering method with three-dimensional features which include the number of peaks and valleys, the average power value, and the non-stationary measurement coefficient is proposed for generating sunny and non-sunny clustering sets, respectively. Results show that the ABNN-I model has an excellent performance in the field of photovoltaic short-term interval forecasting. At a 95% confidence level, the interval coverage from ABNN-I to ABNN-II can be increased by up to 3.1% and the average width of the interval can be reduced by 56%. Therefore, with the help of the high computational capacity of deep learning and the inherent ability to quantify uncertainty of the interval forecast from Bayesian methods, this research provides high-quality interval prediction results for photovoltaic power prediction and solves the problem of difficult modeling for over-fitting that exists in the training process, especially on the non-sunny clustering sets.

Due to the advent of modern embedded systems and mobile devices with constrained resources, there is a great demand for incredibly efficient deep neural networks for machine learning purposes. There is also a growing concern of privacy and confidentiality of user data within the general public when their data is processed and stored in an external server which has further fueled the need for developing such efficient neural networks for real-time inference on local embedded systems. The scope of our work presented in this paper is limited to image classification using a convolutional neural network. A Convolutional Neural Network (CNN) is a class of Deep Neural Network (DNN) widely used in the analysis of visual images captured by an image sensor, designed to extract information and convert it into meaningful representations for real-time inference of the input data. In this paper, we propose a neoteric variant of deep convolutional neural network architecture to ameliorate the performance of existing CNN architectures for real-time inference on embedded systems. We show that this architecture, dubbed CondenseNeXt, is remarkably efficient in comparison to the baseline neural network architecture, CondenseNet, by reducing trainable parameters and FLOPs required to train the network whilst maintaining a balance between the trained model size of less than 3.0 MB and accuracy trade-off resulting in an unprecedented computational efficiency.

On the relative expressiveness of Bayesian and neural networks

Graph Neural Networks (GNNs) is an important branch of deep learning in graph structure. As a model that can reveal deep topological information, GNNs has been widely used in various learning tasks, including physical system, protein interface prediction, disease classification, learning molecular fingerprints and so on. However, in most of these tasks, the graph data we are working with may be noisy and may contain spurious edges. That is to say, there is a lot of uncertainty associated with the underlying graph structure. The method of modeling uncertainty is to use Bayesian framework, in which graph is regarded as random variable. Introducing Bayesian framework into graph-based model, especially for semi-supervised node classification, has been shown that it can produce higher classification accuracy. In this paper, some GNNs models and Bayesian neural networks are introduced to better understand how GNNs are combined with Bayesian. Then, the development of Bayesian graph neural networks(BGNNs) in recent years is summarized, and its application in the field of engineering technology is demonstrated. Finally, the future development of BGNNs is prospected and the full text is summarized.

Deep neural network (DNN) models have achieved state-of-the-art predictive accuracy in a wide range of applications. However, it remains a challenging task to accurately quantify the uncertainty in DNN predictions, especially those of continuous outcomes. To this end, we propose the Bayesian deep noise neural network (B-DeepNoise), which generalizes standard Bayesian DNNs by extending the random noise variable from the output layer to all hidden layers. Our model is capable of approximating highly complex predictive density functions and fully learn the possible random variation in the outcome variables. For posterior computation, we provide a closed-form Gibbs sampling algorithm that circumvents tuning-intensive Metropolis-Hastings methods. We establish a recursive representation of the predictive density and perform theoretical analysis on the predictive variance. Through extensive experiments, we demonstrate the superiority of B-DeepNoise over existing methods in terms of density estimation and uncertainty quantification accuracy. A neuroimaging application is included to show our model's usefulness in scientific studies.

The Ph.D. dissertation focuses on mixed-signal-based SRAM computation-in-memory (CIM) for artificial intelligence (AI) edge devices. CIM is an emerging approach that integrates computation capabilities into memory units, reducing data movement and enhancing performance for memory-intensive tasks like deep neural networks (DNNs). CIM mainly deals with multiply-and-accumulate (MAC) operations which consist most of the computing burden in DNN. Additionally, mixed-signal-based CIM has many advantages over digital-based CIM in terms of area, throughput, and power in low-precision DNN (1-4b) as able to accumulate multiplication results at once with analog domain (voltage, current and charge, etc.) Therefore, mixed-signal-based CIM is an attractive option as low-precision DNNs have achieved competitive precision so far.

Automated image classification is an essential task of the computer vision field. The tagging of images into a set of predefined groups is referred to as image classification. The implementation of computer vision to automate image classification would be beneficial because manual image evaluation and identification can be time-consuming, particularly when there are many images of different classes. Deep learning approaches are proven to overperform existing machine learning techniques in many fields in recent years, and computer vision is one of the most notable examples. The very deep neural network (VDCNN) is a powerful deep learning model for image classification, and this paper examines it briefly using MNIST handwritten digit dataset. This dataset is used to prove the efficacy of very deep neural networks over other deep learning models. The proposed study aims to comprehend the very deep neural network architecture used to accomplish a handwritten digit recognition task. The feasibility of the proposed model is evaluated using mean accuracy, validation accuracy, and standard deviation. The study results of the very deep neural network model are compared to a convolutional neural network and convolutional neural network with batch normalization. According to the results of the comparison study, very deep neural networks achieve high accuracy of 99.1% for a handwritten dataset. The outcome of the proposed work is used to interpret how well a very deep neural network performs when compared to the other two models of deep neural networks. This proposed architecture may be used to automate the classification of handwritten digits dataset.

Summary The interpretation of faults within a geological basin or reservoir from seismic data is a time-consuming, and often manual task associated with high uncertainties. Recently, numerous approaches using machine learning, especially various types of convolutional neural networks, have been presented to automate the process of identifying fault planes within seismic images, which have been shown to outperform traditional fault detection techniques. While these proposed methods show good performance, many of these approaches do not allow investigation of the associated uncertainties that arise in the fault identification process. In this study, we present an application of Bayesian deep convolutional neural networks for identifying faults within seismic datasets. Using an approximate Bayesian inference method a Bayesian deep neural network was trained on a large dataset of synthetic faulted seismic images. The model is then applied to a benchmark dataset and a real data case from NW shelf Australia to identify fault planes, and to investigate the associated uncertainty in the predictive distribution.

Practical Mechanisms for Reducing Processor–Memory Data Movement in Modern Workloads

This dissertation addresses two significant issues in the effective application of machine learning algorithms and models for the physical and engineering sciences. The first is the broad challenge of automated modeling of data across different processes in a physical system. The second is the dilemma of obtaining insightful interpretations on the relationships between the inputs and outcome of a system as inferred from complex, black box machine learning models.Automated Geometric Shape Deviation Modeling for Additive Manufacturing SystemsAdditive manufacturing systems possess an intrinsic capability for one-of-a-kind manufacturing of a vast variety of shapes across a wide spectrum of processes. One major issue in AM systems is geometric accuracy control for the inevitable shape deviations that arise in AM processes. Current effective approaches for shape deviation control in AM involve the specification of statistical or machine learning deviation models for additively manufactured products. However, this task is challenging due to the constraints on the number of test shapes that can be manufactured in practice, and limitations on user efforts that can be devoted for learning deviation models across different shape classes and processes in an AM system. We develop an automated, Bayesian neural network methodology for comprehensive shape deviation modeling in an AM system. A fundamental innovation in this machine learning method is our new and connectable neural network structures that facilitate the transfer of prior knowledge and models on deviations across different shape classes and AM processes. Several case studies on in-plane and out-of-plane deviations, regular and free-form shapes, and different settings of lurking variables serve to validate the power and broad scope of our methodology, and its potential to advance high-quality manufacturing in an AM system.Interpretable Machine LearningMachine learning algorithms and models constitute the dominant set of predictive methods for a wide range of complex, real-world processes. However, interpreting what such methods effectively infer from data is difficult in general. This is because their typical black box natures possess a limited ability to directly yield insights on the underlying relationships between inputs and the outcome for a process. We develop methodologies based on new predictive comparison estimands that effectively enable one to ``mine’’ machine learning models, in the sense of (a) interpreting their inferred associations between inputs and/or functional forms of inputs with the outcome, (b) identifying the inputs that they effectively consider relevant, and (c) interpreting the inferred conditional and two-way associations of the inputs with the outcome. We establish Fisher consistent estimators, and their corresponding standard errors, for our new estimands under a condition on the inputs' distributions. The significance of our predictive comparison methodology is demonstrated with a wide range of simulation and case studies that involve Bayesian additive regression trees, neural networks, and support vector machines. Our extended study of interpretable machine learning for AM systems demonstrates how our method can contribute to smarter advanced manufacturing systems, especially as current machine learning methods for AM are lacking in their ability to yield meaningful engineering knowledge on AM processes.

Deep Neural Networks (DNNs) have been successfully applied to various real-world machine learning applications. However, performing large DNN inference tasks in real-time remains a challenge due to its substantial computational costs. Recently, Spiking Neural Networks (SNNs) have emerged as an alternative way of processing DNN'fs task. Due to its eventbased, data-driven computation, SNN reduces both inference latency and complexity. With efficient conversion methods from traditional DNN, SNN exhibits similar accuracy, while leveraging many state-of-the-art network models and training methods. In this work, an efficient neuromorphic hardware architecture for image recognition task is presented. To preserve accuracy, the analog-to-spiking conversion algorithm is adopted. The system aims to minimize hardware area cost and power consumption, enabling neuromorphic hardware processing in edge devices. Simulation results have shown that, with the MNIST digit recognition task, the system has achieved $\times 20$ reduction in terms of core area cost compared to the state-of-the-art works, with an accuracy of 94.4%, core area of 15 $\mu m^{2}$ at a maximum frequency of 250 MHz.

We investigate different strategies for active learning with Bayesian deep neural networks. We focus our analysis on scenarios where new, unlabeled data is obtained episodically, such as commonly encountered in mobile robotics applications. An evaluation of different strategies for acquisition, updating, and final training on the CIFAR-10 dataset shows that incremental network updates with final training on the accumulated acquisition set are essential for best performance, while limiting the amount of required human labeling labor.