A simulation environment for very large neural networks

Describes a scalable parallel architecture for real-time, large scale neural network simulations. Currently the SIMD architecture is largely adopted for implementing digital neurocomputers. However, it is not efficient for simulating large scale neural networks in real-time because of its limited scalability and flexibility. The authors investigate, as a solution, a wavefront array processing (WAP) architecture based on asynchronous communications. The authors compare both architectures in scalability, performance, and flexibility for simulating multi-layer perceptrons. The authors also briefly discuss implementing a high performance digital neurocomputer based on the WAP.

In this perspective paper we explore the question how audible smiling can be integrated in speech synthesis applications. In human-human communication, smiling can serve various functions, such as signaling politeness or as a marker of trustworthiness and other aspects that raise and maintain the social likeability of a speaker. However, in human-machine communication, audible smiling is nearly unexplored, but could be an advantage in different applications such as dialog systems. The rather limited knowledge of the details of audible smiling and their exploitation for speech synthesis applications is a great challenge. This is also true for modeling smiling in spoken dialogs and testing it with users. Thus, this paper argues to fill the research gaps in identifying factors that constitute and affect audible smiling in order to incorporate it in speech synthesis applications. The major claim is to focus on the dynamics of audible smiling on various levels.

It is well-known that large neural networks with many unshared weights can be very difficult to train. A neural network ensemble consisting of a number of individual neural networks usually performs better than a complex monolithic neural network. One of the motivations behind neural network ensembles is the divide-and-conquer strategy, where a complex problem is decomposed into different components each of which is tackled by an individual neural network. A promising algorithm for training neural network ensembles is the negative correlation learning algorithm which penalizes positive correlations among individual networks by introducing a penalty term in the error function. A penalty coefficient is used to balance the minimization of the error and the minimization of the correlation. It is often very difficult to select an optimal penalty coefficient for a given problem because as yet there is no systematic method available for setting the parameter. This paper first applies negative correlation learning to the traffic flow prediction problem, and then proposes an evolutionary approach to deciding the penalty coefficient automatically in negative correlation learning. Experimental results on the traffic flow prediction problem will be presented.

Event Abstract Back to Event Quantitative Anatomy for assembling large scale neural networks Marcel Oberlaender1* 1 Max Planck Florida Institute, United States Sensory deprivation, as well as neurodegenerative diseases, such as Alzheimer’s, cause substantial changes in brain function and anatomy, which ultimately result in behavioral deficits. Therefore, over the last 5 years, Marcel Oberlaender and his colleagues developed methods to image and quantify 3D neuron and neuronal network anatomy. These methods allow determining the number and three-dimensional distribution of all neurons in large volumes of brain tissue, the tracing of all processes from individual neurons, their classification and interconnection to realistic neural networks (see Figure). Illustration of the “Networks in silico project”. High resolution 3D image stacks of the entire brain lay the foundation to quantify the structure and 3D distribution of all neurons within functional neuronal networks. Here, the whisker-related thalamocortical pathway in rats is reconstructed. So far these methods were limited to certain brain regions such as the somatosensory cortex or thalamus. However, recent developments in imaging techniques and computing power will allow in principle the application of these methods to the entire mouse or rat brain. The department of “Digital Neuroanatomy” at the newly founded “Max Planck Florida Institute for Integrative Biology and Neurosciences” therefore aims to determine the total number and three-dimensional distribution of all neurons in brains of “normal” mice. The resultant “cellular atlas” of the mouse brain will function as an unbiased reference for anatomical changes at cellular level caused by sensory deprivation or disease. Conference: Neuroinformatics 2010 , Kobe, Japan, 30 Aug - 1 Sep, 2010. Presentation Type: Oral Presentation Topic: Workshop 2: Synaptoprojectomes: assembling, using and sharing dense cellular micromaps of brains Citation: Oberlaender M (2010). Quantitative Anatomy for assembling large scale neural networks. Front. Neurosci. Conference Abstract: Neuroinformatics 2010 . doi: 10.3389/conf.fnins.2010.13.00005 Copyright: The abstracts in this collection have not been subject to any Frontiers peer review or checks, and are not endorsed by Frontiers. They are made available through the Frontiers publishing platform as a service to conference organizers and presenters. The copyright in the individual abstracts is owned by the author of each abstract or his/her employer unless otherwise stated. Each abstract, as well as the collection of abstracts, are published under a Creative Commons CC-BY 4.0 (attribution) licence (https://creativecommons.org/licenses/by/4.0/) and may thus be reproduced, translated, adapted and be the subject of derivative works provided the authors and Frontiers are attributed. For Frontiers’ terms and conditions please see https://www.frontiersin.org/legal/terms-and-conditions. Received: 08 Jun 2010; Published Online: 08 Jun 2010. * Correspondence: Marcel Oberlaender, Max Planck Florida Institute, Jupiter, United States, oberlaender@neuro.mpg.de Login Required This action requires you to be registered with Frontiers and logged in. To register or login click here. Abstract Info Abstract The Authors in Frontiers Marcel Oberlaender Google Marcel Oberlaender Google Scholar Marcel Oberlaender PubMed Marcel Oberlaender Related Article in Frontiers Google Scholar PubMed Abstract Close Back to top Javascript is disabled. Please enable Javascript in your browser settings in order to see all the content on this page.

Graph neural networks (GNNs) emerge as a powerful approach to process non-euclidean data structures and have been proved powerful in various application domains such as social networks and e-commerce. While such graph data maintained in real-world systems can be extremely large and sparse, thus employing GNNs to deal with them requires substantial computational and memory overhead, which induces considerable energy and resource cost on CPUs and GPUs. In this article, we present a specialized accelerator architecture, EnGN, to enable high-throughput and energy-efficient processing of large-scale GNNs. The proposed EnGN is designed to accelerate the three key stages of GNN propagation, which is abstracted as common computing patterns shared by typical GNNs. To support the key stages simultaneously, we propose the ring-edge-reduce(RER) dataflow that tames the poor locality of sparsely-and-randomly connected vertices, and the RER PE-array to practice RER dataflow. In addition, we utilize a graph tiling strategy to fit large graphs into EnGN and make good use of the hierarchical on-chip buffers through adaptive computation reordering and tile scheduling. Overall, EnGN achieves performance speedup by 1802.9X, 19.75X, and 2.97X and energy efficiency by 1326.35X, 304.43X, and 6.2X on average compared to CPU, GPU, and a state-of-the-art GCN accelerator HyGCN, respectively.

RF/IR wireless (virtual) synapses are critical components of HYDRA (Hyper-Distributed Robotic Autonomy) neural networks, already discussed in two earlier papers. The HYDRA network has the potential to be very large, up to 10 11 -neurons and 10 18 -synapses, based on already established technologies (cellular RF telephony and IR-wireless LANs). It is organized into almost fully connected IR-wireless clusters. The HYDRA neurons and synapses are very flexible, simple, and low-cost. They can be modified into a broad variety of biologically-inspired brain-like computing capabilities. In this third paper, we focus on neural hardware in general, and on IR-wireless synapses in particular. Such synapses, based on LED/LD-connections, dominate the HYDRA neural cluster. Keywords: bio-inspired, neural, evolutionary, brain, infrared, synapses, 1. INTRODUCTION – BACKGROUND OF HYDRA CONCEPT The new concept of HYDRA (Hyper-Distributed Robotic Autonomy) , a very large neural network, has been presented in two earlier papers

Speech analysis techniques open new perspectives in the processing of dialectal oral data. Speech synthesis can be useful to create or recreate voices of speakers for extinct languages, to re-edit dialectal material using new technologies or to reconstruct utterances of informants that only were registered in notebooks. Speech recognition, applied to sound dialectal sequences, can make easier automatic transcription of oral texts. In this paper the possibilities of speech analysis techniques in their application to the dialectal studies is described. The presentation is illustrated with the results obtained in different projects.

Large scale artificial neural networks (ANNs) have been widely used in data processing applications. The recurrent neural network (RNN) is a special type of neural network equipped with additional recurrent connections. Such a unique architecture enables the recurrent neural network to remember the past processed information and makes it an expressive model for nonlinear sequence processing tasks. However, the large computation complexity makes it difficult to effectively train a recurrent neural network and therefore significantly limits the research on the recurrent neural network in the last 20 years. In recent years, the use of graphics processing units (GPUs) becomes a significant advance to speed up the training process of large scale neural networks by taking advantage of the massive parallelism capabilities of GPUs. In this paper, we propose an efficient GPU implementation of the large scale recurrent neural network and demonstrate the power of scaling up the recurrent neural network with GPUs. We first explore the potential parallelism of the recurrent neural network and propose a fine-grained two-stage pipeline implementation. Experiment results show that the proposed GPU implementation can achieve 2 ~ 11 x speed-up compared with the basic CPU implementation with the Intel Math Kernel Library. We then use the proposed GPU implementation to scale up the recurrent neural network and improve its performance. The experiment results of the Microsoft Research Sentence Completion Challenge demonstrate that the large scale recurrent network without class layer is able to beat the traditional class-based modest-size recurrent network and achieve an accuracy of 47%, the best result achieved by a single recurrent neural network on the same dataset.

Simulation of spiking neural networks — architectures and implementations

In this paper we present a digital hardware platform that is capable of emulating large scale artificial neural networks in real-time. The approach bases on the bio-inspired modified Hebbian learning algorithm incorporating both short- and long-term dynamics. The system allows the configuration of arbitrarily structured networks which are externally loadable through a common PC. In the current state of the project the platform comprises seven modules with a Xilinx XC4028 FPGA processing about 8000 neurons each. A typical application of this approach is in the domain of adaptive autonomous agents.

This paper introduces a waveform generation algorithm from three speech parameters (fundamental frequency fo, spectral envelope, and band aperiodicity). The conventional speech analysis/synthesis system based on a vocoder mainly has a waveform generator based on pitch synchronous overlap and add (PSOLA). Since it uses the fast Fourier transform (FFT) to generate the vocal cord vibration, the processing speed is proportional to the f <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">o</sub> . The algorithm also uses the spectral representation of the aperiodicity, whereas the band aperiodicity is mainly used in speech synthesis applications such as statistical parametric speech synthesis. We propose a waveform generation algorithm that reduces the computational cost and memory usage without degrading the synthesized speech. The algorithm utilizes excitation signal generation by directly using the band aperiodicity. The computational cost in a certain period is fixed because the excitation signal is filtered and processed by the overlap-add (OLA) algorithm. We used the re-synthesized speech to perform two evaluations for the processing speed and sound quality. The results showed that the sound quality of speech synthesized was almost the same by our proposed algorithm as by the conventional algorithm. The proposed algorithm can also reduce computational cost and memory usage.

We propose a new method for analyzing fiber Bragg gratings having an arbitrary index profile and an arbitrary individual grating length. The proposed method is based on the lattice filter model which are widely used in the signal processing community ranging from digital filtering to speech synthesis and explosive seismic signal processing applications. Lattice filter interpretation provides us with an accurate and simple tool for analyzing arbitrary aperiodic fiber grating structures, and gives us further insight to the understanding of the fiber Bragg gratings. To verify the validity of the proposed model, we have fabricated two grating structures; the short period (periodic) fiber Bragg grating structure and the chirped fiber Bragg grating structure. We have observed that the predicted transmission spectrum and the predicted reflectivity using the lattice filter model match very closely to the corresponding measured spectrum curves in the band of the wavelength of our interest.

The intrinsic electrophysiological properties of single neurons can be described by a broad spectrum of models, from realistic Hodgkin-Huxley-type models with numerous detailed mechanisms to the phenomenological models. The adaptive exponential integrate-and-fire (AdEx) model has emerged as a convenient middle-ground model. With a low computational cost but keeping biophysical interpretation of the parameters, it has been extensively used for simulations of large neural networks. However, because of its current-based adaptation, it can generate unrealistic behaviors. We show the limitations of the AdEx model, and to avoid them, we introduce the conductance-based adaptive exponential integrate-and-fire model (CAdEx). We give an analysis of the dynamics of the CAdEx model and show the variety of firing patterns it can produce. We propose the CAdEx model as a richer alternative to perform network simulations with simplified models reproducing neuronal intrinsic properties.

Spiking behavior and epileptiform oscillations in a discrete model of cortical neural networks

This paper reports further development and applications of a computational model permitting analysis of frequency response characteristics of simulated neurons. The original single‐neuron model, as described by Peddicord and Sampson in 1974, is briefly reviewed here. There follows a thorough description of a new interactive implementation of the model on a PDP‐11/45. The implementation permits high‐speed simulation of large neural networks, and is designed for easy use by investigators with no special knowledge of computers or programming. Experiments with the new system have included the design of improved bandpass filters, pacemaker neurons and networks, and a cerebellar circuit. Simulation results are presented and, in the last case, compared with available physiological data. The paper concludes with a critique of the system, suggestions for further experiments, and a comparison with the simulation system developed by Perkel. Two appendices document the system data structures and show a sample simulation run.