Distributed memory and localized memory in neural networks

Research on artificial neural networks (ANNs) has been carried out for more than five decades. A renewed interest appeared in the 80's with the finding of powerful models like J. Hopfield's recurrent networks, T. Kohonen's self-organizing feature maps, and the back-propagation rule. At that time, there was no platform that was at the same time versatile enough for any ANN model to be implemented and fast enough to solve large problems. Super-computers were the sole exception to this rule, but were prohibitively expensive for most applications. However, both research scientists and application engineers clearly identified the need for such a computing power. This triggered many projects in the field. In parallel, research on multi-processor systems started during the 60's. Systolic arrays have been proposed in 1979 as a means to fully exploit the possibilities of VLSI. Two previous theses, by F. Blayo and C. Lehmann, have studied the use of bi-dimensional systolic arrays for neural computation. At first, the presented system, called GENES, has been designed for the Hopfield model. Extensions to other ANNs have also been proposed. The goal of the present thesis is to study, design, build, and analyze an efficient accelerator for neural computation. In a first step, the GENES architecture has been extended towards generality and efficiency. This includes a thorough analysis of ANN models, of other neural computers, and of previous GENES implementations. The result of this work is the GENES IV integrated circuit, whose architecture has been co-designed by P. Ienne and the author. The main part of this thesis discusses the architecture, the design and the analysis of the MANTRA I machine, a neural computer based on a GENES IV array with up to 40 × 40 processing elements (PEs). The delta rule (and hence the Perceptron and Adaline rules), the back-propagation rule, the Hopfield model, and the Kohonen model can be implemented on this system. Although not a generic system, such a machine may be regarded as a multi-model neural computer. A prototype has been running for a year and is used daily by software designers. Several novel features distinguish the MANTRA I machine from other neural systems. First, it belongs to the few existing neural computers, contrary to the majority of implementations, which are specific to an application or to an algorithm. The machine does not hard-wire any algorithm, but provides the necessary primitives to implement the target models. This is a key feature for research, since several algorithms or versions of an algorithm can be tested on a problem. It is an important aspect for applications as well, because different ANN models are often cascaded to solve a problem. The GENES IV array — that is, the computing core of the MANTRA I machine — features synapse-level parallelism (i.e., one real or virtual PE is allocated per synapse or neural connection), while most other systems exploit only neuron-level parallelism (i.e., one PE per neuron). Hence, this system aims at a much finer parallelism grain and is well suited for massively parallel architectures. The problem size that can be computed by a neural accelerator should not be limited by the hardware (except for memory size). Therefore, it is essential to support time-sharing of PEs. On the MANTRA I machine, this is achieved by the concept of virtual arrays. Matrices are divided into sub-matrices that can be mapped onto the physical array, which is then time-shared among them. An efficient mechanism has been implemented to swap sub-matrices in background, while some other computation is performed. Since systolic arrays are pipelined systems, it is important to avoid emptying and re-filling them too often, in order to keep the hardware utilization rate high. Therefore, a systolic instruction flow has been implemented, so that each instruction follows the data for which it has been issued. Like any SIMD system, the MANTRA I machine is composed of a parallel or SIMD module and a control module. The SIMD module consists of the GENES IV array and a set of dedicated units designed for the computation that scales with the number of neurons and would poorly fit on a bi-dimensional array. A complex system of FIFOs and memories sustains the required input/output streams for the systolic array. The control module is a complete SISD system. Its tasks are (1) to control the SIMD module by dispatching instructions, (2) to manage data input and output, (3) to communicate with the external world, and (4) to perform data pre- and post-processing. The SIMD instructions are of the very long instruction word (VLIW) type. Synchronization between the two modules is achieved by an instruction FIFO. The performance of the MANTRA I machine has been analyzed using the delta rule. Measurements show that the sustained performance is very close to its peak value, as long as the problem fits in the memory banks connected to the GENES IV array. Experiments have also been run to investigate the impact of the constraints imposed by the hardware on the convergence of algorithms. Finally, the use of systolic arrays as neural accelerators is discussed in the light of the experience acquired with the GENES IV array and the MANTRA I machine. The weaknesses of the machine are analyzed, and several solutions are proposed to avoid them in a future design. A general discussion of the future of neural computers concludes this thesis.

Video compression is an essential task in video storage and transmission. In this paper, we worked for compressing the video sequence using Self-organizing map (SOM) and then stored the output feature vector of SOM using Hopfield Neural Network. The SOM algorithm is based on unsupervised, competitive learning. It extracts the useful features from video frames by feature extraction method. In our approach, the video sequence is first divided into video frames. The frames are processed by passing to the input of Self-organizing map (SOM) that outputs the informative features by removing some of the redundant information contained in video frames. Self-organizing map is an efficient method for reducing the size of images in the neural network field. The feature vector obtained is further used for storing the patterns using Hopfield network. The feature vector is applied to the input of the Hopfield neural network and stored in the network for encoding the video frames. Thus the compressed video frames are encoded and stored in memory as a codeword. The codeword is used for reconstruction of the video frames by using the same Hopfield network. The codeword is applied to the Self-organizing map network and the video frames are reconstructed from those codeword. Simulation results show that high compression rate is achieved while maintaining the good reconstruction quality of images.

Photometric clustering of regenerated plants of gladiolus by neural networks and its biological validation

Making Artificial Brains: Components, Topology, and Optimization.

The investigation of nonlinear processes in brain systems based on the oscillatory-wave approach is currently one of the most consequential domains of researching signal generation and processing mechanisms in the brain. These investigations may offer insights into various phenomena, such as the mechanism of associative memory, based on network models of biologically relevant neurons. A neural network architecture is proposed in this paper to solve applied problems, such as signal filtering, processing, and recognition of information images. Mechanisms for incomplete fragments were implemented, enabling the extraction and complete restoration of objects from memory. Each neuron in this neural network comprises a Hodgkin–Huxley biophysical model with a Mainen modification, the dynamics of which most plausibly reproduce the processes in the neural cells of the brain. In the research on individual neurons, a constant external current with varying amplitudes was applied. The study determined the average periods of self-oscillations and identified parameter values for the Andronov–Hopf bifurcation. In particular, a stable limit cycle is created and destroyed depending on different scenarios when the bias current is increased or decreased. A stable limit cycle arises via the Andronov–Hopf bifurcation and is terminated via the saddle-node bifurcation on the cycle. The study identified parameter regions whereby two neurons with excitatory and inhibitory synaptic connections synchronize. The neurons were all in self-oscillating mode and exhibited a stable limit cycle in the phase space. A three-layer neural network was constructed, comprising of a reference neuron, sensory, control, and interneurons. The control and intermediate layer connections are arranged based on the Hebbian learning rule. Areas of excitation for various types of neurons were found. Experiments were conducted using the neural network architecture to distinguish binary information patterns encoded through signal phase. Excitatory and inhibitory synaptic connections were employed to encode the images stored in memory. The dynamic patterns are determined through the in-phase or anti-phase mode of phase locking, i.e., phase synchronization relative to the global rhythm. Before reaching the output layer of neurons, the signals are summed on the interneuron layer with the signal of the reference neuron, resulting in the filtering out of a certain information segment (in-phase or anti-phase). The network accurately recognized the patterns after parameter adjustments were made. Areas of neuron phase synchronization were discovered, enabling control of network activity modes. The confirmation of the viability of phase capture and pattern recognition modes using Hodgkin–Huxley–Mainen neurons was achieved. In total, as a result of the implementation of the spike neural network modeling project: 1) two-parameter diagrams of regions of neuronal excitation in various modes are calculated; 2) the effect of excitatory and inhibitory coupling in synaptic currents is used to represent the input signal; 3) an abstract mathematical algorithm for calculating the Hebbian matrix has been transferred to the implementation of synaptic currents between layers of neurons; 4) the effect of phase clusters is used to represent the output pattern; 5) the neurons’ topology in the visual-cerebral department served as a working model for recognizing graphic images through the Hopfield network on a spike neural network, using the Hebbian rule.

The Boolean Satisfiability Problem (BSAT) is one of the most important decision problems in mathematical logic and computational sciences for determining whether or not a solution to a Boolean formula.. Hopfield neural network (HNN) is one of the major type artificial neural network (NN) popularly known for it used in solving various optimization and decision problems based on its energy minimization machinism. The existing models that incorporate standalone network projected non-versatile framework as fundamental Hopfield type of neural network (HNN) employs random search in its training stages and sometimes get trapped at local optimal solution. In this study, Ants Colony Optimzation Algorithm (ACO) as a novel variant of probabilistic metaheuristic algorithm (MA) inspired by the behavior of real Ants, has been incorporated in the training phase of Hopfield types of the neural network (HNN) to accelerate the training process for Random Boolean kSatisfiability reverse analysis (RANkSATRA) based for logic mining. The proposed hybrid model has been evaluated according to robustness and accuracy of the induced logic obtained based on the agricultural soil fertility data set (ASFDS). Based on the experimental simulation results, it reveals that the ACO can effectively work with the Hopfield type of neural network (HNN) for Random 3 Satisfiability Reverse Analysis with 87.5 % classification accuracy

Segmentation of images into textural homogeneous regions is a fundamental problem in an image understanding system. Most region-oriented segmentation approaches suffer from the problem of different thresholds selecting for different images. In this paper an adaptive image segmentation based on vector quantization is presented. It automatically segments images without preset thresholds. The approach contains a feature extraction module and a two-layer hierarchical clustering module, a vector quantizer (VQ) implemented by a competitive learning neural network in the first layer. A near-optimal competitive learning algorithm (NOLA) is employed to train the vector quantizer. NOLA combines the advantages of both Kohonen self- organizing feature map (KSFM) and K-means clustering algorithm. After the VQ is trained, the weights of the network and the number of input vectors clustered by each neuron form a 3- D topological feature map with separable hills aggregated by similar vectors. This overcomes the inability to visualize the geometric properties of data in a high-dimensional space for most other clustering algorithms. The second clustering algorithm operates in the feature map instead of the input set itself. Since the number of units in the feature map is much less than the number of feature vectors in the feature set, it is easy to check all peaks and find the `correct' number of clusters, also a key problem in current clustering techniques. In the experiments, we compare our algorithm with K-means clustering method on a variety of images. The results show that our algorithm achieves better performance.

Partial distortion entropy maximization for online data clustering

The construction of a model of the background of a scene still remains as a challenging task in video surveillance systems, in particular for moving cameras. This work presents a novel approach for constructing a panoramic background model based on competitive learning neural networks and a subsequent piecewise linear interpolation by Delaunay triangulation. The approach can handle arbitrary camera directions and zooms for a Pan-Tilt-Zoom (PTZ) camera-based surveillance system. After testing the proposed approach on several indoor sequences, the results demonstrate that the proposed method is effective and suitable to use for real-time video surveillance applications.

This work presents a new approach to solve the location management problem by using the location areas approach. Hopfield neural network is used in this work to find the optimal configuration of location areas in a mobile network. Toward this end, the location areas configuration of the network is modeled so that the mobility management cost of the system is related to the energy value of this artificial neural network. Since a pure Hopfield neural network was not efficient enough to be used in solving this problem, some modifications applied to it to make it a reasonable approach. Basically, these modifications deal with randomness of selection in manipulating the cells during the solving process. Simulation results are very promising and they lead to network configurations that are unexpected.

The Coastal Environmental Modelling Team at CSIRO Marine and Atmospheric Research, in Hobart, Tasmania, Australia, has been modelling hydrodynamic conditions within the estuarine environment of south-eastern Tasmania for several years. Historical model output has been analysed in an effort to identify prototype hydrodynamic states (i.e., frequently encountered typical hydrodynamic situations) exhibited by the estuarine environment over that period. A competitive-learning neural network, the Self-Organizing Feature Map (SOM), was used to identify these prototype states. Once such a network has been trained, each node in its output layer represents a particular pattern in the input data and nodes representing similar patterns are located near to each other on the two-dimensional output grid, while those representing dissimilar patterns are further apart. Estimated daily average surface hydrodynamic conditions (salinity, temperature and ocean current components) within the south-east Tasmanian estuarine environment, from August 2009 to August 2010, were derived from output provided by the hydrodynamic model. The current components were then analysed using a SOM and subsequent inspection of the SOM output grid enabled a number of prototypical hydrodynamic states to be identified within the model outputs.

Flow of energy and free energy minimization underpins almost every aspect of naturally occurring physical mechanisms. Inspired by this fact this work establishes an energy-based framework that spans the multi-scale range of biological neural systems and integrates synaptic dynamic, synchronous spiking activity and neural states into one consistent working paradigm. Following a bottom-up approach, a hypothetical energy function is proposed for dynamic synaptic models based on the theoretical thermodynamic principles and the Hopfield networks. We show that a synapse exposes stable operating points in terms of its excitatory postsynaptic potential as a function of its synaptic strength. We postulate that synapses in a network operating at these stable points can drive this network to an internal state of synchronous firing. The presented analysis is related to the widely investigated temporal coherent activities (cell assemblies) over a certain range of time scales (binding-by-synchrony). This introduces a novel explanation of the observed (poly)synchronous activities within networks regarding the synaptic (coupling) functionality. On a network level the transitions from one firing scheme to the other express discrete sets of neural states. The neural states exist as long as the network sustains the internal synaptic energy.

Automated detection of lane-blocking freeway incidents using artificial neural networks

A novel approach for pipeline leak fault detection and work status identification based on fuzzy clustering neural network has been studied. This approach do not need construct exact mathematical model. First of all, we preprocess dataset by extended sigmoid function to normalize each input status vector. Together with prior knowledge, a competitive learning neural network is then used to identify work status, and then the structure and detection scheme of the adaptive algorithm were developed to diagnose the leak fault. An experiment was performed at oil pipeline in Shengli Oil Field. We can learn by experiment results that the proposed method has shown the feasibility and effectiveness.

Character translation can be accomplished by a heteroassociative memory in an artificial neural network. Because of the similarity among the characters, the special features of the patterns are important in pattern recognition. In this paper, a neural-network model based on the interpattem association (IPA) concept is presented.1 Generalized logical rules are developed to construct the excitatory and inhibitory interconnections in the heteroassociative memory. An adaptive optical neural network using high-resolution liquid-crystal televisions2 is used to translate between English letters and Chinese characters. Experimental and computer-simulated results have revealed that the IPA model is more effective in recognizing input among similar characters and has a larger storage capacity than the Hopfield model. Furthermore, the IPA model has shown two major advantages: fewer interconnections and fewer gray levels.