Self-organization of orientation maps in a formal neuron model using a cluster learning rule

We study the natural function space for infinitely wide two-layer neural networks with ReLU activation (Barron space) and establish different representation formulae. In two cases, we describe the space explicitly up to isomorphism. Using a convenient representation, we study the pointwise properties of two-layer networks and show that functions whose singular set is fractal or curved (for example distance functions from smooth submanifolds) cannot be represented by infinitely wide two-layer networks with finite path-norm. We use this structure theorem to show that the only $$C^1$$ -diffeomorphisms which preserve Barron space are affine. Furthermore, we show that every Barron function can be decomposed as the sum of a bounded and a positively one-homogeneous function and that there exist Barron functions which decay rapidly at infinity and are globally Lebesgue-integrable. This result suggests that two-layer neural networks may be able to approximate a greater variety of functions than commonly believed.

A neural network controller which is used for controlling unknown discrete-time DARMA systems is described. In a first stage, a two-layered neural network is used to estimate the unknown plant dynamics. The Widrow-Hoff delta rule is used as the learning algorithm for this network so as to minimize the difference between the plant actual response and that predicted by the neural network. In a second stage, the control law is generated online using a second two-layered neural network so that the plant output is brought to a desired reference signal. Simulation examples are presented to evaluate the design.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

In the ontogenesis process, the cerebral cortex undergoes age-related changes. So far as, unlike practically all other systems of mammalian organs, the brain continues to develop and receive new functionality in the postnatal period. Thus with age, there are changes in the bioelectric characteristics of the neocortex. The purpose of the research is to determine the age and sex changes in the bioelectric activity of the cerebral visual cortex of male and female rats of different ages. In the article, we examined changes in absolute (μV2) and normalized (%) indicators of electrical activity of the visual area of rats of different sexes in four age groups: juvenile, young, mature, and presenile age. The research was carried out by the method of registration of bioelectric activity of electrocorticograms (ECoG). Results of multifactorial dispersion analysis of absolute and normalized ECoG indicators of the visual area of the cerebral cortex of rats of all ages showed that there were reliable changes in the frequency-amplitude characteristics of bioelectric activity related to age. At a young age, males have probably lower absolute and normalized power of the delta-rhythm and the normalized beta-like rhythm rate. There was a tendency of decrease in the theta-rhythm. As a result of this redistribution of rhythms a desynchronization of the electrical activity of young males was observed. Mature males have lower absolute power indicators than younger age groups. According to normalized indicators, a synchronization of rhythms of males in the mature age group was observed, which together with values of absolute power can indicate a decrease of the functional activity level of the neocortex and an increase of the influence of endogenous mechanisms on neuronal activity of the visual area of the cerebral cortex. In the presenile age, there was a desynchronization of rhythms. The indicators of an absolute and normalized power of females in the young age group pointed to the synchronization of the bioecoactivity of the neocortex. Among the low-frequency waves, theta-rhythm rhythm dominated in the females of young age. In the mature and presenile age females, there was a probable increase in the percentage of normalized parameters of high-frequency beta waves. The predominance of this rhythm may indicate an increase of cortical tone. The correlation of values of absolute and normalized indicators of bioelectric activity affirmed the desynchronization of the ECoG rhythms of females of mature and presenile age. Sex differences are expressed by a decrease in the absolute power of all ECoG rhythms of the visual area of the cerebral cortex of females relative to males. The age-related changes occurred in a different way in rats of different sex. At a young age, males had desynchronization of rhythms, and females had synchronization, in the mature age it was vice versa. The age changes of the electric activity of the cerebral visual cortex of rats in our opinion may be largely associated with involutional changes of various neurotransmitter and hormonal systems.

Non-destructive orientation mapping is an important characterization tool in materials science and geoscience for understanding and/or improving material properties based on their grain structure. Confocal Raman microscopy is a powerful non-destructive technique for chemical mapping of organic and inorganic materials. Here we demonstrate orientation mapping by means of Polarized Raman Microscopy (PRM). While the concept that PRM is sensitive to orientation changes is known, to our knowledge, an actual quantitative orientation mapping has never been presented before. Using a concept of ambiguity-free orientation determination analysis, we present fast and quantitative single-acquisition Raman-based orientation mapping by simultaneous registration of multiple Raman scattering spectra obtained at different polarizations. We demonstrate applications of this approach for two- and three-dimensional orientation mapping of a multigrain semiconductor, a pharmaceutical tablet formulation and a polycrystalline sapphire sample. This technique can potentially move traditional X-ray and electron diffraction type experiments into conventional optical laboratories.

An anti-Hebbian local learning algorithm for two-layer optical neural networks is introduced. With this learning rule, the weight update for a certain connection depends only on the input and output of that connection and a global, scalar error signal. Therefore the backpropagation of error signals through the network, as required by the commonly used back error propagation algorithm, is avoided. It still guarantees, however, that the synaptic weights are updated in the error descent direction. With the apparent advantage of simpler optical implementation this learning rule is also shown by simulations to be computationally effective.

Spiking Neural Networks (SNNs), An alternative to sigmoidal neural networks, include time into their operations using discrete signals called spikes. Employing spikes enables SNNs to mimic any feedforward sigmoidal neural network with lower power consumption. Recently a new type of SNN has been introduced for classification problems, known as Degree of Belonging SNN (DoB-SNN). DoB-SNN is a two-layer spiking neural network that shows significant potential as an alternative SNN architecture and learning algorithm. This paper introduces a new variant of Spike-Timing Dependent Plasticity (STDP), which is based on the assembly of neurons and expands the DoB-SNN's training algorithm for multilayer architectures. The new learning rule, known as assembly-based STDP, employs trained DoBs in each layer to train the next layer and build strong connections between neurons from the same assembly while creating inhibitory connections between neurons from different assemblies in two consecutive layers. The performance of the multilayer DoB-SNN is evaluated on five datasets from the UCI machine learning repository. Detailed comparisons on these datasets with other supervised learning algorithms show that the multilayer DoB-SNN can achieve better performance on 4/5 datasets and comparable performance on 5th when compared to multilayer algorithms that employ considerably more trainable parameters.

Supervised learning is mainly used to optimize Adaptive Neural Fuzzy Inference System (ANFIS) controllers. In order to generate data for supervised learning, a controller is designed and optimized using Particle Swarm Optimization (PSO) or any other algorithms. This paper proposes and compares reinforcement learning based ANFIS and Approximate Reasoning Intelligent controller (ARIC) controllers. Reinforcement learning based ANFIS reduces the work flow required to train it by directly optimizing the membership functions using Proximal Policy Optimization (PPO) algorithm. ANFIS and ARIC neuro fuzzy controllers are designed for nonlinear dynamics of coffee roasting process using Schwartzberg’s model. A custom layer is designed for every membership function and fuzzy inference operations using MATLAB’s Deep Learning Toolbox. This neural connectionist model of ANFIS and ARIC is used as actor. The critic which evaluates the goodness of action taken is a two-layer neural network with sigmoidal activation function. Simulink environment is also created to represent the dynamics of coffee roasting process. The agent is trained to track roast profile for 50 episodes. The training converged at 50th iteration. After training, the Root Mean Square Error (RMSE) for ARIC architecture reduced from 0.5134 to 0.08122. Similarly, the RMSE of ANFIS improved from 0.2026 to 0.0624.

Stripe-rearing changes multiple aspects of the structure of primary visual cortex

Deep neural networks achieve stellar generalisation even when they have enough parameters to easily fit all their training data. We study this phenomenon by analysing the dynamics and the performance of over-parameterised two-layer neural networks in the teacher–student setup, where one network, the student, is trained on data generated by another network, called the teacher. We show how the dynamics of stochastic gradient descent (SGD) is captured by a set of differential equations and prove that this description is asymptotically exact in the limit of large inputs. Using this framework, we calculate the final generalisation error of student networks that have more parameters than their teachers. We find that the final generalisation error of the student increases with network size when training only the first layer, but stays constant or even decreases with size when training both layers. We show that these different behaviours have their root in the different solutions SGD finds for different activation functions. Our results indicate that achieving good generalisation in neural networks goes beyond the properties of SGD alone and depends on the interplay of at least the algorithm, the model architecture, and the data set.

We introduce the absolute refractory behaviour into the formal neuron model. While a probabilistic approach to such a refractory model has yet been attempted, in this paper, a deterministic analysis is realized. A first result consists in showing a not expensive algorithm to transform each refractory net into an equivalent not refractory one. Such a result is then exploited to obtain an upper bound to the computational complexity of two classical problems: the reachability and stabilization problems. They find their principal motivations in control and learning theories whenever the necessity to a priori determine the lenght of both transients and limit cycles arises. Finally, we prove that, when the connection matrices of nets are symmetric, the complementary problem of stabilization is NP-complete and reachability is P-complete.

The lower and upper bounds for the information capacity of two-layer feedforward neural networks with binary interconnections, integer thresholds for the hidden units, and zero threshold for the output unit is obtained through two steps. First, through a constructive approach based on statistical analysis, it is shown that a specifically constructed (N-2L-1) network with N input units, 2L hidden units, and one output unit is capable of implementing, with almost probability one, any dichotomy of O(W/ln W) random samples drawn from some continuous distributions, where W is the total number of weights of the network. This quantity is then used as a lower bound for the information capacity C of all (N-2L-1) networks with binary weights. Second, an upper bound is obtained and shown to be O(W) by a simple counting argument. Therefore, we have /spl Omega/(W/ln W)/spl les/C/spl les/O(W).

The momentum method has become the workhorse in the deep learning community. To theoretical understand its success, researchers put efforts in demystifying its convergence properties when optimizing neural networks. For the convex problem, it is well-known that the triple momentum (TM) method owns the fastest theoretical convergence rate among all first-order methods. However, there exists no theoretical convergence results about the TM method in solving the non-convex neural networks training problem, let alone its acceleration guarantee. In this paper, we focus on the training process of the TM method for a two-layer ReLU neural network. Inspired by the accurate characterization of the high-resolution dynamical system, we consider the high-resolution ordinary differential equation (ODE) of the TM method. Under the over-parameterized assumption, we derive that the original non-convex optimization problem can be transformed to a strongly convex task. By applying an appropriate Lyapunov function, we prove that the TM method can linearly converge to a global minimum. Compared to the heavy ball method and Nesterov’s accelerated gradient method, our result provides the first guarantee for the acceleration of the TM method in training neural networks. Through empirical experiments, the accelerated convergence of the TM method and the effect of over-parameterization are validated.

After the issue of keywords not effectively representing the theme in topic discovery tasks, this paper proposes a text topic representation method based on Seq2Seq-Attention, aiming to create a concise and refined topic representation. Firstly, the Encoder module uses a two-layer bidirectional recurrent neural network for text information feature extraction, the Decoder sub-module uses a two-layer unidirectional recurrent neural network and combines with an attention mechanism model to complete the decoding task and outputs the desired sequence of popular topic characters. Finally, the effectiveness of the method is verified through comparative experiments.

The key of fast identification algorithm of time-varying modal parameter based on subspace tracking is to find efficient and fast subspace-tracking algorithm.This paper presents a new version of NIC(Novel Information Criterion) using two-layer linear neural network learning for subspace tracking.Comparing with the original algorithm, there is no need to set a key control parameter in advance.Simulation experiments show that new algorithm has a faster convergence in the initial period.

A fairly comprehensive analysis is presented for the gradient descent dynamics for training two-layer neural network models in the situation when the parameters in both layers are updated. General initialization schemes as well as general regimes for the network width and training data size are considered. In the over-parametrized regime, it is shown that gradient descent dynamics can achieve zero training loss exponentially fast regardless of the quality of the labels. In addition, it is proved that throughout the training process the functions represented by the neural network model are uniformly close to that of a kernel method. For general values of the network width and training data size, sharp estimates of the generalization error is established for target functions in the appropriate reproducing kernel Hilbert space.