Investigations in neural network learning using geometrical approach

The paper proposes a fast online learning method for neural network structures by using genetic algorithm (GA) and dynamic back propagation algorithm (BP) jointly. GA is used in the coarse tuning process which adjusts interconnection weights of the neural network. The dynamic back propagation algorithm is subsequently applied to achieve fine adjusting of the network weights. The fitness function, based on the squared error between the teaching signal and the network output value, is redefined at every time step and the proposed GA based algorithm solves a nonstationary function optimization task. At every time step the solution with the best fitness function is used for current representation of the neural network weights and biases. It is shown through the simulations and real time temperature control of drying oven that this learning algorithm has faster convergence ability and better performance on reducing mapping error in the online learning neural network structures. This leads to an improvement of the transient response of neuro adaptive systems. The proposed method has the potential to be applied to many practical areas such as system modeling and control, signal processing and pattern recognition.

Artificial neural networks provide a neurocomputing approach for solving complex problems that might otherwise not have a tractable solution. Neural networks are usually trained using local gradient-based procedures. Such methods are frequently find sub-optimal solutions being trapped in local minima. And It will be difficult for the traditional neural network to solve such problems where the input/output data sets used to train a neural network may not be hundred percent precise but within certain range. A learning algorithm based on interval optimization is presented in this paper. The above disadvantages of the traditional learning algorithm can be settled by using this method.

A fast learning algorithm is proposed for training of multilayer feedforward neural networks, based on a combination of optimal linear Kalman filtering theory and error propagation. In this algorithm, all the information available from the start of the training process to the current training sample is exploited in the update procedure for the weight vector of each neuron in the network in an efficient parallel recursive method. This innovation is a massively parallel implementation and has better convergence properties than the conventional backpropagation learning technique. Its performance is illustrated on some examples, such as a XOR logical operation and a nonlinear mapping of two continuous signals. >

The Sixth International Symposium on Neural Networks (ISNN 2009)

Fast Learning Algorithms

At present, deep learning is becoming one of the most popular approach to creation of the artificial intelligences systems such as speech recognition, natural language processing, computer vision and so on. The paper presents a historical overview of deep learning in neural networks. The model of the artificial neural network is described as well as the learning algorithms for neural networks including the error backpropagation algorithm, which is used to train deep neural networks. The development of neural networks architectures is presented including neocognitron, autoencoders, convolutional neural networks, restricted Boltzmann machine, deep belief networks, long short-term memory, gated recurrent neural networks, and residual networks. Training deep neural networks with many hidden layers is impeded by the vanishing gradient problem. The paper describes the approaches to solve this problem that provide the ability to train neural networks with more than hundred layers. An overview of popular deep learning libraries is presented. Nowadays, for computer vision tasks convolutional neural networks are utilized, while for sequence processing, including natural language processing, recurrent networks are preferred solution, primarily long short-term memory networks and gated recurrent neural networks.

In this paper, a novel neural-fuzzy controller based on fuzzy differential competitive learning is proposed. Since one of the most important parts is the generation of the fuzzy rules in the design of the fuzzy control system, a fast learning algorithm, fuzzy differential competitive learning (FDCL), for the generation of the rules is applied in the fuzzy control system. The FDCL algorithm adopts a principle of learn according to how well it wins. Unlike the previous competitive learning algorithm such as crisp competitive learning algorithms where only one neuron will win and learn at each competition step every neuron in the neural network based on FDCL algorithm will along with its different distance to the input pattern and learns the pattern accordingly. Compared with the ordinary competitive learning algorithm the proposed FDCL algorithm has the various distinguishing features. The FDCL algorithm is implanted in the neural network based fuzzy system and the network adopted is fuzzy associative memory system (FAMS) which simulates the knowledge representation and inference process by using fuzzy notation and by association in neural networks. In FAMS the fuzzy rules will be generated by clustering the input-output training data through the FDCL paradigm. By using the FDCL algorithm the neural network can highly refine knowledge and represent the expert experience.

For High Energy Physics (HEP) experiments, such as the Large Hadron Collider (LHC) experiments, the calorimeter is a key detector to measure the energy of particles. Incident particles interact with the material of the calorimeter, creating cascades of secondary particles, so-called showers. A detailed description of the showering process relies on simulation methods that precisely describe all particle interactions with matter. Constrained by the need for precision, the simulation of calorimeters is inherently slow and constitutes a bottleneck for HEP analysis. Furthermore, with the upcoming high luminosity upgrade of the LHC and a much-increased data production rate, the amount of required simulated events will increase. Several research directions have recently investigated the use of Machine Learning based models to accelerate particular calorimeter response simulation. These models typically require a large amount of data and time for training, and the result is a simulation tuned specifically to this configuration. Meta-learning has emerged recently as a fast learning algorithm using small training datasets. In this paper, we use a meta-learning approach that “learns to learn” to generate showers from multiple calorimeter geometries, using a first-order gradient-based algorithm. We present MetaHEP, the first application of the meta-learning approach to accelerate shower simulation using very high granular data and using one of the calorimeters proposed for the Future Circular Collider (FCC), a next-generation of high-performance particle colliders.

A fast and robust learning algorithm for feedforward neural networks

The main focus of this paper is to introduce a new supervised learning algorithm for spiking neural networks. The learning algorithm minimizes the overall differences between spike times of target and test spike trains by utilizing a new quantitative similarity measure which has been defined in this work. The actual membrane potential of a post-synaptic neuron is adjusted at the time of spikes based on what has been measured from similarity measure in order to generate the desired membrane potential. Finally, by utilizing gradient descent algorithm, the parameters of the spiking neural network are tuned to generate the desired output membrane potential. The proposed algorithm was applied to tune the facilitation, depression, and synaptic weight constants of the Dynamic Synapses Neural Network - DSNN - for the aim of input-output functional mapping. The simulation results show that the system identification task converges to the global optimum. The rate-to-time coding simulation performs with more than 75 percent accuracy. The performance of both system identification and rate-to-time coding is due to adaptation of short and long term synaptic parameters which cannot be accomplished if only synaptic weight is adapted.

This paper presents the development of several efficient learning algorithms for neural networks (ELEANNE). The ELEANNE 1 and ELEANNE 2 are two recursive least-squares learning algorithms, proposed for training single-layered neural networks with analog output. This paper also proposes a new optimization strategy for training single-layered neural networks, which provides the basis for the development of a variety of efficient learning algorithms. This optimization strategy is the source of the ELEANNE 3, a second-order learning algorithm for training single-layered neural networks with binary output. A simplified version of this algorithm, called ELEANNE 4, is also derived on the basis of some simplifying but reasonable assumptions. The two algorithms developed for single-layered neural networks provide the basis for the derivation of ELEANNE 5 and ELEANNE 6, which are proposed for training multilayered neural networks with binary output. The ELEANNE 7 is an efficient algorithm developed for training multilayered neural networks with either binary or analog output. >

A problem of an adaptive neural network system synthesizes in a case of a nonlinear actuation channel's singularities is considered. The singularities can cause a loss of stability if the back propagation algorithm is used without modification. Known approaches of solving the problem are considered and a new modification of adaptive learning algorithm is proposed that can be applied to a wider class of processes.

In this paper, a new AdaBoost learning framework, called WNS-AdaBoost, is proposed for training discriminative models. The proposed approach significantly speeds up the learning process of adaptive boosting (AdaBoost) by reducing the number of data points. For this purpose, we introduce the weighted novelty selection (WNS) sampling strategy and combine it with AdaBoost to obtain an efficient and fast learning algorithm. WNS selects a representative subset of data thereby reducing the number of data points onto which AdaBoost is applied. In addition, WNS associates a weight with each selected data point such that the weighted subset approximates the distribution of all the training data. This ensures that AdaBoost can trained efficiently and with minimal loss of accuracy. The performance of WNS-AdaBoost is first demonstrated in a classification task. Then, WNS is employed in a probabilistic boosting-tree (PBT) structure for image segmentation. Results in these two applications show that the training time using WNS-AdaBoost is greatly reduced at the cost of only a few percent in accuracy.

In this paper, a hybrid supervised learning algorithm for neural network was proposed. The problem of local minimum learning usually occurred in the real application of neural network is tried to be solved or reduced. In order to improve the efficiency and stability of conventional error back-propagation learning algorithm, a hybrid learning method combining the linear multi-regression and backpropagation techniques was developed. To demonstrate the superiority of the method we developed, one example was simulated. The conventional BP learning method was also performed as the comparison with the new method proposed. From the results shown, the conventional BP method easily makes neural model plunge into the local minima. On the contrary, the new method we proposed not only has a fast learning, but also has a better learning efficiency.

University admissions and business personnel offices use a limited number of resources to process an ever-increasing quantity of student and employment applications. Application systems are further constrained to identify and acquire, in a limited time period, those candidates who are most likely to accept an offer of enrolment or employment. Neural networks are a new methodology to this particular domain. Various neural network architectures and learning algorithms are analyzed comparatively to determine the applicability of supervised learning neural networks to the domain problem of personnel resource allocation and to identify optimal learning strategies in this domain. This paper focuses on multilayer perceptron backpropagation, radial basis function, counterpropagation, general regression, fuzzy ARTMAP, and linear vector quantization neural networks. Each neural network predicts the probability of enrolment and nonenrolment for individual student applicants. Backpropagation networks produced the best overall performance. Network performance results are measured by the reduction in counsellors student case load and corresponding increases in student enrolment. The backpropagation neural networks achieve a 56% reduction in counsellor case load.