Architectures, learning and inversion algorithms for multilayer neural networks

A VLSI bio-inspired microsystem for real-time, on-site, robust image analysis and recognition has been developed. A corresponding novel artificial neural network (ANN) learning algorithm using new sigmoid-logarithmic transfer function is invented. Our results show the trained new ANN can recognize low fluorescence patterns better than the conventional sigmoidal ANN does. A differential logarithmic imaging chip is designed for calculating logarithm of relative intensities of fluorescence signals. The single-rail logarithmic circuit and a prototype ANN chip are designed, fabricated and characterized.

This paper presents the development of several efficient learning algorithms for neural networks (ELEANNE). The ELEANNE 1 and ELEANNE 2 are two recursive least-squares learning algorithms, proposed for training single-layered neural networks with analog output. This paper also proposes a new optimization strategy for training single-layered neural networks, which provides the basis for the development of a variety of efficient learning algorithms. This optimization strategy is the source of the ELEANNE 3, a second-order learning algorithm for training single-layered neural networks with binary output. A simplified version of this algorithm, called ELEANNE 4, is also derived on the basis of some simplifying but reasonable assumptions. The two algorithms developed for single-layered neural networks provide the basis for the derivation of ELEANNE 5 and ELEANNE 6, which are proposed for training multilayered neural networks with binary output. The ELEANNE 7 is an efficient algorithm developed for training multilayered neural networks with either binary or analog output. >

Artificial neural networks provide a neurocomputing approach for solving complex problems that might otherwise not have a tractable solution. Neural networks are usually trained using local gradient-based procedures. Such methods are frequently find sub-optimal solutions being trapped in local minima. And It will be difficult for the traditional neural network to solve such problems where the input/output data sets used to train a neural network may not be hundred percent precise but within certain range. A learning algorithm based on interval optimization is presented in this paper. The above disadvantages of the traditional learning algorithm can be settled by using this method.

At present, deep learning is becoming one of the most popular approach to creation of the artificial intelligences systems such as speech recognition, natural language processing, computer vision and so on. The paper presents a historical overview of deep learning in neural networks. The model of the artificial neural network is described as well as the learning algorithms for neural networks including the error backpropagation algorithm, which is used to train deep neural networks. The development of neural networks architectures is presented including neocognitron, autoencoders, convolutional neural networks, restricted Boltzmann machine, deep belief networks, long short-term memory, gated recurrent neural networks, and residual networks. Training deep neural networks with many hidden layers is impeded by the vanishing gradient problem. The paper describes the approaches to solve this problem that provide the ability to train neural networks with more than hundred layers. An overview of popular deep learning libraries is presented. Nowadays, for computer vision tasks convolutional neural networks are utilized, while for sequence processing, including natural language processing, recurrent networks are preferred solution, primarily long short-term memory networks and gated recurrent neural networks.

A problem of an adaptive neural network system synthesizes in a case of a nonlinear actuation channel's singularities is considered. The singularities can cause a loss of stability if the back propagation algorithm is used without modification. Known approaches of solving the problem are considered and a new modification of adaptive learning algorithm is proposed that can be applied to a wider class of processes.

Most of the present navigation sensor integration techniques are based on Kalman filtering estimation procedures. Although Kalman filtering represents one of the best solutions for multi-sensor integration, it has some drawbacks in terms of stability, computation load, immunity to noise effects and observability. Furthermore, Kalman filters perform adequately only under certain predefined dynamic models. Neuron computing, a technology of Artificial Neural Network (ANN), is a powerful tool for solving nonlinear problems that involve mapping input data to output data without having any prior knowledge about the mathematical process involved. This article suggests a multi-sensor integration approach for fusing data from an Inertial Navigation system (INS) and Differential Global Positioning System (DGPS) utilizing multi-layer feed-forward neural networks with a back propagation learning algorithm. The performance of the proposed architecture was tested using two different INS systems(tactical grade IMU and navigation grade IMU) in a land vehicle.

There exist many learning algorithms for finding the synaptic weights of a layered neural network so as to minimize the output errors in response to the plural training data. The most well-known learning algorithms are the backpropagation procedures, in which there are two manners of minimizing the plural error functions in practice. One is minimizing the sum of all error functions totally, and the other is making a reduction of only one error function chosen temporarily and cyclically at each iteration. The former does not always keep the balance and the efficiency in reduction of the plural output errors, and the latter does not guarantee the convergence from the standpoint of numerical analysis.

University admissions and business personnel offices use a limited number of resources to process an ever-increasing quantity of student and employment applications. Application systems are further constrained to identify and acquire, in a limited time period, those candidates who are most likely to accept an offer of enrolment or employment. Neural networks are a new methodology to this particular domain. Various neural network architectures and learning algorithms are analyzed comparatively to determine the applicability of supervised learning neural networks to the domain problem of personnel resource allocation and to identify optimal learning strategies in this domain. This paper focuses on multilayer perceptron backpropagation, radial basis function, counterpropagation, general regression, fuzzy ARTMAP, and linear vector quantization neural networks. Each neural network predicts the probability of enrolment and nonenrolment for individual student applicants. Backpropagation networks produced the best overall performance. Network performance results are measured by the reduction in counsellors student case load and corresponding increases in student enrolment. The backpropagation neural networks achieve a 56% reduction in counsellor case load.

This paper presents a detailed performance analysis of the minimal resource allocation network (M-RAN) learning algorithm, M-RAN is a sequential learning radial basis function neural network which combines the growth criterion of the resource allocating network (RAN) of Platt (1991) with a pruning strategy based on the relative contribution of each hidden unit to the overall network output. The resulting network leads toward a minimal topology for the RAN. The performance of this algorithm is compared with the multilayer feedforward networks (MFNs) trained with 1) a variant of the standard backpropagation algorithm, known as RPROP and 2) the dependence identification (DI) algorithm of Moody and Antsaklis on several benchmark problems in the function approximation and pattern classification areas. For all these problems, the M-RAN algorithm is shown to realize networks with far fewer hidden neurons with better or same approximation/classification accuracy. Further, the time taken for learning (training) is also considerably shorter as M-RAN does not require repeated presentation of the training data.

Based on the critical position of the endpoint quality prediction for basic oxygen furnaces (BOFs) in steelmaking, and the latest results in computational intelligence (CI), this paper deals with the development of a novel memetic algorithm (MA) for neural network (NN) learning. Included in this is the integration of extremal optimization (EO) and Levenberg-Marquardt (LM) gradient search, and its application in BOF endpoint quality prediction. The fundamental analysis reveals that the proposed EO-LM algorithm may provide superior performance in generalization, computation efficiency, and avoid local minima, compared to traditional NN learning methods. Experimental results with production-scale BOF data show that the proposed method can effectively improve the NN model for BOF endpoint quality prediction.

We estimate the spectrum envelope of speech by using neural networks. The neural networks system proposed in our study consists of two parts. The first part is the network for detecting the spectrum peaks, and the other is for interpolating between the extracted peaks. The structure of the latter is derived from the Neville interpolation. The former is based on the peak picking method. We present an operator for detecting the peaks using neural networks. Unfortunately, the result is that the neural networks do not only detect the general peaks but also the local peaks and the extremely closed peaks. We train the neural networks to detect only the general peaks. The learning algorithm uses a steepest descent method such as the backpropagation (BP) algorithm. After the learning, we show the detected peaks and the spectrum envelope when we use the learned pattern as the input data. In addition, we perform an experiment to examine whether the proposed operator corresponds to the XOR Boolean function or not.

For many years, artificial neural networks (ANNs) have been studied and used to model information processing systems based on or inspired by biological neural structures. They not only can provide solutions with improved performance when compared with traditional problem-solving methods, but also give a deeper understanding of human cognitive abilities. Among various existing neural network architectures and learning algorithms, Kohonen’s selforganizing map (SOM) [46] is one of the most popular neural network models. Developed for an associative memory model, it is an unsupervised learning algorithm with a simple structure and computational form, and is motivated by the retina-cortex mapping. Self-organization in general is a fundamental pattern recognition process, in which intrinsic interand intra-pattern relationships among the stimuli and responses are learnt without the presence of a potentially biased or subjective external influence. The SOM can provide topologically preserved mapping from input to output spaces. Although the computational form of the SOM is very simple, numerous researchers have already examined the algorithm and many of its problems, nevertheless research in this area goes deeper and deeper – there are still many aspects to be exploited.

This paper describes the design, implementation and testing of the software tool designated as UV-SRNA-PDA (Universidad del Valle’s artificial neural network simulator for a personal digital assistant), which is oriented to artificial neural networks simulation and complex industrial processes identification. This application works on a PDA (personal digital assistant) Palm T5 using a customized data acquisition system. Two kinds of artificial neural networks were implemented: perceptron and multi-layer perceptron (MLP) using as learning algorithms the following: backpropagation, descendent gradient, variable learning rate and momentum. To test the proposed system, first and second order systems were selected (only the latter is reported here), finding their neural models and validating their results with MATLAB and UV-SRNA 2.0 (PC version of UV-SRNA-PDA). These tests achieved an average training error of 5.62 × 10-3 ± 3.55 × 10-4 and an average validating error of 4.56 × 10-3 ± 5.95 × 10-4. In both cases, the results were better or comparable with those from other software simulation tools. However, the typical training time on the UV-SRNA-PDA was 900 s compared to 3 s in MATLAB and 8 s in UV-SRNA 2.0.

In this paper, a hybrid supervised learning algorithm for neural network was proposed. The problem of local minimum learning usually occurred in the real application of neural network is tried to be solved or reduced. In order to improve the efficiency and stability of conventional error back-propagation learning algorithm, a hybrid learning method combining the linear multi-regression and backpropagation techniques was developed. To demonstrate the superiority of the method we developed, one example was simulated. The conventional BP learning method was also performed as the comparison with the new method proposed. From the results shown, the conventional BP method easily makes neural model plunge into the local minima. On the contrary, the new method we proposed not only has a fast learning, but also has a better learning efficiency.

Learning algorithms in neural networks: recent results