Regularization of the Trajectories of Dynamical Systems by Adjusting Parameters

This paper presents the development of several efficient learning algorithms for neural networks (ELEANNE). The ELEANNE 1 and ELEANNE 2 are two recursive least-squares learning algorithms, proposed for training single-layered neural networks with analog output. This paper also proposes a new optimization strategy for training single-layered neural networks, which provides the basis for the development of a variety of efficient learning algorithms. This optimization strategy is the source of the ELEANNE 3, a second-order learning algorithm for training single-layered neural networks with binary output. A simplified version of this algorithm, called ELEANNE 4, is also derived on the basis of some simplifying but reasonable assumptions. The two algorithms developed for single-layered neural networks provide the basis for the derivation of ELEANNE 5 and ELEANNE 6, which are proposed for training multilayered neural networks with binary output. The ELEANNE 7 is an efficient algorithm developed for training multilayered neural networks with either binary or analog output. >

Feasible trajectories of linear dynamic systems with inequality constraints using higher-order representations

We study a Josephson junction ladder in a magnetic field in the absence of charging effects via a transfer matrix formalism. The eigenvalues of the transfer matrix are found numerically, giving a determination of the different phases of the ladder. The spatial periodicity of the ground state exhibits a devil's staircase as a function of the magnetic flux filling factor $f$. If the transverse Josephson coupling is varied a continuous superconducting-normal transition in the transverse direction is observed, analogous to the breakdown of the KAM trajectories in dynamical systems.

The Sixth International Symposium on Neural Networks (ISNN 2009)

At present, deep learning is becoming one of the most popular approach to creation of the artificial intelligences systems such as speech recognition, natural language processing, computer vision and so on. The paper presents a historical overview of deep learning in neural networks. The model of the artificial neural network is described as well as the learning algorithms for neural networks including the error backpropagation algorithm, which is used to train deep neural networks. The development of neural networks architectures is presented including neocognitron, autoencoders, convolutional neural networks, restricted Boltzmann machine, deep belief networks, long short-term memory, gated recurrent neural networks, and residual networks. Training deep neural networks with many hidden layers is impeded by the vanishing gradient problem. The paper describes the approaches to solve this problem that provide the ability to train neural networks with more than hundred layers. An overview of popular deep learning libraries is presented. Nowadays, for computer vision tasks convolutional neural networks are utilized, while for sequence processing, including natural language processing, recurrent networks are preferred solution, primarily long short-term memory networks and gated recurrent neural networks.

In the paper, the research present situation and development in the wavelet neural network model and a novel learning algorithm for wavelet neural network based on extended Kalman filter are discussed. Based on combining the exceptional property of localization of the wavelet transform and characteristics of self-learning of neural networks, the non-line time series model and network architecture model which combines affine transform with revolving transform is discussed .A novel learning algorithm for wavelet neural network based on extended Kalman filter is proposed to predict the deformation of structure. In comparison with the WNN algorithm, the EKF learning algorithm has improved convergence and can provide much more accuracy learning results.

We estimate the spectrum envelope of speech by using neural networks. The neural networks system proposed in our study consists of two parts. The first part is the network for detecting the spectrum peaks, and the other is for interpolating between the extracted peaks. The structure of the latter is derived from the Neville interpolation. The former is based on the peak picking method. We present an operator for detecting the peaks using neural networks. Unfortunately, the result is that the neural networks do not only detect the general peaks but also the local peaks and the extremely closed peaks. We train the neural networks to detect only the general peaks. The learning algorithm uses a steepest descent method such as the backpropagation (BP) algorithm. After the learning, we show the detected peaks and the spectrum envelope when we use the learned pattern as the input data. In addition, we perform an experiment to examine whether the proposed operator corresponds to the XOR Boolean function or not.

Designing and Learning Algorithm of Neural Networks for Pattern Recognition

Artificial neural networks provide a neurocomputing approach for solving complex problems that might otherwise not have a tractable solution. Neural networks are usually trained using local gradient-based procedures. Such methods are frequently find sub-optimal solutions being trapped in local minima. And It will be difficult for the traditional neural network to solve such problems where the input/output data sets used to train a neural network may not be hundred percent precise but within certain range. A learning algorithm based on interval optimization is presented in this paper. The above disadvantages of the traditional learning algorithm can be settled by using this method.

The paper proposes a fast online learning method for neural network structures by using genetic algorithm (GA) and dynamic back propagation algorithm (BP) jointly. GA is used in the coarse tuning process which adjusts interconnection weights of the neural network. The dynamic back propagation algorithm is subsequently applied to achieve fine adjusting of the network weights. The fitness function, based on the squared error between the teaching signal and the network output value, is redefined at every time step and the proposed GA based algorithm solves a nonstationary function optimization task. At every time step the solution with the best fitness function is used for current representation of the neural network weights and biases. It is shown through the simulations and real time temperature control of drying oven that this learning algorithm has faster convergence ability and better performance on reducing mapping error in the online learning neural network structures. This leads to an improvement of the transient response of neuro adaptive systems. The proposed method has the potential to be applied to many practical areas such as system modeling and control, signal processing and pattern recognition.

Currently, Thai restaurants are popular around the world. There are tons of reviews related to foods and services in social networking websites. These tons of customer reviews make it difficult to analyze the opinions of customer toward foods and services. To help the businesses, the model of opinion mining is proposed for classifying the reviews and to analyze the attitude of customers for improving their products and services. In this research, the artificial neural network is applied to classify the positive and negative reviews. In addition, the mRMR feature selection is used to select the features of data in order to reduce the number of features in the data set. Consequently, the computational times of learning algorithms for neural networks are reduced. The experimental results show that the neural network is an effective model for classifying the Thai restaurant reviews.

Fractional calculus has been adopted in the modelling of many scientific processes and systems. Due to the inherent feature of long term memory of fractional derivatives, it has been used in the learning process of neural networks. A fractional order derivative based back propagation learning algorithm in neural networks is proposed in this paper. Specifically, Riemann-Liouville (R-L), Caputo (C) and Caputo Fabrizio (CF) fractional Derivative based on the back propagation algorithms in a three layer feed-forward neural network employed. To get a faster learning rate without oscillation, momentum factor is incorporated. The effect of fractional order and momentum factor is investigated and compared. The performance of these fractional derivatives based algorithms with integer derivatives based algorithm in terms of mean square error (MSE), particularly the salary based on years of experience is predicted. Results demonstrate that fractional derivative based learning algorithms outperform the integer derivatives.

In this paper, a hybrid supervised learning algorithm for neural network was proposed. The problem of local minimum learning usually occurred in the real application of neural network is tried to be solved or reduced. In order to improve the efficiency and stability of conventional error back-propagation learning algorithm, a hybrid learning method combining the linear multi-regression and backpropagation techniques was developed. To demonstrate the superiority of the method we developed, one example was simulated. The conventional BP learning method was also performed as the comparison with the new method proposed. From the results shown, the conventional BP method easily makes neural model plunge into the local minima. On the contrary, the new method we proposed not only has a fast learning, but also has a better learning efficiency.

In this thesis the study of convergence of piecewise smooth systems and discontinuous dynamical networks is considered. The contributions of the thesis are related both in the investigation of synchronization of networks with discontinuities in the agents's dynamics or in the inter-agents communication and in the investigation of incremental stability of nonsmooth systems. In particular, a framework for the study of synchronization when the nodes' dynamics may be both piecewise smooth and/or nonidentical across the network is presented. The analysis is performed using nonsmooth analysis tools, in particular set-valued Lyapunov functions. In this way, sufficient conditions are derived in order to guarantee global bounded synchronization, with analytical expression of the estimates of the synchronization bound and of the minimum coupling strength required to achieve synchronization. Different hypotheses on the heterogeneity of the agents' dynamics is considered, and the analysis is conducted both for linear and nonlinear coupling protocols. Differently from the few results available in the literature, boundedness of trajectories of the coupled systems is not required a priori, as well as any condition of synchrony among switching signals. Referring to the networks where discontinuities are present at the level of the communication among the agents, in the thesis novel event-triggered control strategies are developed. Such distributed algorithms are able to synchronize networks of general Lipschitz nonlinear systems. The results represent an innovative contribution on the existing literature for two main reasons. The existing literature considers only synchronization of integrators or linear systems and the control signals and communication signals are not both simultaneously piecewise constant, which is instead guaranteed in the developed strategies. Furthermore, a strategy able to guarantee a lower bound for the inter-event times between consecutive updates of the control law is proposed. More in detail, the considered strategy is model based and each node uses the dynamical model of its neighbours to predict their evolution and evaluate the triggering events. Both the cases of exact and not exact information of the neighbours' dynamical model are addressed and synchronization is analytically proven in both the cases. Furthermore, Zeno behaviour are excluded. The focus on the convergence properties and on the incremental stability of nonsmooth dynamical systems leads in the thesis to extend contraction theory, which is a tool used in the literature to address incremental exponential stability of smooth systems. In particular, the extension of such tool to the class of time switching non differentiable dynamical systems which satisfy Caratheodory conditions for the existence and uniqueness of a solution is considered. More in detail, it is proved that infinitesimal contraction of each mode of a switched system of interest gives a sufficient condition for global exponential convergence of trajectories towards each other. Such extension can be related with analogous results already present in the literature for convergent systems, which instead make use of Lyapunov conditions. The results are then used to develop smooth and piecewise smooth distributed protocols to coordinate and synchronize networks of piecewise smooth agents. Furthermore, the open problem of proving incremental stability of Filippov systems is introduced and preliminary results are given for the case of planar systems. All the results in the thesis are analytically proven and illustrated via a set of representative numerical examples.

Notice of Violation of IEEE Publication Principles<br><br>"Comparative Study Among Different Neural Net Learning Algorithms Applied to Rainfall Predication"<br>by Smita Kulkarni and Milind Mushrif <br>in the Proceedings of the International Conference on Electronic Systems, Signal Processing and Computing Technologies" January 2014, pp. 209-216<br><br>After careful and considered review of the content and authorship of this paper by a duly constituted expert committee, this paper has been found to be in violation of IEEE's Publication Principles.<br><br>This paper is a duplication of the original text from the paper cited below. The original text was copied without attribution (including appropriate references to the original author(s) and/or paper title) and without permission.<br><br>Due to the nature of this violation, reasonable effort should be made to remove all past references to this paper, and future references should be made to the following article:<br><br>"Comparative Study Amoung Different Neural Net Learning Algorithms Applied to Rainfall Time Series"<br>by Surajit Chattopadhyay and Goutami Chattopadhyay<br>in Meteorological Applications, 15, Wiley Interscience, April 2008, pp. 273-280<br><br> <br/> The present article reports studies to identify a non-linear methodology to forecast the time series of average summer-monsoon rainfall over India. Three advanced backpropagation neural network learning rules namely, momentum learning, conjugate gradient descent (CGD) learning, and Levenberg -- Marquardt (LM) learning, and a statistical methodology in the form of asymptotic regression are implemented for this purpose. Monsoon rainfall data pertaining to the years from 1871 to 1999 are explored. After a thorough skill comparison using statistical procedures the study reports the potential of CGD as a learning algorithm for the backpropagation neural network to predict the said time series.