Antiforgetting Incremental Learning Algorithm for Interval Type-2 Fuzzy Neural Network

This paper proposes a recurrent interval type-2 fuzzy neural network with asymmetric membership functions (RT2FNN-A). The RT2FNN-A uses the interval asymmetric type-2 fuzzy sets and it implements the FLS in a five layer neural network structure which contains four layer forward network and a feedback layer. Each asymmetric fuzzy member function (AFMF) is constructed by parts of four Gaussian functions. The corresponding learning algorithm is derived by gradient descent method. Finally, the RT2FNN-A is applied in identification of nonlinear dynamic system. Simulation results are shown to illustrate the effectiveness of the RT2FNN-A systems.

This paper presents the development of several efficient learning algorithms for neural networks (ELEANNE). The ELEANNE 1 and ELEANNE 2 are two recursive least-squares learning algorithms, proposed for training single-layered neural networks with analog output. This paper also proposes a new optimization strategy for training single-layered neural networks, which provides the basis for the development of a variety of efficient learning algorithms. This optimization strategy is the source of the ELEANNE 3, a second-order learning algorithm for training single-layered neural networks with binary output. A simplified version of this algorithm, called ELEANNE 4, is also derived on the basis of some simplifying but reasonable assumptions. The two algorithms developed for single-layered neural networks provide the basis for the derivation of ELEANNE 5 and ELEANNE 6, which are proposed for training multilayered neural networks with binary output. The ELEANNE 7 is an efficient algorithm developed for training multilayered neural networks with either binary or analog output. >

In the paper, the research present situation and development in the wavelet neural network model and a novel learning algorithm for wavelet neural network based on extended Kalman filter are discussed. Based on combining the exceptional property of localization of the wavelet transform and characteristics of self-learning of neural networks, the non-line time series model and network architecture model which combines affine transform with revolving transform is discussed .A novel learning algorithm for wavelet neural network based on extended Kalman filter is proposed to predict the deformation of structure. In comparison with the WNN algorithm, the EKF learning algorithm has improved convergence and can provide much more accuracy learning results.

Many real scenarios in machine learning are non-stationary. These challenges forces to develop new algorithms that are able to deal with changes in the underlying problem to be learnt. These changes can be gradual or abrupt. As the dynamics of the changes can be different, the existing machine learning algorithms exhibit difficulties to cope with them. In this work we propose a new method, that is based in the introduction of a forgetting function in an incremental online learning algorithm for two-layer feedforward neural networks. This forgetting function gives a monotonically crescent importance to new data. Due to this fact, the network forgets in presence of changes while maintaining a stable behavior when the context is stationary. The theoretical basis for the method is given and its performance is illustrated by evaluating its behavior. The results confirm that the proposed method is able to work in evolving environments.

Extreme learning machine (ELM) is an easy-to use and effective learning algorithm of single-hidden layer feed-forward neural networks (SLFNs). The classical learning algorithm in neural network, e. g. backpropagation, requires setting several user-defined parameters and may get into local minimum. However, ELM only requires setting the number of hidden neurons and the activation function. It does not require adjusting the input weights and hidden layer biases during the implementation of the algorithm, and it produces only one optimal solution. Therefore, ELM has the advantages of fast learning speed and good generalization performance. In this paper, ELM is introduced in predicting reservoir permeability. By comparing to SVM, we analyze its feasibility and advantages in reservoir permeability prediction. The experimental results show that ELM has similar accuracy compared to SVR, but it has obvious advantages in parameter selection and learning speed.

The present commentary addresses the Quartz & Sejnowski (Q&S) target article from the point of view of the dynamical learning algorithm for neural networks. These techniques implicitly adopt Q&S's neural constructivist paradigm. Their approach hence receives support from the biological and psychological evidence. Limitations of constructive learning for neural networks are discussed with an emphasis on grammar learning.

Fractional calculus has been adopted in the modelling of many scientific processes and systems. Due to the inherent feature of long term memory of fractional derivatives, it has been used in the learning process of neural networks. A fractional order derivative based back propagation learning algorithm in neural networks is proposed in this paper. Specifically, Riemann-Liouville (R-L), Caputo (C) and Caputo Fabrizio (CF) fractional Derivative based on the back propagation algorithms in a three layer feed-forward neural network employed. To get a faster learning rate without oscillation, momentum factor is incorporated. The effect of fractional order and momentum factor is investigated and compared. The performance of these fractional derivatives based algorithms with integer derivatives based algorithm in terms of mean square error (MSE), particularly the salary based on years of experience is predicted. Results demonstrate that fractional derivative based learning algorithms outperform the integer derivatives.

New incremental learning algorithm using extended data expression, based on probabilistic compounding, is presented in this paper. Incremental learning algorithm generates an ensemble of weak classifiers and compounds these classifiers to a strong classifier, using a weighted majority voting, to improve classification performance. We introduce new probabilistic weighted majority voting founded on extended data expression. In this case class distribution of the output is used to compound classifiers. UChoo, a decision tree classifier for extended data expression, is used as a base classifier, as it allows obtaining extended output expression that defines class distribution of the output. Extended data expression and UChoo classifier are powerful techniques in classification and rule refinement problem. In this paper extended data expression is applied to obtain probabilistic results with probabilistic majority voting. To show performance advantages, new algorithm is compared with Learn++, an incremental ensemble-based algorithm.

This paper proves the problem of losing incremental samples' information of the present SVM incremental learning algorithm from both theoretic and experimental aspects, and proposes a new incremental learning algorithm with support vector machine based on hyperplane-distance. According to the geometric character of support vector, the algorithm uses Hyperplane-Distance to extract the samples, selects samples which are most likely to become support vector to form the vector set of edge, and conducts the support vector machine training on the vector set. This method reduces the number of training samples and effectively improves training speed of incremental learning. The results of experiment performed on Chinese webpage classification show that this algorithm can reduce the number of training samples effectively and accumulate historical information. The HD-SVM algorithm has higher training speed and better precision of classification.

To learn things incrementally without the catastrophic interference, we have proposed Resource Allocating Network with Long-Term Memory (RAN-LTM). In RAN-LTM, not only training data but also memory items stored in long-term memory are trained. In this paper, we propose an extended RAN-LTM called Resource Allocating Network by Local Linear Regression (RAN-LLR), in which its centers are not trained but selected based on output errors and the connections are updated by solving a linear regression problem. To reduce the computation and memory costs, the modified connections are restricted based on RBF activity. In the experiments, we first apply RAN-LLR to a one-dimensional function approximation problem to see how the negative interference is effectively suppressed. Then, the performance of RAN-LLR is evaluated for a real-world prediction problem. The experimental results demonstrate that the proposed RAN-LLR can learn fast and accurately with less memory costs compared with the conventional models.

A fast and robust algorithm for training feedforward neural networks (FNNs) by using a variable forgetting factor and U-D factorization-based recursive prediction error (RPE) method is proposed. In comparison with the backpropagation (BP) and RPE based learning algorithms, the proposed algorithm, called UD-RPE, can provide much more accurate learning results in fewer iterations with fewer hidden nodes and improve convergence rate and numerical stability (robustness). In addition, it is less sensitive to start-up parameters, such as initial weights and initial covariance matrix, and the randomness in the observed data. It also has good generalization ability and needs less learning time. Simulation results of nonlinear dynamic system modeling and identification show that the algorithm proposed here is an effective and efficient learning algorithm for FNNs.

A new fast learning algorithm for a multilayer feedforward neural network is presented. The new algorithm, based on an innovative variable step size and gradient averaging, has excellent properties to overcome major drawbacks of the backpropagation algorithm and shows high convergence speed. The improved performance of the new algorithm in comparison with other algorithms is verified via computer simulations.

Semi-supervised learning is especially important in data mining applications because it can make use of plentiful unlabeled data to train the high-quality learning models. Semi-Supervised Support Vector Machine (S3VM) is a powerful semi-supervised learning model. However, the high computational cost and non-convexity severely impede the S3VM method in large-scale applications. Although several learning algorithms were proposed for S3VM, scaling up S3VM is still an open problem. To address this challenging problem, in this paper, we propose a new incremental learning algorithm to scale up S3VM (IL-S3VM) based on the path following technique in the framework of Difference of Convex (DC) programming. The traditional DC programming based algorithms need multiple outer loops and are not suitable for incremental learning, and traditional path following algorithms are limited to convex problems. Our new IL-S3VM algorithm based on the path-following technique can directly update the solution of S3VM to converge to a local minimum within one outer loop so that the efficient incremental learning can be achieved. More importantly, we provide the finite convergence analysis for our new algorithm. To the best of our knowledge, our new IL-S3VM algorithm is the first efficient path following algorithm for a non-convex problem (i.e., S3VM) with local minimum convergence guarantee. Experimental results on a variety of benchmark datasets not only confirm the finite convergence of IL-S3VM, but also show a huge reduction of computational time compared with existing batch and incremental learning algorithms, while retaining the similar generalization performance.

This paper presents an adaptive update scheme for all digital delay locked loop(ADDLL). To reduce the output jitter induced by the power fluctuation, the proposed ADDLL adaptively adjust the variable delay line step using a phase error detector. The phase error detector detects the phase error variation every sixteen external clock period. Once the phase error over the threshold, the adaptive update engine is activated to suppress the power fluctuation. The proposed ADDLL achieve that the tDQSCK (DQS rising edge output timing location from rising edge of CK) drift is under +/-35ps when the power fluctuation is +/-60mV.

Bayesian network utilizes graphical model to describe dependencies among variables in probabilistic way, it is one of the most important model for uncertainty processing in Artificial Intelligence. Incremental learning of Bayesian networks has been received more attentions in recent years, in this paper a novel method is proposed to learn Bayesian network from incremental data. In this method, a novel incremental scoring function is designed to adaptively adjust the tendency of matching new and old data in the process of incremental learning. We propose an improved adaptive incremental structure learning algorithm for Bayesian network. Theoretical analysis and experimental results both demonstrate the proposed method outperforms other state-of-the-art methods.