Two regularizers for recursive least squared algorithms in feedforward multilayered neural networks

Sparse adaptive filtering algorithms are utilized to exploit potential sparse structure information as well as to mitigate noises in many unknown sparse systems. Sparse recursive least square (RLS) algorithms have been attracted intensely attentions due to their low-complexity and easy- implementation. Basically, these algorithms are constructed by standard RLS algorithm and sparse penalty functions (e.g., l_1-norm). However, existing sparse RLS algorithms do not exploit the sparsity efficiently. In this paper, an improved adaptive filtering algorithm is proposed by incorporating a novel correntropy induced metric (CIM) constraint into RLS, which is termed as RLS- CIM algorithm. Specifically, we adopt a well-known Gaussian kernel in CIM and further devise a novel variable kernel width to control the sparse penalty in different transient-error scenarios. Numerical simulation results are given to corroborate the proposed algorithm via mean square deviation (MSD).

This paper proposes a recursive least-squares (RLS) algorithm with multiple time-varying forgetting factors for on-line parameter estimation of an induction machine (IM). The regressive mathematical model of the IM is also introduced which is simple and appropriate for online parameter estimation. The estimator inputs using the proposed RLS algorithm are easily measurable variables such as the stator voltages and currents as well as the rotor speed of the IM. The simulation results obtained compare the estimated parameters with the IM parameters achieved using other RLS algorithms such as a standard RLS algorithm and a RLS with a constant forgetting factor. The comparison shows that the proposed RLS algorithm is better than others for on-line parameter estimation of the IM.

Envelope tracking has many important applications in electrical distribution systems for either flicker meters or flicker compensators. This paper presents a novel control technique for flicker mitigation. This technique is based on the instantaneous tracking of the measured voltage envelope. The ADAptive LINEar neuron (ADALINE) algorithm, and the Recursive Least Square (RLS) algorithm are introduced for the flicker envelope tracking. The ADALINE algorithm is characterized by a light computational demand, unlike that of existing techniques; the RLS algorithm outperforms the ADALINE algorithm by its fast convergence and robust tracking performance. Both the ADALINE and the RLS algorithms give accurate results even under rapid dynamic changes. The paper investigates the effects of different parameters on the performance of the ADALINE algorithm and that of the RLS algorithm. Extensive simulations of the proposed flicker tracking algorithms are conducted to evaluate their performance for the tracking and mitigation of voltage flicker. The ADALINE and the RLS algorithms are examined by tracking and mitigating the flicker produced by a resistance welder and an arc furnace in a simple distribution system simulated in the PSCAD/EMTDC package.

A new recursive least square (RLS) algorithm based on nonlinear principal component analysis (NPCA) is proposed to address the blind source separation (BSS) problem in non-stationary environment. Forgetting factor is introduced to improve the tracking ability in non-stationary environment. The Kalman filter is used to solve the NPCA problem since its outstanding tracking performance in non-stationary environments. Simulations using the real speech source signals are used to illustrate the performance of the new RLS algorithm in static and non-stationary environments. Results show that the new RLS algorithm has faster convergence rate and better tracking capacity compared with the stochastic gradient algorithm, and previous RLS algorithm.

The recursive least square (RLS) algorithm is insensitive to the dispersions of the correlation matrix. Consequently, the usage of the RLS algorithm to update the adaptive recursive filter coefficients are attractive for the system modeling and identification fields. In this paper, an error function RLS (EFRLS) algorithm is derived for the adaptive recursive filter. Furthermore, a scaled version of the EFRLS algorithm is proposed where the computation complexity is significantly reduced. The scaled EFRLS algorithm can be realized in real time using the available digital signal processors. The convergence to the optimal solution of both the EFRLS algorithm and its scaled version is ensured while the general RLS algorithm fails to converge to the optimal solution for the multimodal performance criterion.

In this paper, a method of updating filter coefficients is proposed in which the RLS (Recursive Least Squares) algorithm is applied to a noise canceler with a filter for crosstalk removal. The RLS algorithm has an advantage over the Least Mean Square algorithm by virtue of better convergence. When the filter coefficients are updated including the filter for crosstalk removal, the error signal becomes nonlinear in terms of filter coefficients. Hence, a nonlinear RLS algorithm is derived in which such nonlinearity is taken into account. It is shown that the convergence is further improved in comparison with the conventional RLS algorithm. Finally, a method is proposed to reduce the noise contained in the transmitted voice signals of telephone by means of the present noise canceler and its effectiveness is evaluated. © 2003 Wiley Periodicals, Inc. Electron Comm Jpn Pt 3, 86(5): 36–44, 2003; Published online in Wiley InterScience (www.interscience.wiley.com). DOI 10.1002/ecjc.10039

Recursive implementations of the least squares algorithm start the computation with a known set of initial conditions. The information contained in new data samples is then used to update the old estimates. Therefore the initialization procedure is seen to be an integral part of recursive algorithms. A number of methods for initializing recursive least squares (RLS) algorithms have been proposed in the literature. The two most common methods are the fast exact initialization and the soft constrained initialization. This paper discusses the equivalence relationship between RLS algorithms that use soft constrained initialization and a widely used technique in statistics called ridge regression.

Vanadium redox flow battery (VRB) energy storage system has been widely utilized in renewable energy applications such as wind power integration and green buildings. An online electrical model of VRB is needed to monitor and estimate battery states by using real time data since the internal parameters of battery are varying during operation. In this paper, an improved second-order Thevenin model is proposed to describe the operating characteristics of VRB. To overcome the drawback of recursive least squares (RLS) algorithm in tracking parameter identification, this paper proposes a method based on time-varying forgetting factor recursive least Squares (TFF-RLS) algorithm to identify model parameters. According to the data collected by the pulse charging experiment, the RLS algorithm and the TFF-RLS algorithm are used to identify the model parameters. Experiments are carried out on a 5kWh/3kWh VRB experimental platform in Wuhan University of Technology. Two online identification methods are compared. The results show that the TFF-RLS method demonstrates a better performance on tracking the variation of parameters dynamically, and improving estimation accuracy compared with the RLS method.

In this paper, we proposed a recurrent kernel recursive least square (RLS) algorithm for online learning. In classical kernel methods, the kernel function number grows as the number of training sample increases, which makes the computational cost of the algorithm very high and only applicable for offline learning. In order to make the kernel methods suitable for online learning where the system is updated when a new training sample is obtained, a compact dictionary (support vectors set) should be chosen to represent the whole training data, which in turn reduces the number of kernel functions. For this purpose, a sparsification method based on the Hessian matrix of the loss function is applied to continuously examine the importance of the new training sample and determine the update of the dictionary according to the importance measure. We show that the Hessian matrix is equivalent to the correlation matrix of the training samples in the RLS algorithm. This makes the sparsification method able to be easily incorporated into the RLS algorithm and reduce the computational cost futher. Simulation results show that our algorithm is an effective learning method for online chaotic signal prediction and nonlinear system identification.

The Cerebellar Model Articulation Controller (CMAC) neural network is an associative memory that is biologically inspired by the cerebellum, which is found in the brains of animals. The standard CMAC uses the least mean squares algorithm to train the weights. Recently, the recursive least squares (RLS) algorithm was proposed as a superior algorithm for training the CMAC online as it can converge in just one epoch, and does not require tuning of a learning rate. However, the RLS algorithm was found to be very computationally demanding as its computational complexity is dependent on the square of the number of weights required which can be huge for the CMAC. Here, we show a more efficient RLS algorithm that uses inverse QR decomposition and additionally provides a regularized solution, improving generalization. However, while the inverse QR decomposition based RLS algorithm reduces computation time significantly; it is still not fast enough for use in CMACs greater than two dimensions. To further improve efficiency we show that by using kernel methods the CMAC computational complexity can be transformed to become dependent on the number of unique training data. Additionally, it is shown how modeling error can be improved through use of higher order basis functions.

This chapter presents recursive least-squares (RLS) algorithms, which are based on numerically robust Householder transformations. Section 7.1 introduces the conventional orthogonal Householder transform and provides its geometrical interpretation. The hyperbolic Householder and the row (orthogonal and hyperbolic) Householder transforms are also briefly described. In Section 7.2, the Householder RLS (HRLS) algorithm is presented, its relation with the other known square-root RLS algorithms is discussed, and two applications, where the HRLS algorithm has been successfully applied, are presented. By considering a block-by-block update approach to the RLS problem, the block exact Householder QRD-RLS algorithm is developed in Section 7.3, which constitutes a generalization of the conventional QRD-RLS algorithm. Section 7.4 presents an inverse QRD-RLS algorithm, which employs block updating via the application of appropriate row orthogonal Householder transformations. Finally, a sliding window block RLS algorithm, which comprises a pair of row Householder transforms, is introduced in Section 7.5.

The authors propose a new discrete time blind deconvolution technique for linear channels driven by cyclostationary inputs. This problem arises in digital communications, seismic signal processing, and many other applications. In particular, homomorphic approaches are applied to the cyclic autocorrelation of the fractionally-spaced sampled output of the channel. First, the method identifies the differential cepstrum parameters of the complex channel by means of a recursive least squares (RLS) algorithm. The RLS algorithm is based on the special characteristics of a cyclic autocorrelation matrix and an appropriate matrix inversion lemma. Once the differential cepstrum parameters are recovered, then the impulse response of the channel/equalizer is obtained by simple recursive formulas. Only partial information is required, i.e., the cyclic period and the distribution of the input data. It is shown that the method can directly identify the characteristics of either the channel or its inverse, provided that an unknown channel satisfies a special condition. The method is evaluated by means of computer simulations and is found to perform efficiently. >

In this work, we propose an adaptive linear interference suppression scheme based on single-carrier frequency domain equalization (SC-FDE) for multiuser direct-sequence ultra-wideband (DS-UWB) systems. In a block-by-block based transmission, by inserting the cyclic prefix (CP) between each data block, the inter-block interference (IBI) can be ignored and the Toeplitz channel matrix can be expressed as an equivalent circulant channel matrix. With this feature, we propose a simple signal model, in which only a vector formed adaptive filter needs to be trained in the frequency domain. Based on the new signal model, an adaptive block conjugate gradient (BCG) algorithm is developed. Complexity analysis shows that the proposed BCG algorithm is significantly simpler than the recursive least squares (RLS) algorithm. The bit-error rate (BER) performance of the proposed scheme is compared with the least-mean-square (LMS) and RLS algorithm in a number of scenarios and the simulation results show fast convergence speed and excellent performance of the proposed BCG algorithm.

This paper proposes a novel algorithm, called an order extended FTF (fast transversal filter) algorithm, which is suitable for adaptive arrays based on spatial and temporal signal processing such as decision feedback adaptive arrays. The proposed algorithm can be implemented with hardware whose complexity is proportional to the filter tap length, while the RLS (recursive least squares) algorithm requires hardware in proportion to the filter tap length. In spite of its low hardware complexity, the order extended FTF algorithm has an initial acquisition performance as fast as the RLS algorithm.

In a TDMA mobile communication system, an adaptive decision feedback equalizer (DFE) is the common device to compensate the intersymbol interference caused by the multipath effect. Due to the fast fading characteristic of channels, most conventional DFEs use the recursive least square (RLS) algorithm. However, it is known that the RLS algorithm is computationally intensive and not easy to implement. The LMS is well known for its simple structure, however, due to its poor tracking property, it cannot be used. Recently, a multiple-training and multiple-tracking LMS (MTT-LMS) algorithm was developed. It was shown that the DFE trained by the MTT-LMS algorithm can yield a satisfactory result at moderate mobile speed. The MTT-LMS algorithm retains the simple structure of the LMS algorithm and is suitable for hardware implementation. In this paper, we consider the ASIC design of the MTT-LMS DFE for application to North American IS-54 systems. We have designed an efficient architecture which not only has small chip area but low clock rate. We also develop a new decision device to provide the desired signal for the MTT-LMS algorithm and the implementation of the device is simpler than the conventional one.