Rates of Convergence for an Adaptive Filtering Algorithm Driven by Stationary Dependent Data

Biased stochastic conjugate gradient algorithm with adaptive step size for nonconvex problems

An adaptive Volterra filter (AVF) is one of the identification methods to identify Volterra kernels of a target nonlinear system via any adaptive algorithm. However, the convergence speed and the identification accuracy of the AVF may deteriorate in the case of colored input signal. An adaptive Wiener filter (AWF) is one of the solutions to solve the problem of the AVF. Since the AWF guarantees the orthogonality of the Gaussian white noise on each order input signal, the identification accuracy is improved compared with the AVF. However, when the AWF is used for identification of the target nonlinear system, the auto-correlation matrix of each order input signal vector may have different eigenvalues and convergence speed becomes slower. One of the solutions for this problem is stochastic gradient adaptive algorithm. In this paper, we examine the identification ability for loudspeaker systems by the AWF with stochastic gradient adaptive algorithm. Simulation and experiment results demonstrate that the convergence speed can be improved compared with the AVF.

In this paper, several reported blind adaptive beam- forming algorithms, including the stochastic gradient (SG) methods and the recursive least squares (RLS) methods are reviewed. Then, the conjugate gradient (CG) algorithm is introduced as a tradeoff between SG and RLS algorithms. On the basis of this idea, a novel constrained constant modulus (CCM) design criterion is described and employed for deriving a modified conjugate gradient (CG) optimization algorithm, which does not require the information of the desired user beforehand. Simulations are performed to compare the performance of the proposed method with both SG and RLS methods. The results indicate that the proposed approach has a significant faster convergence rate and superior resolution compared with the SG algorithms and lower computation complexity compared with the RLS algorithms.

This paper studies several identification methods for Hammerstein systems with piece-wise linearities. By using the key term separation technique, the model of the nonlinear Hammerstein systems be changed to an identification model, then based on the derived model, a stochastic gradient identification algorithm, a forgetting factor stochastic gradient algorithm and a modified stochastic gradient algorithm are used to estimate all the unknown parameters of the systems. An example is provided to show the effectiveness of the proposed algorithms.

SummaryIn this paper, by means of the adaptive filtering technique and the multi‐innovation identification theory, an adaptive filtering‐based multi‐innovation stochastic gradient identification algorithm is derived for Hammerstein nonlinear systems with colored noise. The new adaptive filtering configuration consists of a noise whitening filter and a parameter estimator. The simulation results show that the proposed algorithm has higher parameter estimation accuracies and faster convergence rates than the multi‐innovation stochastic gradient algorithm for the same innovation length. As the innovation length increases, the filtering‐based multi‐innovation stochastic gradient algorithm gives smaller parameter estimation errors than the recursive least squares algorithm.

In this work we investigate the performance of blind adaptive receivers for direct sequence code division multiple access (DS-CDMA) systems that employ stochastic gradient (SG) algorithms with variable step size mechanisms. Two low complexity variable step size mechanisms are proposed for constrained minimum variance (CMV) receivers that operate with SG algorithms and are also incorporated in the channel estimation algorithms. We examine the proposed adaptation techniques and compare them with recently reported approaches in frequency selective fading channels and nonstationary environments. Simulation experiments in a number of scenarios show that, although less complex, the proposed mechanisms for DS-CDMA receivers are superior to recently reported methods.

This paper develops a novel passive stochastic gradient algorithm. In passive stochastic approximation, the stochastic gradient algorithm does not have control over the location where noisy gradients of the cost function are evaluated. Classical passive stochastic gradient algorithms use a kernel that approximates a Dirac delta to weigh the gradients based on how far they are evaluated from the desired point. In this paper we construct a multi-kernel passive stochastic gradient algorithm. The algorithm performs substantially better in high dimensional problems and incorporates variance reduction. We analyze the weak convergence of the multi-kernel algorithm and its rate of convergence. In numerical examples, we study the multi-kernel version of the passive least mean squares (LMS) algorithm for transfer learning to compare the performance with the classical passive version.

We consider a simple estimator of the density mode using the frequency polygon estimate. We investigate strong consistency of the estimator for strong mixing sequence of real variables under mild assumptions. We study the almost sure rate of convergence and we show that the estimator can achieve optimal almost sure rates of convergence for appropriate choices of the bin widths. The asymptotic normality of the simple estimator is given and a simulation study is performed. Our asymptotic results are obtained without any differentiability condition assumed on the density around the mode.

Parameter identification algorithm for ship manoeuvrability and wave peak model based multi-innovation stochastic gradient algorithm use data filtering technique

This work develops and analyzes a class of adaptive biased stochastic optimization (ABSO) algorithms from the perspective of the GEneralized Adaptive gRadient (GEAR) method that contains Adam, AdaGrad, RMSProp, etc. Particularly, two preferred biased stochastic optimization (BSO) algorithms, the biased stochastic variance reduction gradient (BSVRG) algorithm and the stochastic recursive gradient algorithm (SARAH), equipped with GEAR, are first considered in this work, leading to two ABSO algorithms: BSVRG-GEAR and SARAH-GEAR. We present a uniform analysis of ABSO algorithms for minimizing strongly convex (SC) and Polyak-Łojasiewicz (PŁ) composite objective functions. Second, we also use our framework to develop another novel BSO algorithm, adaptive biased stochastic conjugate gradient (coined BSCG-GEAR), which achieves the well-known oracle complexity. Specifically, under mild conditions, we prove that the resulting ABSO algorithms attain a linear convergence rate on both PŁ and SC cases. Moreover, we show that the complexity of the resulting ABSO algorithms is comparable to that of advanced stochastic gradient-based algorithms. Finally, we demonstrate the empirical superiority and the numerical stability of the resulting ABSO algorithms by conducting numerical experiments on different applications of machine learning.

A comparison of stochastic gradient and minimum entropy deconvolution algorithms

Phase retrieval, recovering a signal from phaseless measurements, is generally considered to be an NP-hard problem. This paper adopts an amplitude-based nonconvex optimization cost function to develop a new stochastic gradient algorithm, named stochastic reweighted phase retrieval (SRPR). SRPR is a stochastic gradient iteration algorithm, which runs in two stages: First, we use a truncated sample stochastic variance reduction algorithm to initialize the objective function. The second stage is the gradient refinement stage, which uses continuous updating of the amplitude-based stochastic weighted gradient algorithm to improve the initial estimate. Because of the stochastic method, each iteration of the two stages of SRPR involves only one equation. Therefore, SRPR is simple, scalable, and fast. Compared with the state-of-the-art phase retrieval algorithm, simulation results show that SRPR has a faster convergence speed and fewer magnitude-only measurements required to reconstruct the signal, under the real- or complex- cases.

We analyze the stochastic average gradient (SAG) method for optimizing the sum of a finite number of smooth convex functions. Like stochastic gradient (SG) methods, the SAG method's iteration cost is independent of the number of terms in the sum. However, by incorporating a memory of previous gradient values the SAG method achieves a faster convergence rate than black-box SG methods. The convergence rate is improved from $$O(1/\sqrt{k})$$O(1/k) to O(1 / k) in general, and when the sum is strongly-convex the convergence rate is improved from the sub-linear O(1 / k) to a linear convergence rate of the form $$O(\rho ^k)$$O(?k) for $$\rho < 1$$?<1. Further, in many cases the convergence rate of the new method is also faster than black-box deterministic gradient methods, in terms of the number of gradient evaluations. This extends our earlier work Le Roux et al. (Adv Neural Inf Process Syst, 2012), which only lead to a faster rate for well-conditioned strongly-convex problems. Numerical experiments indicate that the new algorithm often dramatically outperforms existing SG and deterministic gradient methods, and that the performance may be further improved through the use of non-uniform sampling strategies.

The least mean square methods include two typical parameter estimation algorithms, which are the projection algorithm and the stochastic gradient algorithm, the former is sensitive to noise and the latter is not capable of tracking the time‐varying parameters. On the basis of these two typical algorithms, this study presents a generalised projection identification algorithm (or a finite data window stochastic gradient identification algorithm) for time‐varying systems and studies its convergence by using the stochastic process theory. The analysis indicates that the generalised projection algorithm can track the time‐varying parameters and requires less computational effort compared with the forgetting factor recursive least squares algorithm. The way of choosing the data window length is stated so that the minimum parameter estimation error upper bound can be obtained. The numerical examples are provided.

A stochastic conjugate gradient multi-modulus algorithm based on global artificial fish swarm and genetic optimization DNA encoding sequences(GAFS-GDNA-SMMA) is proposed to solve the high computational loads, low convergence rate, and large mean square error(MSE) of the multi-modulus blind equalization algorithm(MMA). In this proposed algorithm, the optimal DNA coding sequence can be found and used as the initial optimal weight vector of the stochastic conjugate gradient MMA(SMMA) after decoding the optimization DNA sequences. The simulation results show that the proposed algorithm has the faster convergence speed and smaller mean square error comparison with MMA, SMMA, GA-DNA-SMMA(Genetic algorithm and DNA optimization based SMMA)