Estimation of Autoregressive Parameters by the Constrained Total Least Square Algorithm Using a Bootstrap Method

The Total Least Squares (TLS) method is a generalized least square technique to solve an overdetermined system of equations Ax\simeqb . The TLS solution differs from the usual Least Square (LS) in that it tries to compensate for arbitrary present in both A and b . In certain problems the perturbations of A and b are linear functions of a common noise source vector. In this case we obtain a generalization of the TLS criterion called the Constrained Total Least Squares (CTLS) method by taking into account the linear dependence of the terms in A and b . If the columns of A and b are linearly related then the CTLS solution is obtained in terms of the largest eigenvalue and corresponding eigenvector of a certain matrix. The CTLS technique can be applied to problems like Maximum Likelihood Signal Parameter Estimation, Frequency Estimation of Sinusoids in white or colored by Linear Prediction and others.

A unified treatment of several least squares (LS) algorithms is presented for bearings-only tracking of a target moving at constant acceleration. The close link between the maximum likelihood (ML) estimator and other nonlinear and "linearized" LS algorithms is explored under the assumption of Gaussian bearing noise. In this context, a new asymptotically unbiased closed-form instrumental variables (IV) algorithm is derived. Reduced-bias total least squares (TLS) and constrained TLS (CTLS) algorithms are developed. The equivalence of the ML algorithm to the structured TLS (STLS) algorithm is established. Simulation examples are provided to demonstrate the improved performance of the IV and TLS estimators vis-a-vis the pseudolinear estimator.

Approximate model equations often relate given measurements to unknown parameters whose estimate is sought. The least-squares (LS) estimation criterion assumes the measured data to be exact, and seeks parameters which minimize the model errors. Existing extensions of LS, such as the total LS (TLS) and constrained TLS (CTLS) take the opposite approach, namely assume the model equations to be exact, and attribute all errors to measurement inaccuracies. We introduce the extended LS (XLS) criterion, which accommodates both error sources. We define 'pseudo-linear' models, with which we provide an iterative algorithm for minimization of the XLS criterion. Under certain statistical assumptions, we show that XLS coincides with a statistical criterion, which we term the 'joint maximum-a-posteriori-maximum-likelihood' (JMAP-ML) criterion. We identify the differences between the JMAP-ML and ML criteria, and explain the observed superiority of JMAP-ML over ML under non-asymptotic conditions.

The classical Least-Squares (LS) adjustment has been widely used in processing and analysing observations from Global Satellite Navigation System (GNSS). However, in detecting temporal correlations of GNSS observations, which can be described by means of autoregressive (AR) process, the LS method may not provide reliable estimates of process coefficients, since the Yule-Walker (YW) equations refer to structured Errors-In-Variables (EIV) equations. In this contribution, we proposed a Total Least-Squares (TLS) solution with the singular cofactor matrix to solve the YW equations. The proposed TLS solution is obtained based on the fact that random errors belong to column space of its cofactor matrix. In addition the proposed solution does not need any substitution of the squared true parameter vector as done by the current publications. Finally, we simulate the AR process to prove that our solution is more reliable than the existing methods.

On the accuracy of total least squares and least squares techniques in the presence of errors on all data

Summary In this study, we show significant improvements over conventional pressure-transient analysis on the basis of nonlinear regression by using total least squares (TLS), which minimizes errors in both pressure and time simultaneously. To our knowledge, TLS has not been applied to pressure-transient analysis before in this sense. TLS regression is not an easy problem to solve mathematically, especially for nonlinear pressure-transient-model functions. In this work, we compare four different versions of TLS. We formulate a robust approximation of the TLS solution, which can handle a variety of tests and reservoir models yet does not compromise the performance of the analysis. We show that our technique reduces ambiguity in the estimation of parameters to a large extent, especially in the presence of noise in time. Using our TLS algorithm, we obtain much narrower confidence intervals on the parameter estimates of a variety of real data sets, compared to the conventional least squares (LS) approach. For synthetic data sets, we observe that the TLS estimates are often closer to the true values than estimates made with LS, especially for poorly determined problems. When the deviation is in pressure only, TLS and LS results are comparable. However, in the presence of deviations in time in addition to pressure, the performance of TLS algorithms is substantially better. We, therefore, expect that our technique will provide more accurate estimation of reservoir parameters, allowing for better forecasting of reservoir performance.

Cardiovascular time series are often affected by noise and contain components generated by different physiological sources. Least squares (LS) modelling is a powerful tool to identify these components but it provides biased estimates when the input measurements are noisy. The total least squares (TLS) model is appropriate when noise is present in all the measures. If the noise is additive, zero-mean, independent and white, TLS method results in asymptotically unbiased, consistent estimates. The authors applied the TLS method to the following applications obtaining good results: impulsive response estimation of the transfer function which relates the respiratory sinus arrhythmia with the respiratory activity; estimation of oscillatory components in RR-interval time series, cancelling of the respiratory component in the RR-interval time series. A comparison with the LS method, performed on simulated signals, showed a better impulsive response estimation and a higher resolution frequency estimate also in a highly nonstationary environment.

In the paper by Wu et al, an interesting method was proposed to calculate the calibration parameters of a three-axis magnetometer based on the constrained total least-squares method. It is claimed in this comment that in order to construct the error of the design matrix A in equation (13) in that paper, it is the unknown magnetic field strength rather than the measured one shown in Appendix A of that paper that should be used. This results in a different loss function from that in the original paper, i.e. equation (16) therein, and hence the main algorithm should be modified accordingly.

BackgroundThe response of many biomedical systems can be modelled using a linear combination of damped exponential functions. The approximation parameters, based on equally spaced samples, can be obtained using Prony’s method and its variants (e.g. the matrix pencil method). This paper provides a tutorial on the main polynomial Prony and matrix pencil methods and their implementation in MATLAB and analyses how they perform with synthetic and multifocal visual-evoked potential (mfVEP) signals.This paper briefly describes the theoretical basis of four polynomial Prony approximation methods: classic, least squares (LS), total least squares (TLS) and matrix pencil method (MPM). In each of these cases, implementation uses general MATLAB functions. The features of the various options are tested by approximating a set of synthetic mathematical functions and evaluating filtering performance in the Prony domain when applied to mfVEP signals to improve diagnosis of patients with multiple sclerosis (MS).ResultsThe code implemented does not achieve 100%-correct signal approximation and, of the methods tested, LS and MPM perform best. When filtering mfVEP records in the Prony domain, the value of the area under the receiver-operating-characteristic (ROC) curve is 0.7055 compared with 0.6538 obtained with the usual filtering method used for this type of signal (discrete Fourier transform low-pass filter with a cut-off frequency of 35 Hz).ConclusionsThis paper reviews Prony’s method in relation to signal filtering and approximation, provides the MATLAB code needed to implement the classic, LS, TLS and MPM methods, and tests their performance in biomedical signal filtering and function approximation. It emphasizes the importance of improving the computational methods used to implement the various methods described above.

Bilinear equation method for unbiased identification of linear FIR systems in the presence of input and output noises

Three important parametric models for describing the correlation functions and spectra of stationary stochastic processes are the autoregressive (AR), moving average (MA), and autoregressive-moving average (ARMA) models. Quite recently, the MATLAB toolbox “ARMASA” has been made publicly available. This toolbox provides state-of-the-art algorithms to perform automatic identification and selection between the models based on the estimated prediction error. ARMASA works on a single segment of data, whereas in some applications, the data are available as multiple segments. We could process each segment independently and average the estimated autocorrelation functions or spectra afterward. Better performance, however, can be expected when all segments are processed simultaneously, for two reasons. Initially, the bias in the estimated model parameters depends on the number of observations in a segment. Averaging models from segments does not reduce this bias, in contrast to identification from all the data at once. In either case, the variance of the estimated model parameters will decrease. By considering all the data at once this variance reduction can be considered in the order selection stage, decreasing the selection bias. In this paper, we extend the parameter estimation algorithms of the ARMASA toolbox to the case of multiple segments. AR parameter estimation from segments has been considered before, but not for the MA and ARMA models. In addition, in the AR case, model order selection can be done easily when the segments are independent. But, in the application we are primarily interested in, the segments are formed by lines of scanning transmission electron microscopy (STEM) images, which can be correlated with each other. Because the correlation among data segments influences the effective number of independent observations available it becomes necessary to estimate that number. This is not an easy problem in general. For observations from a stochastic process with an isotropic 2-D autocorrelation function we can, however, formulate an accurate estimator in terms of the line autocorrelation function. This situation arises, for example, in STEM images of amorphous specimens. From the AR model identification for segments, the extension to the MA and ARMA models is rather straightforward because MA and ARMA algorithms are based on Durbin's methods, whose first step is the estimation of a long AR model. We compare the segment-based algorithms to conventional ARMASA with simulations of stationary stochastic processes. An illustrative example involving simulated STEM images shows large potential benefits.

Comparison of the ARMA, ARIMA, and the autoregressive artificial neural network models in forecasting the monthly inflow of Dez dam reservoir

In a previous paper [Zhu et al., J. Opt. Soc. Am. A 14, 799 (1997)] an iterative algorithm for obtaining the total least-squares (TLS) solution of a linear system based on the Rayleigh quotient formulation was presented. Here we derive what to our knowledge are the first statistical properties of this solution. It is shown that the Rayleigh-quotient-form TLS (RQF-TLS) estimator is equivalent to the maximum-likelihood estimator when noise terms in both data and operator elements are independent and identically distributed Gaussian. A perturbation analysis of the RQF-TLS solution is derived, and from it the mean square error of the RQF-TLS solution is obtained in closed form, which is valid at small noise levels. We then present a wavelet-based multiresolution scheme for obtaining the TLS solution. This method was employed with a multigrid algorithm to solve the linear perturbation equation encountered in optical tomography. Results from numerical simulations show that this method requires substantially less computation than the previously reported one-grid TLS algorithm. The method also allows one to identify regions of interest quickly from a coarse-level reconstruction and restrict the reconstruction in the following fine resolutions to those regions. Finally, the method is less sensitive to noise than the one-grid TLS and multigrid least-squares algorithms.

To improve availability and performance of fuel cells, the operating temperature of molten carbonate fuel cells (MCFC) stack should be strictly maintained within a specified operation range, and an efficient control technique should be employed to meet this objective. While most modern control strategies are based on process models, many existing models of MCFC are not ready to be applied in synthesis and operation of control systems. In this study, we developed an auto-regressive moving average (ARMA) model and machine learning methods of least squares support vector machine (LS-SVM), artificial neural network (ANN) and partial least squares (PLS) for the MCFC system based on input-output operating data. The ARMA model showed the best tracking performance. A model predictive control method for the operation of MCFC system was developed based on the proposed ARMA model. The control performance of the proposed MPC methods was compared with that of conventional controllers using numerical simulations performed on various process models including an MCFC process. Numerical results show that ARMA model based control provides improved control performance compared to other control methods.

The restricted total least squares (RTLS) problem, presented in this paper, is devised for solving overdetermined sets of linear equations $AX \approx B$ in which the data $[ A;B ]$ are perturbed by errors of the form $E^ * = DEC$. D and C are known matrices and E is an arbitrary but bounded matrix. By choosing D and C appropriately, the RTLS problem formulation can handle any weighted least squares (LS), generalized LS, total LS, and generalized total LS problem. Also, equality constraints can be imposed. In order to solve these problems, a computationally efficient and numerically reliable restricted TLS algorithm, based on the restricted singular value decomposition (RSVD), of the matrix triplet $( [ A;B ],D,C )$, is developed. This RSVD is a generalization of the ordinary SVD for triple matrix products. The matrices involved may be rank-deficient and the explicit formation of matrix inverses and products is avoided. Using the RSVD, some properties of the RTLS problem are proven.