Maximum likelihood autoregressive model parameter estimation with noise corrupted independent snapshots

This paper presents a novel approach for estimating autoregressive (AR) model parameters using deep neural network (DNN) in the AR-Wiener filtering speech enhancement. Unlike conventional DNN that predicts one kind of target, the DNN used in this paper is trained to predict the AR model parameters of speech and noise simultaneously at offline stage. We train this network by minimizing the Euclidean distance between the output of DNN and the AR model parameters of clean speech and noise. At online stage, the acoustic features are first extracted from noisy speech as the input of the DNN. Then, AR model parameters of speech and noise are estimated by the DNN simultaneously. Finally, the Wiener filter is constructed by the AR model parameters of speech and noise. However, the AR model parameters only models the spectral shape not the spectral details, there are still some residual noise between the harmonics. In order to solve this problem, we introduce the speech-presence probability (SPP), that is, in the test stage, the SPP is estimated and is used to update the Wiener filter. The experimental results show that our approach has higher performance compared with some existing approaches.

In the framework of speech enhancement, several parametric approaches based on an a priori model for a speech signal have been proposed. When using an autoregressive (AR) model, three issues must be addressed. 1) How to deal with AR parameter estimation? Indeed, due to additive noise, the standard least squares criterion leads to biased estimates of AR parameters. 2) Can an estimation of the variance of the additive noise for each speech frame be obtained? A voice activity detector is often used for its estimation. 3) Which estimation rules and techniques (filtering, smoothing, etc.) can be considered to retrieve the speech signal? Our contribution in this paper is threefold. First, we propose to view the identification of the noisy AR process as an errors-in-variables problem. This blind method has the advantage of providing accurate estimations of both the AR parameters and the variance of the additive noise. Second, we propose an alternative algorithm to standard Kalman smoothing, based on a constrained minimum variance estimation procedure with a lower computational cost. Third, the combination of these two steps is investigated. It provides better results than some existing speech enhancement approaches in terms of signal-to-noise-ratio (SNR), segmental SNR, and informal subjective tests.

Fusion identification and estimation of multisensor multichannel AR signals with missing measurements and sensor biases

In a Fourier transform infrared (IR) spectrometer, the Michelson interference signal extrapolation method based on linear prediction is often used to improve spectral resolution. In this method, an autoregressive (AR) model is established for the Michelson interference signal in the spectrometer. Once the AR model parameters are determined, the AR process is predictable. The interference signal can be used to figure out the AR model's parameters. Based on this, the AR model can be used to extrapolate the interference signal to improve the spectral resolution. In this paper, the forward-backward linear prediction total least squares (FB-TLS) method is proposed to estimate the parameters of the AR model. The parameters that are estimated are used to improve the IR spectral resolution. By simulating different order and signal-to-noise ratio situations, the effects of the Burg, the least square, and the FB-TLS parameter estimation methods on spectral resolution enhancement are studied. The simulation results demonstrate that the FB-TLS parameter estimation method can effectively suppress noise and avoid spurious peaks. The experimental results demonstrate that the FB-TLS parameter estimation method is effective for spectral resolution enhancement technology based on linear prediction. When the FB-TLS method is used to enhance NH3 IR spectral resolution from 2 cm-1 to 1 cm-1, the spectral prediction error in the NH3 characteristic band is only 0.21% compared with the measured NH3 spectrum, whose spectral resolution is 1 cm-1.

Multi-rate Gaussian Bayesian network soft sensor development with noisy input and missing data

The best accuracy for estimated spectra is obtained with parsimonious time series models, which have the smallest number of parameters to guarantee unbiased models. Durbin 's method for the estimation of moving average (MA) parameters uses the estimated parameters of a long autoregressive (AR) model to calculate MA parameters. Probably all pejorative remarks on the quality of Durbin's method in the literature are based on suboptimal or wrong choices for the method of AR estimation or for the order of the intermediate AR model. That AR order should be considerably higher than the order of the best predicting AR model and it should grow with the sample size. Furthermore, the Burg estimates for the AR parameters give the best results because they have the smallest variance of all AR methods with a small bias. The triangular autocorrelation bias of the popular Yule-Walker method of AR estimation can cause large bias errors in finite samples, which makes it unsuited. Durbin's method applied to the proper number of AR parameters estimated with Burg's method outperforms all other known MA estimation methods, asymptotically as well as in finite samples. The accuracy is generally close to the Cramer-Rao bound.

This paper applies a new expectation maximization (EM) based identification method to estimate a generic FitzHugh-Nagumo (FHN) model under unknown Gaussian measurement noise. It is well noted that such FHN model is an elementary neuronal dynamics description and plays a significant role in deep understanding and basic analysis for complicated mechanisms of some neural diseases. Different from the most existing relevant identification methods, the applied new method is additionally capable of supplying users with variance estimation for the unknown measurement noise corrupting on the membrane potential apart from model parameter estimates. All unknown parameters can be iteratively estimated by a particle smoothing based EM algorithm, which consists of an expectation (E) step and a maximization (M) step. Smoothed joint-state particles are produced by a new particle smoothing algorithm to evaluate an expectation of a log-likelihood function in the E step, and the model parameters and noise variance can then be efficiently optimized by the gradient based methods in the M step. The resulting estimations own global convergence for a relatively wide range of parameter initializations. Finally, good convergence behaviors of the estimated model parameters and noise variance are demonstrated by a numerical simulation for a classic FHN model by using 10 simulation realizations with random initializations varying within ±100% of their respective true values.

Data assimilation is predominantly used for state estimation, combining observational data with model predictions to produce an updated model state that most accurately approximates the true system state whilst keeping the model parameters fixed. This updated model state is then used to initiate the next model forecast. Even with perfect initial data, inaccurate model parameters will lead to the growth of prediction errors. To generate reliable forecasts, we need good estimates of both the current system state and the model parameters. This article presents research into data assimilation methods for morphodynamic model state and parameter estimation. First, we focus on state estimation and describe implementation of a three‐dimensional variational (3D‐Var) data assimilation scheme in a simple 2D morphodynamic model of Morecambe Bay, UK. The assimilation of observations of bathymetry derived from synthetic aperture radar (SAR) satellite imagery and a ship‐borne survey is shown to significantly improve the predictive capability of the model over a 2‐year run. Here, the model parameters are set by manual calibration; this is laborious and is found to produce different parameter values depending on the type and coverage of the validation dataset. The second part of this article considers the problem of model parameter estimation in more detail. We explain how, by employing the technique of state augmentation, it is possible to use data assimilation to estimate uncertain model parameters concurrently with the model state. This approach removes inefficiencies associated with manual calibration and enables more effective use of observational data. We outline the development of a novel hybrid sequential 3D‐Var data assimilation algorithm for joint state‐parameter estimation and demonstrate its efficacy using an idealised 1D sediment transport model. The results of this study are extremely positive and suggest that there is great potential for the use of data assimilation‐based state‐parameter estimation in coastal morphodynamic modelling. Copyright © 2012 Royal Meteorological Society

Objective: Data being missed is an unavoidable problem in cohort studies. This paper compares the imputation effect of eight common missing data imputation methods involved in cutting longitudinal data through simulation study to provide a valuable reference for the treatment of missing data in longitudinal studies. Methods: The simulation study is based on R language software and generates missing longitudinal data by the Monte Carlo method. By comparing the average absolute deviation, average relative deviation, and TypeⅠerror from the regression analysis of different imputation methods, the imputation effect of varying imputation methods on missing longitudinal data and the influence on subsequent multivariate analysis are evaluated. Results: The mean imputation, k nearest neighbor (KNN), regression imputation, and random forest all have a similar imputation effect, which is also steady. However, the hot deck is inferior to the above imputation methods. K-means clustering and expectation maximization (EM) algorithm are among the worst and unstable. Mean imputation, EM algorithm, random forest, KNN, and regression imputation can control TypeⅠerror. Still, multiple imputations, hot deck, and K-means clustering cannot effectively manage the TypeⅠerror. Conclusions: For missing data in longitudinal studies, mean imputation, KNN, regression imputation, and random forest can be used as better imputation methods under the mechanism of missing at random. When the missing ratio is not too large, multiple imputations and hot deck can also perform well, but K-means clustering and EM algorithm are not recommended.

SummaryFor the multi‐sensor multi‐channel autoregressive (AR) moving average signals with white measurement noises and an AR‐colored measurement noise, a multi‐stage information fusion identification method is presented when model parameters and noise variances are partially unknown. The local estimators of model parameters and noise variances are obtained by the multidimensional recursive instrumental variable algorithm and correlation method, and the fused estimators are obtained by taking the average of the local estimators. They have the strong consistency. Substituting them into the optimal information fusion Kalman filter weighted by scalars, a self‐tuning fusion Kalman filter for multi‐channel AR moving average signals is presented. Applying the dynamic error system analysis method, it is proved that the proposed self‐tuning fusion Kalman filter converges to the optimal fusion Kalman filter in a realization, so that it has asymptotic optimality. A simulation example for a target tracking system with three sensors shows its effectiveness. Copyright © 2014 John Wiley & Sons, Ltd.

In this paper, we study the autoregressive (AR) model with normal inverse Gaussian (NIG) innovations. The NIG distribution is semi heavy-tailed and is helpful in capturing the extreme observations present in the data. The expectation-maximization (EM) algorithm is used to estimate the parameters of the considered AR(p) model. The efficacy of the estimation procedure is shown on the simulated data for AR(2) and AR(1) models. A comparative study is presented, where the classical estimation algorithms are also incorporated, namely, Yule-Walker and conditional least squares methods along with EM method for model parameter estimation. In simulation study, the maximum likelihood estimation (MLE) of NIG distribution by EM algorithm and iterative Newton-Raphson method are also compared. The real-life applications of the introduced model are demonstrated on the NASDAQ stock market index data and US gasoline price data. The studies show that AR(1) model with NIG residuals is good fit for financial data with extreme values as well as for gasoline price data.

An innovative Bayesian system identification method using autoregressive model

Unsupervised texture segmentation/classification using 2-D autoregressive modeling and the stochastic expectation-maximization algorithm

Three important parametric models for describing the correlation functions and spectra of stationary stochastic processes are the autoregressive (AR), moving average (MA), and autoregressive-moving average (ARMA) models. Quite recently, the MATLAB toolbox “ARMASA” has been made publicly available. This toolbox provides state-of-the-art algorithms to perform automatic identification and selection between the models based on the estimated prediction error. ARMASA works on a single segment of data, whereas in some applications, the data are available as multiple segments. We could process each segment independently and average the estimated autocorrelation functions or spectra afterward. Better performance, however, can be expected when all segments are processed simultaneously, for two reasons. Initially, the bias in the estimated model parameters depends on the number of observations in a segment. Averaging models from segments does not reduce this bias, in contrast to identification from all the data at once. In either case, the variance of the estimated model parameters will decrease. By considering all the data at once this variance reduction can be considered in the order selection stage, decreasing the selection bias. In this paper, we extend the parameter estimation algorithms of the ARMASA toolbox to the case of multiple segments. AR parameter estimation from segments has been considered before, but not for the MA and ARMA models. In addition, in the AR case, model order selection can be done easily when the segments are independent. But, in the application we are primarily interested in, the segments are formed by lines of scanning transmission electron microscopy (STEM) images, which can be correlated with each other. Because the correlation among data segments influences the effective number of independent observations available it becomes necessary to estimate that number. This is not an easy problem in general. For observations from a stochastic process with an isotropic 2-D autocorrelation function we can, however, formulate an accurate estimator in terms of the line autocorrelation function. This situation arises, for example, in STEM images of amorphous specimens. From the AR model identification for segments, the extension to the MA and ARMA models is rather straightforward because MA and ARMA algorithms are based on Durbin's methods, whose first step is the estimation of a long AR model. We compare the segment-based algorithms to conventional ARMASA with simulations of stationary stochastic processes. An illustrative example involving simulated STEM images shows large potential benefits.

Optimal filtering of complex turbulent systems with memory depth through consistency constraints