Sub-optimal adaptive matched-field processing of a three-dimensional array in a changing environment

Summary The primary purpose of this paper is to propose empirical measures of the structural differences between two communities of plants or animals composed of the same species. Structure is defined to consist of; 1) the species in the community, 2) the pattern of interactions as represented by the covariance or correlation matrix of successive observations on each species, and 3) the mean abundances of each species in each of the two communities. Statistical tests are proposed for testing whether the covariance matrices and the vectors of mean densities for each community are equal and empirical measures of the differences between the covariance matrices and mean vectors are proposed. Given unequal covariance or correlation matrices the factor analysis model is used to derive empirical measures of the degree to which each variable of the ecosystem is responsible for the observed defferences in the pattern of interactions in each community. These tests and measures were applied to data gathered by Hunter (1966) on the abundances of six species of Drosophila censused monthly over a period of approximately two and a half years in two adjacent, but different habitats near Bogota, colombia. The two covariance matrices were significantly different indicating different patterns of interactions in the two Drosophila communities.

Efficient estimation of approximate factor models via penalized maximum likelihood

Estimating large covariance and precision (inverse covariance) matrices has become increasingly important in high dimensional statistics because of its wide applications. The estimation problem is challenging not only theoretically due to the constraint of its positive definiteness, but also computationally because of the curse of dimensionality. Many types of estimators have been proposed such as thresholding under the sparsity assumption of the target matrix, banding and tapering the sample covariance matrix. However, these estimators are not always guaranteed to be positive-definite, especially, for finite samples, and the sparsity assumption is rather restrictive. We propose a novel two-stage adaptive method based on the Cholesky decomposition of a general covariance matrix. By banding the precision matrix in the first stage and adapting the estimates to the second stage estimation, we develop a computationally efficient and statistically accurate method for estimating high dimensional precision matrices. We demonstrate the finite-sample performance of the proposed method by simulations from autoregressive, moving average, and long-range dependent processes. We illustrate its wide applicability by analyzing financial data such S&P 500 index and IBM stock returns, and electric power consumption of individual households. The theoretical properties of the proposed method are also investigated within a large class of covariance matrices.

The performance of the conventional Gram-Schmidt orthogonalization of covariance matrix (RGS) beamforming method will decrease significantly when non-ideal factors exists such as the appearances of fast moving interferences, array platform movement. In order to improve the robustness of interference suppression, a derivative constrained Gram-Schmidt orthogonalization of covariance matrix (CRGS) beamforming method with widened nulls is proposed in this paper. In the proposed method, the number of interference P is initially estimated and the first subspace is reconstructed by Gram-Schmidt orthogonalization of the first P columns of sample covariance matrix. Afterwards, the derivative constrained vectors and the second subspace spanned of these vectors are constructed. At last the adaptive weight vector is obtained by orthogonally projecting the quiescent weight vector onto the interference subspace made up of the first subspace and the second subspace. Based on the numeral simulation results, it is verified that the proposed method can form widened nulls and effectively improve the robustness of interference suppression.

A mainlobe interference suppression method is proposed based on eigen-projection processing and covariance matrix reconstruction. In the proposed method, the eigen-projection matrix is calculated based on the eigenvector of mainlobe interference to suppress the mainlobe interference in received echo data. The interference covariance matrix is reconstructed after the eigenvalue of mainlobe interference is replaced by the average value of noise eigenvalues to eliminate the effect of mainlobe interference to adaptive weight vector. Finally, the output of adaptive beamforming is calculated by processed echo data and adaptive weight vector obtained based on reconstructed covariance matrix. Simulation results show that the proposed method can suppress the mainlobe interference effectively and achieve excellent performance .

The smoothness of subdiagonals of the Cholesky factor of large covariance matrices is closely related to the degree of nonstationarity of autoregressive models for time series data. Heuristically, one expects for nearly stationary covariance matrix entries in each subdiagonal of the Cholesky factor of its inverse to be approximately the same in the sense that the sum of the absolute values of successive differences is small. Statistically, such smoothness is achieved by regularizing each subdiagonal using fused-type lasso penalties. We rely on the standard Cholesky factor as the new parameter within a regularized normal likelihood setup which guarantees: (a) joint convexity of the likelihood function, (b) strict convexity of the likelihood function restricted to each subdiagonal even when n < p, and (c) positive-definiteness of the estimated covariance matrix. A block coordinate descent algorithm, where each block is a subdiagonal, is proposed, and its convergence is established under mild conditions. Lack of decoupling of the penalized likelihood function into a sum of functions involving individual subdiagonals gives rise to some computational challenges and advantages relative to two recent algorithms for sparse estimation of the Cholesky factor, which decouple row-wise. Simulation results and real data analysis show the scope and good performance of the proposed methodology. Software for our method is freely available in R language. Supplementary materials for this article are available online.

Amainlobe interference suppression method is proposed in this paper, which can still work when the signal of interest (SOI) is present in the training data. In this method, the iterative adaptive approach (IAA) is applied to spatial spectrum estimation at first. Then, IAA spatial spectrum is used to reconstruct the interference-plus-noise covariance matrix (INCM). Next, the eigenvector associated with mainlobe interference in INCM is determined and the eigen-projection matrix can be calculated to suppress the mainlobe interference. Meanwhile, the sidelobe-interference-plus-noise covariance matrix (SINCM) can be reconstructed. Finally, the adaptive weight vector is obtained. One main advantage is that the proposed method can deal with coherent mainlobe interference and sidelobe interferences simultaneously. Simulation results demonstrate the effectiveness of the proposed method.

Covariance matrices play an important role in many multivariate techniques and hence a good covariance estimation is crucial in this kind of analysis. In many applications a sparse covariance matrix is expected due to the nature of the data or for simple interpretation. Hard thresholding, soft thresholding, and generalized thresholding were therefore developed to this end. However, these estimators do not always yield well-conditioned covariance estimates. To have sparse and well-conditioned estimates, we propose doubly shrinkage estimators: shrinking small covariances towards zero and then shrinking covariance matrix towards a diagonal matrix. Additionally, a richness index is defined to evaluate how rich a covariance matrix is. According to our simulations, the richness index serves as a good indicator to choose relevant covariance estimator.

Upcoming and existing large-scale surveys of galaxies require accurate theoretical predictions of the dark matter clustering statistics for thousands of mock galaxy catalogs. We demonstrate that this goal can be achieve with our new Parallel Particle-Mesh (PM) Nbody code (PPM-GLAM) at a very low computational cost. We run about 15,000 simulations with ~2 billion particles that provide ~1% accuracy of the dark matter power spectra P(k) for wave-numbers up to k~ 1h/Mpc. Using this large data-set we study the power spectrum covariance matrix, the stepping stone for producing mock catalogs. In contrast to many previous analytical and numerical results, we find that the covariance matrix normalised to the power spectrum C(k,k')/P(k)P(k') has a complex structure of non-diagonal components. It has an upturn at small k, followed by a minimum at k=0.1-0.2h/Mpc. It also has a maximum at k=0.5-0.6h/Mpc. The normalised covariance matrix strongly evolves with redshift: C(k,k')~delta(t)^alpha P(k)P(k'), where delta is the linear growth factor and alpha ~ 1-1.25, which indicates that the covariance matrix depends on cosmological parameters. We also show that waves longer than 1Gpc have very little impact on the power spectrum and covariance matrix. This significantly reduces the computational costs and complexity of theoretical predictions: relatively small volume ~ (1Gpc)^3 simulations capture the necessary properties of dark matter clustering statistics. All the power spectra obtained from many thousands of our simulations are publicly available.

Estimation of the high-dimensional banded covariance matrix is widely used in multivariate statistical analysis. To ensure the validity of estimation, we aim to test the hypothesis that the covariance matrix is banded with a certain bandwidth under the high-dimensional framework. Though several testing methods have been proposed in the literature, the existing tests are only powerful for some alternatives with certain sparsity levels, whereas they may not be powerful for alternatives with other sparsity structures. The goal of this paper is to propose a new test for the bandedness of high-dimensional covariance matrix, which is powerful for alternatives with various sparsity levels. The proposed new test also be used for testing the banded structure of covariance matrices of error vectors in high-dimensional factor models. Based on these statistics, a consistent bandwidth estimator is also introduced for a banded high dimensional covariance matrix. Extensive simulation studies and an application to a prostate cancer dataset from protein mass spectroscopy are conducted for evaluating the effectiveness of the proposed adaptive tests blue and bandwidth estimator for the banded covariance matrix.

High dimensional covariance matrices have attracted much attention of statisticians and econometricians during the past decades. Vast literature is devoted to the research in high dimensional covariance matrices. However, most of them are for constant covariance matrices. In many applications, constant covariance matrices are not appropriate, e.g. in portfolio allocation, dynamic covariance matrices would make much more sense. Simply assuming each entry of a covariance matrix is a function of time to introduce a dynamic structure would not work. In this paper, we are going to introduce a class of high dimensional dynamic covariance matrices in which a kind of additive structure is embedded. We will show the proposed high dimensional dynamic covariance matrices have many advantages in applications. An estimation procedure is also proposed to estimate the proposed high dimensional dynamic covariance matrices. Asymptotic properties are built to justify the proposed estimation procedure. Intensive simulation studies show the proposed estimation procedure works very well when sample size is finite. Finally, we apply the proposed high dimensional dynamic covariance matrices, together with the proposed estimation procedure, to portfolio allocation. The results look very interesting.

Journal Article Nonparametric estimation of large covariance matrices of longitudinal data Get access Wei Biao Wu, Wei Biao Wu Search for other works by this author on: Oxford Academic Google Scholar Mohsen Pourahmadi Mohsen Pourahmadi Search for other works by this author on: Oxford Academic Google Scholar Biometrika, Volume 90, Issue 4, December 2003, Pages 831–844, https://doi.org/10.1093/biomet/90.4.831 Published: 01 December 2003

The adaptive beamforming is one of the key techniques for array signal processing. However, the matrix inversion operation in the existing methods will cost a large amount of computational complexity, which results in poor real-time processing ability. In order to reduce the amount of computational cost, a fast and robust adaptive beamforming method based on complex-valued radial basis function (CRBF) neural network is proposed. In the proposed method, the CRBF neural network is established, thus the direct matrix inversion is avoided by the nonlinear mapping processing from the array covariance matrix to the adaptive weight vector, and therefore the calculation speed of adaptive weight vectors is increased. Based on the simulation results, the proposed method is verified that the speed of adaptive beamforming is increased compared with sample matrix inversion (SMI) algorithm method and an improved performance is achieved compared with that of conventional real-valued RBF neural network beamformer.

A Low-complexity Minimum-variance Beamformer Based on Orthogonal Decomposition of the Compounded Subspace.

Inferences in panel data with interactive effects using large covariance matrices