Spectral characterizations of structured big data covariance matrices

Summary The primary purpose of this paper is to propose empirical measures of the structural differences between two communities of plants or animals composed of the same species. Structure is defined to consist of; 1) the species in the community, 2) the pattern of interactions as represented by the covariance or correlation matrix of successive observations on each species, and 3) the mean abundances of each species in each of the two communities. Statistical tests are proposed for testing whether the covariance matrices and the vectors of mean densities for each community are equal and empirical measures of the differences between the covariance matrices and mean vectors are proposed. Given unequal covariance or correlation matrices the factor analysis model is used to derive empirical measures of the degree to which each variable of the ecosystem is responsible for the observed defferences in the pattern of interactions in each community. These tests and measures were applied to data gathered by Hunter (1966) on the abundances of six species of Drosophila censused monthly over a period of approximately two and a half years in two adjacent, but different habitats near Bogota, colombia. The two covariance matrices were significantly different indicating different patterns of interactions in the two Drosophila communities.

Estimating large covariance and precision (inverse covariance) matrices has become increasingly important in high dimensional statistics because of its wide applications. The estimation problem is challenging not only theoretically due to the constraint of its positive definiteness, but also computationally because of the curse of dimensionality. Many types of estimators have been proposed such as thresholding under the sparsity assumption of the target matrix, banding and tapering the sample covariance matrix. However, these estimators are not always guaranteed to be positive-definite, especially, for finite samples, and the sparsity assumption is rather restrictive. We propose a novel two-stage adaptive method based on the Cholesky decomposition of a general covariance matrix. By banding the precision matrix in the first stage and adapting the estimates to the second stage estimation, we develop a computationally efficient and statistically accurate method for estimating high dimensional precision matrices. We demonstrate the finite-sample performance of the proposed method by simulations from autoregressive, moving average, and long-range dependent processes. We illustrate its wide applicability by analyzing financial data such S&P 500 index and IBM stock returns, and electric power consumption of individual households. The theoretical properties of the proposed method are also investigated within a large class of covariance matrices.

We examined the ability of eigenvalue tests to distinguish field—collected from random, assemblage structure data sets. Eight published time series of species abundances were used in the analysis, including data sets for: fishes, birds, mammals, stream benthos, and crabs. To test the efficacy of eigenvalue tests, we constructed 1000 randomly generated data sets for each real data set, whose means and variances were identical to the means and variances of the original data matrices. The data sets were then subjected to a principal components analysis (PCA) and eigenvalue tests used to identify significant eigenvalues for both correlation and covariance matrix solutions. We also examined the effects of: (1) number of species (= number of variables), (2) number of samples (= replication), (3) variance structure, on the performance of the test. Using PCA's based on the correlation matrix and with sample sizes typically encountered in the field, the eigenvalue tests generally performed at the .05 level when a = .01. Slightly poorer results were obtained with the covariance matrix. Increasing the number of samples to at least three times the number of species generally gave a level coverage of an a level test (i.e., a = .05, .01). Increasing variance in the data set only affected test outcomes at levels of replication less than twice the number of species. We conclude that the eigenvalue tests can be used to detect patterns in PCA's of assemblage structure data, if the number of samples is at least three times the number of species and either a covariance or correlation matrix solution is used. It is assumed that these patterns represent ecologically meaningful patterns of variation.

We study the eigenvalue distribution of random matrices pertinent to the analysis of deep neural networks. The matrices resemble the product of the sample covariance matrices, however, an important difference is that the analog of the population covariance matrix is now a function of random data matrices (synaptic weight matrices in the deep neural network terminology). The problem has been treated in recent work [J. Pennington, S. Schoenholz and S. Ganguli, The emergence of spectral universality in deep networks, Proc. Mach. Learn. Res. 84 (2018) 1924–1932, arXiv:1802.09979] by using the techniques of free probability theory. Since, however, free probability theory deals with population covariance matrices which are independent of the data matrices, its applicability in this case has to be justified. The justification has been given in [L. Pastur, On random matrices arising in deep neural networks: Gaussian case, Pure Appl. Funct. Anal. (2020), in press, arXiv:2001.06188] for Gaussian data matrices with independent entries, a standard analytical model of free probability, by using a version of the techniques of random matrix theory. In this paper, we use another version of the techniques to extend the results of [L. Pastur, On random matrices arising in deep neural networks: Gaussian case, Pure Appl. Funct. Anal. (2020), in press, arXiv:2001.06188] to the case where the entries of the data matrices are just independent identically distributed random variables with zero mean and finite fourth moment. This, in particular, justifies the mean field approximation in the infinite width limit for the deep untrained neural networks and the property of the macroscopic universality of random matrix theory in this case.

One of the major challenges in multivariate analysis is the estimation of population covariance matrix from the sample covariance matrix (SCM). Most recent covariance matrix estimators use either shrinkage transformations or asymptotic results from Random Matrix Theory (RMT). Both of these techniques try to achieve a similar goal which is to remove noisy correlations and add structure to SCM to overcome the bias-variance trade-off. Both methods have their respective pros and cons. In this paper, we propose an improved estimator which exploits the advantages of these techniques by taking optimally weighted convex combination of covariance matrices estimated by shrinkage transformation and a filter based on RMT. It is a generalized estimator which can adapt to changing sampling noise conditions by performing hyperparameter optimization. Using data from six of the world's biggest stock exchanges, we show that the proposed estimator outperforms the existing estimators in minimizing the out-of-sample risk of the portfolio and hence predicts population statistics more precisely. The proposed estimator can be useful in a wide range of machine learning and signal processing applications. © 1994-2012 IEEE.

Many contemporary studies involve the classification of a subject into two classes based on n observations of the p variables associated with the subject. Under the assumption that the variables are normally distributed, the well-known linear discriminant analysis (LDA) assumes a common covariance matrix over the two classes while the quadratic discriminant analysis (QDA) allows different covariance matrices. When p is much smaller than n, even if they both diverge, the LDA and QDA have the smallest asymptotic misclassification rates for the cases of equal and unequal covariance matrices, respectively. However, modern statistical studies often face classification problems with the number of variables much larger than the sample size n, and the classical LDA and QDA can perform poorly. In fact, we give an example in which the QDA performs as poorly as random guessing even if we know the true covariances. Under some sparsity conditions on the unknown means and covariance matrices of the two classes, we propose a sparse QDA based on thresholding that has the smallest asymptotic misclassification rate conditional on the training data. We discuss an example of classifying normal and tumor colon tissues based on a set of p = 2; 000 genes and a sample of size n = 62, and another example of a cardiovascular study for n = 222 subjects with p = 2; 434 genes. A simulation is also conducted to check the performance of the proposed method.

Suppose that X is a p-variate random vector, measured in two populations, with mean vector μ j and covariance matrix ψ j in population j,j = 1, 2. In this article we study normal theory estimation of the parameters under the following constraints. It is assumed that (a) the covariance matrices have p identical eigenvectors, (b) in the coordinate system given by the common eigenvectors, only q < p of the means are different between groups, and (c) only the q eigenvalues associated with the same q common eigenvectors are different between groups. There are two main areas of application of these models: (a) in morphometric studies, particularly if size differences between groups are to be removed from the analysis, and (b) in testing for equality of mean vectors and covariance matrices, especially in situations where the number of variables p is large. For applications of type (b), we suggest a randomization test and compare its performance to the T 2 test and to the likelihood ratio test for equality of both populations.

This paper discusses the effects of introducing nonlinear interactions and noise-filtering to the covariance matrix used in Markowitz’s portfolio allocation model, evaluating the technique’s performances for daily data from seven financial markets between January 2000 and August 2018. We estimated the covariance matrix by applying Kernel functions, and applied filtering following the theoretical distribution of the eigenvalues based on the Random Matrix Theory. The results were compared with the traditional linear Pearson estimator and robust estimation methods for covariance matrices. The results showed that noise-filtering yielded portfolios with significantly larger risk-adjusted profitability than its non-filtered counterpart for almost half of the tested cases. Moreover, we analyzed the improvements and setbacks of the nonlinear approaches over linear ones, discussing in which circumstances the additional complexity of nonlinear features seemed to predominantly add more noise or predictive performance.

Sparse Bayesian factor models are routinely implemented for parsimonious dependence modeling and dimensionality reduction in high-dimensional applications. We provide theoretical understanding of such Bayesian procedures in terms of posterior convergence rates in inferring high-dimensional covariance matrices where the dimension can be larger than the sample size. Under relevant sparsity assumptions on the true covariance matrix, we show that commonly-used point mass mixture priors on the factor loadings lead to consistent estimation in the operator norm even when $p\gg n$. One of our major contributions is to develop a new class of continuous shrinkage priors and provide insights into their concentration around sparse vectors. Using such priors for the factor loadings, we obtain similar rate of convergence as obtained with point mass mixture priors. To obtain the convergence rates, we construct test functions to separate points in the space of high-dimensional covariance matrices using insights from random matrix theory; the tools developed may be of independent interest. We also derive minimax rates and show that the Bayesian posterior rates of convergence coincide with the minimax rates upto a $\sqrt{\log n}$ term.

Since the advent of Cognitive Radio and exploitation of unused spectrum by secondary devices, spectrum has gained immense importance. In cognitive radio network setting, due to very low received signal to noise ratio (SNR), spectrum sensing has been a challenging task. Further, signal detection is obscured due to presence of accumulated interference for which the statistical distribution is unknown and has significant distributional uncertainty. Covariance based signal detection methods have been proposed by various authors and proved to be better than other methods due to its independence on SNR, accuracy, and medium algorithmic complexity. Some results from Random Matrix Theory (RMT) on modelling of covariance matrices can be used in signal detection. This paper presents an algorithm for detection of Chi-Square distributed independent random signals using Wishart matrix which model the covariance matrix. A centering matrix is employed to scale the sample covariance matrix with Wishart distribution. The signals under various fading environment are used to evaluate the performance of the proposed algorithm. Also, a DVBT-2K signal is used to evaluate the performance for more realistic conditions. Furthermore, quantification of distributional uncertainty of accumulated interference in DTV based CRN is calculated using differential entropy of pdf of the interference. The quantified uncertainty is used to calculate the new threshold for detection. As per our knowledge, it is observed in the proposed work that the Maximum-Minimum Eigenvalue (MME) based proposed algorithm performs better than other proposed schemes in terms of probability of detection and complexity.

Modelling and forecasting covariance matrices of asset returns play a crucial role in many financial fields, such as portfolio allocation and asset pricing. The availability of high-frequency intraday data enables the modelling of the realized covariance matrix directly. However, most models in the literature suffer from the curse of dimensionality, i.e. the number of parameters needed increases at the rate of the square of the number of assets. To solve the problem, we propose a factor model with a diagonal Conditional Autoregressive Wishart model for the factor realized covariance matrices. Consequently, the positive definiteness of the estimated covariance matrix is ensured with the proposed model. Asymptotic theory is derived for the estimated parameters. In the extensive empirical analysis, we find that the number of parameters can be reduced significantly; to only about one-tenth of the benchmark model. Furthermore, the proposed model maintains a comparable performance with a benchmark vector autoregressive model for different forecast horizons.

Upcoming and existing large-scale surveys of galaxies require accurate theoretical predictions of the dark matter clustering statistics for thousands of mock galaxy catalogs. We demonstrate that this goal can be achieve with our new Parallel Particle-Mesh (PM) Nbody code (PPM-GLAM) at a very low computational cost. We run about 15,000 simulations with ~2 billion particles that provide ~1% accuracy of the dark matter power spectra P(k) for wave-numbers up to k~ 1h/Mpc. Using this large data-set we study the power spectrum covariance matrix, the stepping stone for producing mock catalogs. In contrast to many previous analytical and numerical results, we find that the covariance matrix normalised to the power spectrum C(k,k')/P(k)P(k') has a complex structure of non-diagonal components. It has an upturn at small k, followed by a minimum at k=0.1-0.2h/Mpc. It also has a maximum at k=0.5-0.6h/Mpc. The normalised covariance matrix strongly evolves with redshift: C(k,k')~delta(t)^alpha P(k)P(k'), where delta is the linear growth factor and alpha ~ 1-1.25, which indicates that the covariance matrix depends on cosmological parameters. We also show that waves longer than 1Gpc have very little impact on the power spectrum and covariance matrix. This significantly reduces the computational costs and complexity of theoretical predictions: relatively small volume ~ (1Gpc)^3 simulations capture the necessary properties of dark matter clustering statistics. All the power spectra obtained from many thousands of our simulations are publicly available.

Estimation of the high-dimensional banded covariance matrix is widely used in multivariate statistical analysis. To ensure the validity of estimation, we aim to test the hypothesis that the covariance matrix is banded with a certain bandwidth under the high-dimensional framework. Though several testing methods have been proposed in the literature, the existing tests are only powerful for some alternatives with certain sparsity levels, whereas they may not be powerful for alternatives with other sparsity structures. The goal of this paper is to propose a new test for the bandedness of high-dimensional covariance matrix, which is powerful for alternatives with various sparsity levels. The proposed new test also be used for testing the banded structure of covariance matrices of error vectors in high-dimensional factor models. Based on these statistics, a consistent bandwidth estimator is also introduced for a banded high dimensional covariance matrix. Extensive simulation studies and an application to a prostate cancer dataset from protein mass spectroscopy are conducted for evaluating the effectiveness of the proposed adaptive tests blue and bandwidth estimator for the banded covariance matrix.

High dimensional covariance matrices have attracted much attention of statisticians and econometricians during the past decades. Vast literature is devoted to the research in high dimensional covariance matrices. However, most of them are for constant covariance matrices. In many applications, constant covariance matrices are not appropriate, e.g. in portfolio allocation, dynamic covariance matrices would make much more sense. Simply assuming each entry of a covariance matrix is a function of time to introduce a dynamic structure would not work. In this paper, we are going to introduce a class of high dimensional dynamic covariance matrices in which a kind of additive structure is embedded. We will show the proposed high dimensional dynamic covariance matrices have many advantages in applications. An estimation procedure is also proposed to estimate the proposed high dimensional dynamic covariance matrices. Asymptotic properties are built to justify the proposed estimation procedure. Intensive simulation studies show the proposed estimation procedure works very well when sample size is finite. Finally, we apply the proposed high dimensional dynamic covariance matrices, together with the proposed estimation procedure, to portfolio allocation. The results look very interesting.

The major problem in adaptively processing data from a large array is inversion of the covariance matrix. The matrix must be well-conditioned to be inverted. The number of independent samples that is needed to form a well-conditioned matrix is at least twice the number of array elements. The integration time to form a well-conditioned large array covariance matrix may then be unreasonably long relative to the coherent time of signal in a changing environment. A sub-optimal subarray processing has been developed. The adaptive weight vectors are calculated for each subarray. These weight vectors are linearly combined to form a full-array adaptive weight vector and are applied to the full-array covariance matrix. The required integration time is reduced to that which is needed to form the well-conditioned subarray covariance matrices. In a recent shallow water experiment, a three-dimensional array had been deployed. The array consisted of 5 VLAs each with 30-element spaced evenly on a circular baseline of a radius of 200 m. The subarray processing approach had been used to localize a moving source that had a fast range/bearing change relative to the baseline.