Optimal Shrinkage Covariance Matrix Estimation Under Random Sampling From Elliptical Distributions

Estimation of the high-dimensional banded covariance matrix is widely used in multivariate statistical analysis. To ensure the validity of estimation, we aim to test the hypothesis that the covariance matrix is banded with a certain bandwidth under the high-dimensional framework. Though several testing methods have been proposed in the literature, the existing tests are only powerful for some alternatives with certain sparsity levels, whereas they may not be powerful for alternatives with other sparsity structures. The goal of this paper is to propose a new test for the bandedness of high-dimensional covariance matrix, which is powerful for alternatives with various sparsity levels. The proposed new test also be used for testing the banded structure of covariance matrices of error vectors in high-dimensional factor models. Based on these statistics, a consistent bandwidth estimator is also introduced for a banded high dimensional covariance matrix. Extensive simulation studies and an application to a prostate cancer dataset from protein mass spectroscopy are conducted for evaluating the effectiveness of the proposed adaptive tests blue and bandwidth estimator for the banded covariance matrix.

The covariance matrix, which should be estimated from the data, plays an important role in many multivariate procedures, and its positive definiteness (PDness) is essential for the validity of the procedures. Recently, many regularized estimators have been proposed and shown to be consistent in estimating the true matrix and its support under various structural assumptions on the true covariance matrix. However, they are often not PD. In this paper, we propose a simple modification to make a regularized covariance matrix be PD while preserving its support and the convergence rate. We focus on the matrix $$\ell_{\infty }$$ norm error in covariance matrix estimation because it could allow us to bound the error in the downstream multivariate procedure relying on it. Our proposal in this paper is an extension of the fixed support positive-definite (FSPD) modification by Choi et al. (2019) from spectral and Frobenius norms to the matrix $$\ell_{\infty }$$ norm. Like the original FSPD, we consider a convex combination between the initial estimator (the regularized covariance matrix without PDness) and a given form of the diagonal matrix minimize the $$\ell_{\infty }$$ distance between the initial estimator and the convex combination, and find a closed-form expression for the modification. We apply the procedure to the minimum variance portfolio (MVP) optimization problem and show that the vector $$\ell_{\infty }$$ error in the estimation of the optimal portfolio weight is bounded by the matrix $$\ell _{\infty }$$ error of the plug-in covariance matrix estimator. We illustrate the MVP results with S&P 500 daily returns data from January 1978 to December 2014.

We introduce nonparametric regularization of the eigenvalues of a sample covariance matrix through splitting of the data (NERCOME), and prove that NERCOME enjoys asymptotic optimal nonlinear shrinkage of eigenvalues with respect to the Frobenius norm. One advantage of NERCOME is its computational speed when the dimension is not too large. We prove that NERCOME is positive definite almost surely, as long as the true covariance matrix is so, even when the dimension is larger than the sample size. With respect to the Stein’s loss function, the inverse of our estimator is asymptotically the optimal precision matrix estimator. Asymptotic efficiency loss is defined through comparison with an ideal estimator, which assumed the knowledge of the true covariance matrix. We show that the asymptotic efficiency loss of NERCOME is almost surely 0 with a suitable split location of the data. We also show that all the aforementioned optimality holds for data with a factor structure. Our method avoids the need to first estimate any unknowns from a factor model, and directly gives the covariance or precision matrix estimator, which can be useful when factor analysis is not the ultimate goal. We compare the performance of our estimators with other methods through extensive simulations and real data analysis.

Covariance matrix plays an important role in risk management, asset pricing, and portfolio allocation. Covariance matrix estimation becomes challenging when the dimensionality is comparable or much larger than the sample size. A widely used approach for reducing dimensionality is based on multi-factor models. Although it has been well studied and quite successful in many applications, the quality of the estimated covariance matrix is often degraded due to a nontrivial amount of missing data in the factor matrix for both technical and cost reasons. Since the factor matrix is only approximately low rank or even has full rank, existing matrix completion algorithms are not applicable. We consider a new matrix completion paradigm using the factor models directly and apply the alternating direction method of multipliers for the recovery. Numerical experiments show that the nuclear-norm matrix completion approaches are not suitable but our proposed models and algorithms are promising.

The estimation of portfolio value-at-risk (VaR) requires a good estimate of the covariance matrix. As it is well known that a sample covariance matrix based on some historical rolling window is noisy and is a poor estimate for the high-dimensional population covariance matrix, to estimate the conditional portfolio VaR we develop a framework using the dynamic conditional covariance model, within which various de-noising tools are employed for the estimation of the unconditional covariance target. Various de-noising treatments in our study include shrinkage methods, random matrix theory methods and regularization methods. We validate the model empirically by using various coverage tests and loss function measures and discover that the choice of de-noising treatments for the covariance target plays a critical role in measuring the accuracy of the dynamic portfolio VaR estimates. In our large-scale empirical evaluation of de-noising tools, the regularization methods seem to produce the poorest VaR estimates under various coverage tests and loss function measures, while the shrinkage methods and the random matrix theory methods produce comparable results.

Many applications require precise estimates of high-dimensional covariance matrices. The standard estimator is the sample covariance matrix, which is conceptually simple, fast to compute and has favorable properties in the limit of infinitely many observations. The picture changes when the dimensionality is of the same order as the number of observations. In such cases, the eigenvalues of the sample covariance matrix are highly biased, the condition number becomes large and the inversion of the matrix gets numerically unstable.
\nA number of alternative estimators are superior in the high-dimensional setting, which include as subcategories structured estimators, regularized estimators and spectrum correction methods. In this thesis I contribute to all three areas. 
\nIn the area of structured estimation, I focus on models with low intrinsic dimensionality. I analyze the bias in Factor Analysis, the state-of-the-art factor model and propose Directional Variance Adjustment (DVA) Factor Analysis, which reduces bias and yields improved estimates of the covariance matrix.
\nAnalytical shrinkage of Ledoit and Wolf (LW-Shrinkage) is the most popular regularized estimator. I contribute in three aspects: first, I provide a theoretical analysis of the behavior of LW-Shrinkage in the presence of pronounced eigendirections, a case of great practical relevance. I show that LW-Shrinkage does not perform well in this setting and propose aoc-Shrinkage which yields significant improvements. Second, I discuss the effect of autocorrelation on LW-Shrinkage and review the Sancetta-Estimator, an extension of LW-Shrinkage to autocorrelated data. I show that the Sancetta-Estimator is biased and propose a theoretically and empirically superior estimator with reduced bias. Third, I propose an extension of shrinkage to multiple shrinkage targets. Multi-Target Shrinkage is not restricted to covariance estimation and allows for many interesting applications which go beyond regularization, including transfer learning. I provide a detailed theoretical and empirical analysis.
\nSpectrum correction approaches the problem of covariance estimation by improving the estimates of the eigenvalues of the sample covariance matrix. I discuss the state-of-the-art approach, Nonlinear Shrinkage, and propose a cross-validation based covariance (CVC) estimator which yields competitive performance at increased numerical stability and greatly reduced complexity and computational cost. On all data sets considered, CVC is on par or superior in comparison to the regularized and structured estimators.
\nIn the last chapter, I conclude with a discussion of the advantages and disadvantages of all covariance estimators presented in this thesis and give situation-specific recommendations. In addition, the appendix contains a systematic analysis of Linear Discriminant Analysis as a model application, which sheds light on the interdependency between the generative model of the data and various covariance estimators.

Sparse Bayesian factor models are routinely implemented for parsimonious dependence modeling and dimensionality reduction in high-dimensional applications. We provide theoretical understanding of such Bayesian procedures in terms of posterior convergence rates in inferring high-dimensional covariance matrices where the dimension can be larger than the sample size. Under relevant sparsity assumptions on the true covariance matrix, we show that commonly-used point mass mixture priors on the factor loadings lead to consistent estimation in the operator norm even when $p\gg n$. One of our major contributions is to develop a new class of continuous shrinkage priors and provide insights into their concentration around sparse vectors. Using such priors for the factor loadings, we obtain similar rate of convergence as obtained with point mass mixture priors. To obtain the convergence rates, we construct test functions to separate points in the space of high-dimensional covariance matrices using insights from random matrix theory; the tools developed may be of independent interest. We also derive minimax rates and show that the Bayesian posterior rates of convergence coincide with the minimax rates upto a $\sqrt{\log n}$ term.

We first develop a reduced-rank minimum mean square error (MMSE) detector for direct-sequence (DS) code division multiple access (CDMA) by forcing the linear MMSE detector to lie in a signal subspace of a reduced dimension. While a reduced-rank MMSE detector has lower complexity, it cannot outperform the full-rank MMSE detector. We then concentrate on the blind reduced-rank MMSE detector which is obtained from an estimated covariance matrix. Our analysis and simulation results show that when the desired user's signal is in a low-dimensional subspace, there exists an optimal subspace so that the blind reduced-rank MMSE detector lying in this subspace has the best performance. By properly choosing a subspace, we guarantee that the optimal blind reduced-rank MMSE detector is obtained. An adaptive blind reduced-rank MMSE detector, based on a subspace tracking algorithm, is developed. The adaptive blind reduced-rank MMSE detector exhibits superior steady-state performance and fast convergence speed.

We demonstrate that by adopting the new class of "expected-likelihood" (EL) covariance matrix estimates, instead of the traditional maximum-likelihood (ML) estimates, we can significantly enhance adaptive detection performance. These new estimates are found by searching within the properly parameterized class of admissible covariance matrices for the one that produces the likelihood ratio (LR) that is "closest possible" to the LR generated by the true (exact) covariance matrix.

We consider large complex random sample covariance matrices obtained from “spiked populations,” that is, when the true covariance matrix is diagonal with all but finitely many eigenvalues equal to 1. We investigate the limiting behavior of the largest eigenvalues when the population and the sample sizes both become large. Under some conditions on moments of the sample distribution, we prove that the asymptotic fluctuations of the largest eigenvalues are the same as for a complex Gaussian sample with the same true covariance. The real setting is also considered.

Journal Article Nonparametric estimation of large covariance matrices of longitudinal data Get access Wei Biao Wu, Wei Biao Wu Search for other works by this author on: Oxford Academic Google Scholar Mohsen Pourahmadi Mohsen Pourahmadi Search for other works by this author on: Oxford Academic Google Scholar Biometrika, Volume 90, Issue 4, December 2003, Pages 831–844, https://doi.org/10.1093/biomet/90.4.831 Published: 01 December 2003

Covariance matrix estimation is one of the most important problems in statistics. To accommodate the complexity of modern datasets, it is desired to have estimation procedures that not only can incorporate the structural assumptions of covariance matrices, but are also robust to outliers from arbitrary sources. In this paper, we define a new concept called matrix depth and then propose a robust covariance matrix estimator by maximizing the empirical depth function. The proposed estimator is shown to achieve minimax optimal rate under Huber’s $\\varepsilon$-contamination model for estimating covariance/scatter matrices with various structures including bandedness and sparsity.

In radar target detection, the adaptive normalised matched filter (ANMF) detector is often used to detect moving targets in sea and land clutter environment. Its performance depends on estimation of clutter covariance matrix. The sample covariance matrix (SCM) estimator has been thoroughly analysed for independent and identically distributed Gaussian clutter vectors. Many radar systems on service use range-oversampling receivers. Referred to the range-oversampling model in the weather and oceanic radars, the spatial correlation model of range-oversampled clutter vectors is established, the error of the SCM estimator is analysed, and the range-oversampling gain, relevant to the oversampling factor and the receiver's bandwidth, is derived. The experiments using raw radar clutter data are made to verify the range-oversampling gain, showing that the range oversampling improves the performance of the ANMF detector.

In this paper, we study the problem of high-dimensional covariance matrix estimation with missing observations. We propose a simple procedure computationally tractable in high-dimension and that does not require imputation of the missing data. We establish non-asymptotic sparsity oracle inequalities for the estimation of the covariance matrix involving the Frobenius and the spectral norms which are valid for any setting of the sample size, probability of a missing observation and the dimensionality of the covariance matrix. We further establish minimax lower bounds showing that our rates are minimax optimal up to a logarithmic factor.

Recently, in the context of covariance matrix estimation, in order to improve as well as to regularize the performance of the Tyler's estimator [1] also called the Fixed-Point Estimator (FPE) [2], a "shrinkage" fixed-point estimator has been introduced in [3]. First, this work extends the results of [3,4] by giving the general solution of the "shrinkage" fixed-point algorithm. Secondly, by analyzing this solution, called the generalized robust shrinkage estimator, we prove that this solution converges to a unique solution when the shrinkage parameter $\beta$ (losing factor) tends to 0. This solution is exactly the FPE with the trace of its inverse equal to the dimension of the problem. This general result allows one to give another interpretation of the FPE and more generally, on the Maximum Likelihood approach for covariance matrix estimation when constraints are added. Then, some simulations illustrate our theoretical results as well as the way to choose an optimal shrinkage factor. Finally, this work is applied to a Space-Time Adaptive Processing (STAP) detection problem on real STAP data.