Regularized covariance matrix estimation with high dimensional data for supervised anomaly detection problems

High dimensional covariance matrices have attracted much attention of statisticians and econometricians during the past decades. Vast literature is devoted to the research in high dimensional covariance matrices. However, most of them are for constant covariance matrices. In many applications, constant covariance matrices are not appropriate, e.g. in portfolio allocation, dynamic covariance matrices would make much more sense. Simply assuming each entry of a covariance matrix is a function of time to introduce a dynamic structure would not work. In this paper, we are going to introduce a class of high dimensional dynamic covariance matrices in which a kind of additive structure is embedded. We will show the proposed high dimensional dynamic covariance matrices have many advantages in applications. An estimation procedure is also proposed to estimate the proposed high dimensional dynamic covariance matrices. Asymptotic properties are built to justify the proposed estimation procedure. Intensive simulation studies show the proposed estimation procedure works very well when sample size is finite. Finally, we apply the proposed high dimensional dynamic covariance matrices, together with the proposed estimation procedure, to portfolio allocation. The results look very interesting.

Estimating large covariance and precision (inverse covariance) matrices has become increasingly important in high dimensional statistics because of its wide applications. The estimation problem is challenging not only theoretically due to the constraint of its positive definiteness, but also computationally because of the curse of dimensionality. Many types of estimators have been proposed such as thresholding under the sparsity assumption of the target matrix, banding and tapering the sample covariance matrix. However, these estimators are not always guaranteed to be positive-definite, especially, for finite samples, and the sparsity assumption is rather restrictive. We propose a novel two-stage adaptive method based on the Cholesky decomposition of a general covariance matrix. By banding the precision matrix in the first stage and adapting the estimates to the second stage estimation, we develop a computationally efficient and statistically accurate method for estimating high dimensional precision matrices. We demonstrate the finite-sample performance of the proposed method by simulations from autoregressive, moving average, and long-range dependent processes. We illustrate its wide applicability by analyzing financial data such S&P 500 index and IBM stock returns, and electric power consumption of individual households. The theoretical properties of the proposed method are also investigated within a large class of covariance matrices.

This paper develops an improved covariance matrix estimator in the mean-variance optimisation setting. Well-known problems with the sample covariance matrix are that it is singular when the number of observations is less than the number of assets, and can be nearly singular when the number of observations exceeds the number of assets. Therefore, using the sample covariance matrix as an input in mean-variance optimisation can result in unreasonable optimal portfolios and badly biased estimates of Sharpe ratios. We address this problem by imposing structure on the estimated covariance matrix by putting constraints on the Sharpe ratio, asset return variances, and the variance of the global minimum variance portfolio. We show that the constrained maximum likelihood estimator (CMLE) performs better than the sample covariance matrix. Moreover, when the shrinkage approach is applied to the CMLE and single index covariance matrix, it performs better than the shrinkage estimator of Ledoit and Wolf (2004).

Summary The primary purpose of this paper is to propose empirical measures of the structural differences between two communities of plants or animals composed of the same species. Structure is defined to consist of; 1) the species in the community, 2) the pattern of interactions as represented by the covariance or correlation matrix of successive observations on each species, and 3) the mean abundances of each species in each of the two communities. Statistical tests are proposed for testing whether the covariance matrices and the vectors of mean densities for each community are equal and empirical measures of the differences between the covariance matrices and mean vectors are proposed. Given unequal covariance or correlation matrices the factor analysis model is used to derive empirical measures of the degree to which each variable of the ecosystem is responsible for the observed defferences in the pattern of interactions in each community. These tests and measures were applied to data gathered by Hunter (1966) on the abundances of six species of Drosophila censused monthly over a period of approximately two and a half years in two adjacent, but different habitats near Bogota, colombia. The two covariance matrices were significantly different indicating different patterns of interactions in the two Drosophila communities.

Statistical inference for high-dimensional data is increasingly essential due to the prevalence of big data. While recent advancements have improved testing linear hypotheses for high-dimensional mean vectors, exploration of testing linear hypotheses for high-dimensional covariance matrices remains limited. This article focuses on a general linear hypothesis testing (GLHT) problem for high-dimensional covariance matrices, encompassing various specific cases such as testing the equality of variances, testing for a given covariance matrix, assessing the homogeneity of covariance matrices, and examining linear combinations of covariance matrices. We propose and study a new test statistic for the GLHT problem on high-dimensional covariance matrices. Under certain regularity conditions and the null hypothesis, we demonstrate that the test statistic shares the same limiting distribution as a Chi-squared-type mixture. This mixture’s distribution can be accurately approximated using a three-cumulant matched Chi-squared-approximation approach. Additionally, we establish the asymptotic power of the proposed test against some local alternatives. Simulation studies and a real-world financial data analysis demonstrate that the proposed test outperforms several competitors in terms of size control.

Estimation of the high-dimensional banded covariance matrix is widely used in multivariate statistical analysis. To ensure the validity of estimation, we aim to test the hypothesis that the covariance matrix is banded with a certain bandwidth under the high-dimensional framework. Though several testing methods have been proposed in the literature, the existing tests are only powerful for some alternatives with certain sparsity levels, whereas they may not be powerful for alternatives with other sparsity structures. The goal of this paper is to propose a new test for the bandedness of high-dimensional covariance matrix, which is powerful for alternatives with various sparsity levels. The proposed new test also be used for testing the banded structure of covariance matrices of error vectors in high-dimensional factor models. Based on these statistics, a consistent bandwidth estimator is also introduced for a banded high dimensional covariance matrix. Extensive simulation studies and an application to a prostate cancer dataset from protein mass spectroscopy are conducted for evaluating the effectiveness of the proposed adaptive tests blue and bandwidth estimator for the banded covariance matrix.

Detecting emergence of a low-rank signal from high-dimensional data is an important problem arising from many applications such as camera surveillance and swarm monitoring using sensors. We consider a procedure based on the largest eigenvalue of the sample covariance matrix over a sliding window to detect the change. To achieve dimensionality reduction, we present a sketching-based approach for rank change detection using the low-dimensional linear sketches of the original high-dimensional observations. The premise is that when the sketching matrix is a random Gaussian matrix, and the dimension of the sketching vector is sufficiently large, the rank of sample covariance matrix for these sketches equals the rank of the original sample covariance matrix with high probability. Hence, we may be able to detect the low-rank change using sample covariance matrices of the sketches without having to recover the original covariance matrix. We character the performance of the largest eigenvalue statistic in terms of the false-alarm-rate and the expected detection delay, and present an efficient online implementation via subspace tracking.

SummarySample covariance matrices play a central role in numerous popular statistical methodologies, for example principal components analysis, Kalman filtering and independent component analysis. However, modern random matrix theory indicates that, when the dimension of a random vector is not negligible with respect to the sample size, the sample covariance matrix demonstrates significant deviations from the underlying population covariance matrix. There is an urgent need to develop new estimation tools in such cases with high‐dimensional data to recover the characteristics of the population covariance matrix from the observed sample covariance matrix. We propose a novel solution to this problem based on the method of moments. When the parametric dimension of the population spectrum is finite and known, we prove that the proposed estimator is strongly consistent and asymptotically Gaussian. Otherwise, we combine the first estimation method with a cross‐validation procedure to select the unknown model dimension. Simulation experiments demonstrate the consistency of the proposed procedure. We also indicate possible extensions of the proposed estimator to the case where the population spectrum has a density.

Radar networks can provide spatial diversity by observing targets from different perspectives with adequately separated receiving stations. This article proposes a mainlobe interference suppression approach for radar networks based on covariance matrix reconstruction. First, we formulate the mainlobe interference suppression under multiradar configuration as a multichannel signal enhancement problem and perform range and Doppler equalization to improve interferences’ spatial correlation. Then, we estimate the signal and interference covariance matrix from the sampling covariance matrix by solving a robust principal component analysis (RPCA) model. After that, we devise a time-domain-constrained (TDC) linear estimator dependent on the reconstructed covariance matrices to enhance target echo. To remove those duplicate false targets caused by signal enhancement and estimate the state of the physical targets, we establish and solve a generalized multiple measurement vector (GMMV) model based on the fact that targets have unique locations in the state space. Finally, we confirm the effectiveness of the proposed approach with numerical experiments and demonstrate better performance in comparison with the subspace projection method.

Many applications require precise estimates of high-dimensional covariance matrices. The standard estimator is the sample covariance matrix, which is conceptually simple, fast to compute and has favorable properties in the limit of infinitely many observations. The picture changes when the dimensionality is of the same order as the number of observations. In such cases, the eigenvalues of the sample covariance matrix are highly biased, the condition number becomes large and the inversion of the matrix gets numerically unstable.
\nA number of alternative estimators are superior in the high-dimensional setting, which include as subcategories structured estimators, regularized estimators and spectrum correction methods. In this thesis I contribute to all three areas. 
\nIn the area of structured estimation, I focus on models with low intrinsic dimensionality. I analyze the bias in Factor Analysis, the state-of-the-art factor model and propose Directional Variance Adjustment (DVA) Factor Analysis, which reduces bias and yields improved estimates of the covariance matrix.
\nAnalytical shrinkage of Ledoit and Wolf (LW-Shrinkage) is the most popular regularized estimator. I contribute in three aspects: first, I provide a theoretical analysis of the behavior of LW-Shrinkage in the presence of pronounced eigendirections, a case of great practical relevance. I show that LW-Shrinkage does not perform well in this setting and propose aoc-Shrinkage which yields significant improvements. Second, I discuss the effect of autocorrelation on LW-Shrinkage and review the Sancetta-Estimator, an extension of LW-Shrinkage to autocorrelated data. I show that the Sancetta-Estimator is biased and propose a theoretically and empirically superior estimator with reduced bias. Third, I propose an extension of shrinkage to multiple shrinkage targets. Multi-Target Shrinkage is not restricted to covariance estimation and allows for many interesting applications which go beyond regularization, including transfer learning. I provide a detailed theoretical and empirical analysis.
\nSpectrum correction approaches the problem of covariance estimation by improving the estimates of the eigenvalues of the sample covariance matrix. I discuss the state-of-the-art approach, Nonlinear Shrinkage, and propose a cross-validation based covariance (CVC) estimator which yields competitive performance at increased numerical stability and greatly reduced complexity and computational cost. On all data sets considered, CVC is on par or superior in comparison to the regularized and structured estimators.
\nIn the last chapter, I conclude with a discussion of the advantages and disadvantages of all covariance estimators presented in this thesis and give situation-specific recommendations. In addition, the appendix contains a systematic analysis of Linear Discriminant Analysis as a model application, which sheds light on the interdependency between the generative model of the data and various covariance estimators.

A sample \( X_{1},\ldots ,X_{n} \) consisting of independent identically distributed random vectors in \( \mathbb {R}^{p} \) with zero mean and covariance matrix \( \mathbf {\Sigma }\) is considered. The recovery of spectral projectors of high-dimensional covariance matrices from a sample of observations is a key problem in statistics arising in numerous applications. This chapter describes a bootstrap procedure for constructing confidence sets for the spectral projector \( \mathbf {P}_{r} \) related to rth eigenvalue of the covariance matrix \(\mathbf {\Sigma }\) from given data on the base of corresponding spectral projector \(\widehat{\mathbf {P}}_{r}\) of the sample covariance matrix \(\widehat{\mathbf {\Sigma }}\). This approach does not use the asymptotical distribution of \( \Vert \mathbf {P}_{r} - \widehat{\mathbf {P}}_{r} \Vert _{2} \) and does not require the computation of its moment characteristics. The performance of the bootstrap approximation procedure is analyzed.

In this paper, we proposed a new testing statistic for testing the equality of mean vectors from two multivariate normal populations when the covariance matrices are unknown and unequal in high–dimensional data. A new test is proposed based on the idea of keeping more information from the sample covariance matrices as much as possible. A proposed test is invariant under scalar transformations and location shifts. We showed that the asymptotic distribution of proposed statistic is standard normal distribution when number of random variables approach infinity. We also compared the performance of the proposed test with other three existing tests by the simulation study. The simulation results showed that the attained significance level of proposed test close to setting nominal significance level satisfactorily. The attained power of proposed test outperforms as the other comparative tests under form of covariance matrices considered which can be arranged to block diagonal matrix structure. The attained power becomes more powerful when the dimension increases for a given sample size or vice versa, or relationship level between random variables in each sample increases. Finally, the proposed test is also illustrated with an analysis of DNA microarray data.

A robust algorithm for source number estimation based on the formation of the Hankel covariance matrix is presented. First, multiple data snapshots are taken successively from overlapped subarrays in a way similar to the forward spatial smoothing method to construct the special Hankel covariance matrix and for the total number of subarrays, these special covariance matrices are generated. Then, the average of these matrices is employed in singular value decomposition to generate the corresponding eigenvalues. Finally, the resulting eigenvalues are evaluated via the rule presented in this paper as the Moving Gradient Criterion (MGC) to estimate the number of sources by detection of the largest singular values. The greatest difference between the proposed algorithm and the other conventional methods is the form of the covariance matrix with the observed signal that can handle both non-coherent as well as fully coherent sources. Also, the proposed MGC rule adopted with this form of the covariance matrix is the strength of this work. Numerical simulations demonstrate the high superiority of the proposed approach over the competing methods such as MDL, AIC, SORTE, RAE and MSEE methods, especially in the cases of very closely spaced sources, low SNR values, low sensors number and low snapshots number.

Journal Article Nonparametric estimation of large covariance matrices of longitudinal data Get access Wei Biao Wu, Wei Biao Wu Search for other works by this author on: Oxford Academic Google Scholar Mohsen Pourahmadi Mohsen Pourahmadi Search for other works by this author on: Oxford Academic Google Scholar Biometrika, Volume 90, Issue 4, December 2003, Pages 831–844, https://doi.org/10.1093/biomet/90.4.831 Published: 01 December 2003

In this paper the model of experiment in which several characteristics are measured or the measurements are repeated in time, is considered. It is assumed that the covariance matrix of the characteristics/time points is unknown, but has the banded Toeplitz structure. The problem of estimating such a covariance structure was considered by e.g. Cui et al. (2016), Filipiak et al. (2018). We discuss two methods that aim to deal with singularity and numerical ill-conditioning of the estimators of covariance matrix under high-dimensional regime. The method of estimation of banded Toeplitz covariance matrix based on a shrinking, described by Ledoit and Wolf (2004) for unstructured covariance matrices is proposed. The estimators obtained with the use of the proposed method are then compared with the estimators obtained by the projection of sample covariance matrix onto the asymptotic cone of the nonnegative definite Toeplitz matrices (Filipiak et al. 2018). For this purpose simulation studies concerning bias and risk are conducted for several sets of parameters.