High dimensional covariance matrix estimation using multi-factor models from incomplete information

Many applications require precise estimates of high-dimensional covariance matrices. The standard estimator is the sample covariance matrix, which is conceptually simple, fast to compute and has favorable properties in the limit of infinitely many observations. The picture changes when the dimensionality is of the same order as the number of observations. In such cases, the eigenvalues of the sample covariance matrix are highly biased, the condition number becomes large and the inversion of the matrix gets numerically unstable.
\nA number of alternative estimators are superior in the high-dimensional setting, which include as subcategories structured estimators, regularized estimators and spectrum correction methods. In this thesis I contribute to all three areas. 
\nIn the area of structured estimation, I focus on models with low intrinsic dimensionality. I analyze the bias in Factor Analysis, the state-of-the-art factor model and propose Directional Variance Adjustment (DVA) Factor Analysis, which reduces bias and yields improved estimates of the covariance matrix.
\nAnalytical shrinkage of Ledoit and Wolf (LW-Shrinkage) is the most popular regularized estimator. I contribute in three aspects: first, I provide a theoretical analysis of the behavior of LW-Shrinkage in the presence of pronounced eigendirections, a case of great practical relevance. I show that LW-Shrinkage does not perform well in this setting and propose aoc-Shrinkage which yields significant improvements. Second, I discuss the effect of autocorrelation on LW-Shrinkage and review the Sancetta-Estimator, an extension of LW-Shrinkage to autocorrelated data. I show that the Sancetta-Estimator is biased and propose a theoretically and empirically superior estimator with reduced bias. Third, I propose an extension of shrinkage to multiple shrinkage targets. Multi-Target Shrinkage is not restricted to covariance estimation and allows for many interesting applications which go beyond regularization, including transfer learning. I provide a detailed theoretical and empirical analysis.
\nSpectrum correction approaches the problem of covariance estimation by improving the estimates of the eigenvalues of the sample covariance matrix. I discuss the state-of-the-art approach, Nonlinear Shrinkage, and propose a cross-validation based covariance (CVC) estimator which yields competitive performance at increased numerical stability and greatly reduced complexity and computational cost. On all data sets considered, CVC is on par or superior in comparison to the regularized and structured estimators.
\nIn the last chapter, I conclude with a discussion of the advantages and disadvantages of all covariance estimators presented in this thesis and give situation-specific recommendations. In addition, the appendix contains a systematic analysis of Linear Discriminant Analysis as a model application, which sheds light on the interdependency between the generative model of the data and various covariance estimators.

Accurate estimation of covariance matrix is the basis of effective development of high-dimensional optimal portfolio. Random matrix theory provides an effective means to improve the estimation of high-dimensional covariance matrix. Based on the empirical research on the export excess return rate of traditional cultural industry, the result is a combination with higher precision and lower risk. Based on relevant data, this paper constructs a stochastic matrix theoretical model to analyze the impact of human capital, new fixed asset investment, independent innovation and technology purchase, financing sources, and other factors on high-tech industry export. Based on the results of empirical analysis, some policy suggestions are put forward, such as increasing r&d investment, improving enterprises’ innovation ability, and introducing core scientific and technological talents to promote the export scale growth of high-tech industries. Model comparative analysis is made on the influence of traditional industries’ concentrated export, foreign trade environment, and cost advantage on the export of high-tech products. Through comparative analysis of the influencing factors of the export of high-tech products in the two periods before and after the export crisis of traditional culture, the new changes of the influencing factors of the export of high-tech products are analyzed. In addition, through the change of the internal and external environment of the current high-tech product export, the reasons for the new change of the influencing factors are discussed.

In this paper, we study the problem of high-dimensional covariance matrix estimation with missing observations. We propose a simple procedure computationally tractable in high-dimension and that does not require imputation of the missing data. We establish non-asymptotic sparsity oracle inequalities for the estimation of the covariance matrix involving the Frobenius and the spectral norms which are valid for any setting of the sample size, probability of a missing observation and the dimensionality of the covariance matrix. We further establish minimax lower bounds showing that our rates are minimax optimal up to a logarithmic factor.

Estimation of the high-dimensional banded covariance matrix is widely used in multivariate statistical analysis. To ensure the validity of estimation, we aim to test the hypothesis that the covariance matrix is banded with a certain bandwidth under the high-dimensional framework. Though several testing methods have been proposed in the literature, the existing tests are only powerful for some alternatives with certain sparsity levels, whereas they may not be powerful for alternatives with other sparsity structures. The goal of this paper is to propose a new test for the bandedness of high-dimensional covariance matrix, which is powerful for alternatives with various sparsity levels. The proposed new test also be used for testing the banded structure of covariance matrices of error vectors in high-dimensional factor models. Based on these statistics, a consistent bandwidth estimator is also introduced for a banded high dimensional covariance matrix. Extensive simulation studies and an application to a prostate cancer dataset from protein mass spectroscopy are conducted for evaluating the effectiveness of the proposed adaptive tests blue and bandwidth estimator for the banded covariance matrix.

Normally in Principal Component Analysis (PCA) (Sirovich & Kirby, 1987; Turk & Pentland, 1991), the 2D image matrices are firstly transformed to 1D image vectors by vectorization. The vectorization of a matrix is the column vector obtain by stacking the columns of the matrix on top of one another. The covariance or scatter matrix are formulated from the these image vectors. The covariance matrix will be well estimated if and only if the number of available training samples is not far smaller than the dimension of this matrix. In fact, it is too hard to collect this the number of samples. Then, normally in 1D subspace analysis, the estimated covariance matrix is not well estimated and not full rank.

Journal Article Nonparametric estimation of large covariance matrices of longitudinal data Get access Wei Biao Wu, Wei Biao Wu Search for other works by this author on: Oxford Academic Google Scholar Mohsen Pourahmadi Mohsen Pourahmadi Search for other works by this author on: Oxford Academic Google Scholar Biometrika, Volume 90, Issue 4, December 2003, Pages 831–844, https://doi.org/10.1093/biomet/90.4.831 Published: 01 December 2003

This paper considers the problem of estimating a high-dimensional (HD) covariance matrix when the sample size is smaller, or not much larger, than the dimensionality of the data, which could potentially be very large. We develop a regularized sample covariance matrix (RSCM) estimator which can be applied in commonly occurring sparse data problems. The proposed RSCM estimator is based on estimators of the unknown optimal (oracle) shrinkage parameters that yield the minimum mean squared error (MMSE) between the RSCM and the true covariance matrix when the data is sampled from an unspecified elliptically symmetric distribution. We propose two variants of the RSCM estimator which differ in the approach in which they estimate the underlying sphericity parameter involved in the theoretical optimal shrinkage parameter. The performance of the proposed RSCM estimators are evaluated with numerical simulation studies. In particular when the sample sizes are low, the proposed RSCM estimators often show a significant improvement over the conventional RSCM estimator by Ledoit and Wolf (2004). We further evaluate the performance of the proposed estimators in classification and portfolio optimization problems with real data wherein the proposed methods are able to outperform the benchmark methods.

Covariance matrix estimation is one of the most important problems in statistics. To accommodate the complexity of modern datasets, it is desired to have estimation procedures that not only can incorporate the structural assumptions of covariance matrices, but are also robust to outliers from arbitrary sources. In this paper, we define a new concept called matrix depth and then propose a robust covariance matrix estimator by maximizing the empirical depth function. The proposed estimator is shown to achieve minimax optimal rate under Huber’s $\\varepsilon$-contamination model for estimating covariance/scatter matrices with various structures including bandedness and sparsity.

The covariance matrix, which should be estimated from the data, plays an important role in many multivariate procedures, and its positive definiteness (PDness) is essential for the validity of the procedures. Recently, many regularized estimators have been proposed and shown to be consistent in estimating the true matrix and its support under various structural assumptions on the true covariance matrix. However, they are often not PD. In this paper, we propose a simple modification to make a regularized covariance matrix be PD while preserving its support and the convergence rate. We focus on the matrix $$\ell_{\infty }$$ norm error in covariance matrix estimation because it could allow us to bound the error in the downstream multivariate procedure relying on it. Our proposal in this paper is an extension of the fixed support positive-definite (FSPD) modification by Choi et al. (2019) from spectral and Frobenius norms to the matrix $$\ell_{\infty }$$ norm. Like the original FSPD, we consider a convex combination between the initial estimator (the regularized covariance matrix without PDness) and a given form of the diagonal matrix minimize the $$\ell_{\infty }$$ distance between the initial estimator and the convex combination, and find a closed-form expression for the modification. We apply the procedure to the minimum variance portfolio (MVP) optimization problem and show that the vector $$\ell_{\infty }$$ error in the estimation of the optimal portfolio weight is bounded by the matrix $$\ell _{\infty }$$ error of the plug-in covariance matrix estimator. We illustrate the MVP results with S&P 500 daily returns data from January 1978 to December 2014.

One of the major challenges in multivariate analysis is the estimation of population covariance matrix from the sample covariance matrix (SCM). Most recent covariance matrix estimators use either shrinkage transformations or asymptotic results from Random Matrix Theory (RMT). Both of these techniques try to achieve a similar goal which is to remove noisy correlations and add structure to SCM to overcome the bias-variance trade-off. Both methods have their respective pros and cons. In this paper, we propose an improved estimator which exploits the advantages of these techniques by taking optimally weighted convex combination of covariance matrices estimated by shrinkage transformation and a filter based on RMT. It is a generalized estimator which can adapt to changing sampling noise conditions by performing hyperparameter optimization. Using data from six of the world's biggest stock exchanges, we show that the proposed estimator outperforms the existing estimators in minimizing the out-of-sample risk of the portfolio and hence predicts population statistics more precisely. The proposed estimator can be useful in a wide range of machine learning and signal processing applications. © 1994-2012 IEEE.

Matrix completion is a modern missing data problem where both the missing structure and the underlying parameter are high dimensional. Although missing structure is a key component to any missing data problems, existing matrix completion methods often assume a simple uniform missing mechanism. In this work, we study matrix completion from corrupted data under a novel low-rank missing mechanism. The probability matrix of observation is estimated via a high dimensional low-rank matrix estimation procedure, and further used to complete the target matrix via inverse probabilities weighting. Due to both high dimensional and extreme (i.e., very small) nature of the true probability matrix, the effect of inverse probability weighting requires careful study. We derive optimal asymptotic convergence rates of the proposed estimators for both the observation probabilities and the target matrix.

The study aimed at identifying the role and importance of risk management in enhancing and managing the strategic and dynamic marketing decisions that relate to strategic problems of multi-dimensions and of great depth and complexity. This kind of decision requires in-depth research of the risks and the capacity for future analysis and discussions, which deal with all hypotheses and possibilities. Thus, the decision makers are critical resource for the company. Thus, it is necessary to stress on the importance of studying risk management and its role in enhancing the strategic marketing decision-making process, which is considered a way for companies to drive the future, which is the center of this study. In order to achieve the objectives of this study, a questionnaire was designed and distributed in person on a sample of managers of industrial companies in the Jordanian private sector.45 out of 65 distributed questionnaires were found to be valid. The data were statistically analyzed and the results showed that the strategic marketing decision makers in the industrial company realize the importance of risk management and employ it in the process of strategic marketing decision-making. However, the actual management or perfect investment of risk management in the industrial sector is still limited due to the existence of obstacles and limitations concerning managers and leaders and the available technical, information and material capabilities.

Risk Management at Crunch Time: Are Chief Risk Officers Compliance Champions or Business Partners?

Modelling and forecasting covariance matrices of asset returns play a crucial role in many financial fields, such as portfolio allocation and asset pricing. The availability of high-frequency intraday data enables the modelling of the realized covariance matrix directly. However, most models in the literature suffer from the curse of dimensionality, i.e. the number of parameters needed increases at the rate of the square of the number of assets. To solve the problem, we propose a factor model with a diagonal Conditional Autoregressive Wishart model for the factor realized covariance matrices. Consequently, the positive definiteness of the estimated covariance matrix is ensured with the proposed model. Asymptotic theory is derived for the estimated parameters. In the extensive empirical analysis, we find that the number of parameters can be reduced significantly; to only about one-tenth of the benchmark model. Furthermore, the proposed model maintains a comparable performance with a benchmark vector autoregressive model for different forecast horizons.

Inferences in panel data with interactive effects using large covariance matrices