Factored sparse inverse covariance matrices

Estimating large covariance and precision (inverse covariance) matrices has become increasingly important in high dimensional statistics because of its wide applications. The estimation problem is challenging not only theoretically due to the constraint of its positive definiteness, but also computationally because of the curse of dimensionality. Many types of estimators have been proposed such as thresholding under the sparsity assumption of the target matrix, banding and tapering the sample covariance matrix. However, these estimators are not always guaranteed to be positive-definite, especially, for finite samples, and the sparsity assumption is rather restrictive. We propose a novel two-stage adaptive method based on the Cholesky decomposition of a general covariance matrix. By banding the precision matrix in the first stage and adapting the estimates to the second stage estimation, we develop a computationally efficient and statistically accurate method for estimating high dimensional precision matrices. We demonstrate the finite-sample performance of the proposed method by simulations from autoregressive, moving average, and long-range dependent processes. We illustrate its wide applicability by analyzing financial data such S&P 500 index and IBM stock returns, and electric power consumption of individual households. The theoretical properties of the proposed method are also investigated within a large class of covariance matrices.

A key component in most parametric classifiers is the estimation of an inverse covariance matrix. In hyperspectral images, the number of bands can be in the hundreds, leading to covariance matrices having tens of thousands of elements. Lately, the use of linear regression in estimating the inverse covariance matrix has been introduced in the time-series literature. This paper adopts and expands these ideas to ill-posed hyperspectral image classification problems. The results indicate that at least some of the approaches can give a lower classification error than traditional methods such as the linear discriminant analysis and the regularized discriminant analysis. Furthermore, the results show that, contrary to earlier beliefs, estimating long-range dependencies between bands appears necessary to build an effective hyperspectral classifier and that the high correlations between neighboring bands seem to allow differing sparsity configurations of the inverse covariance matrix to obtain similar classification results.

Stealth target becomes serious growing threats to the military radar, especially with the recently sub-Nyquist scattering wave deception jammer (SN-SWDJ) which has not countering solution yet, because of the fast sampling rate. This paper engages in improving detection of stealth target (F-117A) under SN-SWDJ effect by using stratospheric balloon-borne bistatic radar system with higher aspect vision. The potential problems of proposed scheme come from heterogeneity caused by sudden gusts of wind, and on the other hand, stealth target is always included in the array data due to the coupling of SN-SWDJ with target motion. To tackle these problems, we introduce an optimal suppression technique with a novel combination of knowledge-aided space-time adaptive processing (KA-STAP) and fast time robust adaptive beamforming (FTRAB) algorithms. FTRAB is applied to remove stealth target components from the array data, as the initial guess of KA-STAP covariance matrix. However, our proposed scheme has a higher complexity of the covariance matrix inversion. To reduce this complexity, we propose the linear affine combination of the inverse covariance matrices' estimation instead of the covariance matrix itself. Simulations demonstrate that the proposed scheme gives much higher location accuracy and reduces complexity comparing with other algorithms.

In this work, we describe advanced numerical tools for working with multivariate functions and for the analysis of large data sets. These tools will drastically reduce the required computing time and the storage cost, and, therefore, will allow us to consider much larger data sets or finer meshes. Covariance matrices are crucial in spatio-temporal statistical tasks, but are often very expensive to compute and store, especially in three dimensions. Therefore, we approximate covariance functions by cheap surrogates in a low-rank tensor format. We apply the Tucker and canonical tensor decompositions to a family of Matérn- and Slater-type functions with varying parameters and demonstrate numerically that their approximations exhibit exponentially fast convergence. We prove the exponential convergence of the Tucker and canonical approximations in tensor rank parameters. Several statistical operations are performed in this low-rank tensor format, including evaluating the conditional covariance matrix, spatially averaged estimation variance, computing a quadratic form, determinant, trace, loglikelihood, inverse, and Cholesky decomposition of a large covariance matrix. Low-rank tensor approximations reduce the computing and storage costs essentially. For example, the storage cost is reduced from an exponential 𝒪 ⁢ ( n d ) {\mathcal{O}(n^{d})} to a linear scaling 𝒪 ⁢ ( d ⁢ r ⁢ n ) {\mathcal{O}(drn)} , where d is the spatial dimension, n is the number of mesh points in one direction, and r is the tensor rank. Prerequisites for applicability of the proposed techniques are the assumptions that the data, locations, and measurements lie on a tensor (axes-parallel) grid and that the covariance function depends on a distance, ∥ x - y ∥ {\|x-y\|} .

Summary The primary purpose of this paper is to propose empirical measures of the structural differences between two communities of plants or animals composed of the same species. Structure is defined to consist of; 1) the species in the community, 2) the pattern of interactions as represented by the covariance or correlation matrix of successive observations on each species, and 3) the mean abundances of each species in each of the two communities. Statistical tests are proposed for testing whether the covariance matrices and the vectors of mean densities for each community are equal and empirical measures of the differences between the covariance matrices and mean vectors are proposed. Given unequal covariance or correlation matrices the factor analysis model is used to derive empirical measures of the degree to which each variable of the ecosystem is responsible for the observed defferences in the pattern of interactions in each community. These tests and measures were applied to data gathered by Hunter (1966) on the abundances of six species of Drosophila censused monthly over a period of approximately two and a half years in two adjacent, but different habitats near Bogota, colombia. The two covariance matrices were significantly different indicating different patterns of interactions in the two Drosophila communities.

We propose to use acoustic models with sparse inverse covariance matrices to deal with the well-known over-fitting problem of discriminative training, especially when training data are limited. Compared with traditional diagonal or full covariance models, significant improvement by using sparse inverse covariance matrices has been achieved with maximum likelihood training. In state-of-the-art large vocabulary continuous speech recognition systems, discriminative training is commonly employed to achieve the best system performance. This paper investigates training acoustic models with sparse inverse covariance matrices using one of the most widely used discriminative training criteria-maximum mutual information (MMI). A lasso regularization term is added to the traditional objective function for MMI to automatically sparsify the inverse covariance matrices. The whole training process is then derived by maximizing the new objective function. This is achieved through iteratively maximizing a weak-sense auxiliary function. The final problem is shown to be a convex optimization problem and can be efficiently solved. Experimental results on the published Wall Street Journal and our collected Mandarin data sets show that the acoustic models with sparse inverse covariance matrices consistently outperform the conventional diagonal and full covariance models.

On the accuracy of covariance matrix: Hessian versus Gauss–Newton methods in atmospheric remote sensing with infrared spectroscopy

Recent studies have demonstrated improved skill in numerical weather prediction via the use of spatially correlated observation error covariance information in data assimilation systems. In this case, the observation weighting matrices (inverse error covariance matrices) used in the assimilation may be full matrices rather than diagonal. Thus, the computation of matrix–vector products in the variational minimization problem may be very time‐consuming, particularly if the parallel computation of the matrix–vector product requires a high degree of communication between processing elements. Hence, we introduce a well‐known numerical approximation method, called the fast multipole method (FMM), to speed up the matrix–vector multiplications in data assimilation. We explore a particular type of FMM that uses a singular value decomposition (SVD‐FMM) and adjust it to suit our new application in data assimilation. By approximating a large part of the computation of the matrix–vector product, the SVD‐FMM technique greatly reduces the computational complexity compared with the standard approach. We develop a novel possible parallelization scheme of the SVD‐FMM for our application, which can reduce the communication costs. We investigate the accuracy of the SVD‐FMM technique in several numerical experiments: we first assess the accuracy using covariance matrices that are created using different correlation functions and lengthscales, then investigate the impact of reconditioning the covariance matrices on the accuracy, and finally examine the feasibility of the technique in the presence of missing observations. We also provide theoretical explanations for some numerical results. Our results show that the SVD‐FMM technique can compute the matrix–vector product with good accuracy in a wide variety of circumstances and, hence, has potential as an efficient technique for assimilation of a large volume of observational data within a short time interval.

A joint convex penalty for inverse covariance matrix estimation

Inferences in panel data with interactive effects using large covariance matrices

The multivariate time series (MTS) classification is an important classification problem in which data has the temporal attribute. Because relationships between many variables of the MTS are complex and time-varying, existing methods perform not well in MTS classification with many attribute variables. Thus, in this paper, we propose a novel model-based classification method, called Kullback-Leibler Divergence-based Gaussian Model Classification (KLD-GMC), which converts the original MTS data into two important parameters of the multivariate Gaussian model: the mean vector and the inverse covariance matrix. The inverse covariance is the most important parameter, which can obtain the information between the variables. So that the more variables, the more information could be obtained by the inverse covariance, KLD-GMC can deal with the relationship between variables well in the MTS. Then the sparse inverse covariance of each subsequence is solved by Graphical Lasso. Furthermore, the Kullback-Leibler divergence is used as the similarity measurement to implement the classification of unlabeled subsequences, because it can effectively measure the similarity between different distributions. Experimental results on classical MTS datasets demonstrate that our method can improve the performance of multivariate time series classification and outperform the state-of-the-art methods.

The affine equivariant sign covariance matrix: asymptotic behavior and efficiencies

This paper revisits the linear model with noisy inputs, in which the performance of the total least squares (TLS) method is far from acceptable. Under the assumption of Gaussian noises, the maximum likelihood (ML) estimation of the system response is reformulated as a general balanced least squares (BLS) problem. Unlike TLS, which minimizes the trace of the product between the empirical and inverse theoretical covariance matrices, BLS promotes solutions with similar values of both the empirical and theoretical error covariance matrices. The general BLS problem is reformulated as a semidefinite program with a rank constraint, which can be relaxed in order to obtain polynomial time algorithms. Moreover, we provide new theoretical results regarding the scenarios in which the relaxation is tight, as well as additional insights on the performance and interpretation of BLS. Finally, some simulation results illustrate the satisfactory performance of the proposed method.

Summary - Dominance models are parameterized under conditions of inbreeding. The properties of an infinitesimal dominance model are reconsidered. It is shown that mixedmodel methodology is justifiable as normality assumptions can be met. Tabular methods for calculating genotypic covariances among inbred relatives are described. These methods employ 5 parameters required to accommodate additivity, dominance and inbreeding. Rules for calculating inverse genotypic covariance matrices are presented. These inverse matrices can be used directly to set up the mixed-model equations. The mixed-model methodology allowing for dominance and inbreeding provides a powerful framework to better explain and utilize the observed variation in quantitative traits.

Linear prediction of range-dependent inverse covariance matrix (PICM) sequences