Structured Covariance Matrices for Statistical Image Object Recognition

In this paper a novel algorithm for Gaussian Selection (GS) of mixtures used in a continuous speech recognition system is presented. The system is based on hidden Markov models (HMM), using Gaussian mixtures with full covariance matrices as output distributions. The purpose of Gaussian selection is to increase the speed of a speech recognition system, without degrading the recognition accuracy. The basic idea is to form hyper-mixtures by clustering close mixtures into a single group by means of Vector Quantization (VQ) and assigning it unique Gaussian parameters for estimation. In the decoding process only those hyper-mixtures which are above a designated threshold are selected, and only mixtures belonging to them are evaluated, improving computational efficiency. There is no problem with the clustering and evaluation if overlaps between the mixtures are small, and their variances are of the same range. However, in real case, there are numerous models which do not fit this profile. A Gaussian selection scheme proposed in this paper addresses this problem. For that purpose, beside the clustering algorithm, it also incorporates an algorithm for mixture grouping. The particular mixture is assigned to a group from the predefined set of groups, based on a value aggregated from eigenvalues of the covariance matrix of that mixture using Ordered Weighted Averaging operators (OWA). After the grouping of mixtures is carried out, Gaussian mixture clustering is performed on each group separately.

This paper presents a new algorithm for speaker recognition based on the combination between the classical Vector Quantization (VQ) and Covariance Matrix (CM) methods. The combined VQ-CM method improves the identification rates of each method alone, with comparable computational burden. It offers a straightforward procedure to obtain a model similar to GMM with full covariance matrices. Experimental results also show that it is more robust against noise than VQ or CM alone.

High dimensional covariance matrices have attracted much attention of statisticians and econometricians during the past decades. Vast literature is devoted to the research in high dimensional covariance matrices. However, most of them are for constant covariance matrices. In many applications, constant covariance matrices are not appropriate, e.g. in portfolio allocation, dynamic covariance matrices would make much more sense. Simply assuming each entry of a covariance matrix is a function of time to introduce a dynamic structure would not work. In this paper, we are going to introduce a class of high dimensional dynamic covariance matrices in which a kind of additive structure is embedded. We will show the proposed high dimensional dynamic covariance matrices have many advantages in applications. An estimation procedure is also proposed to estimate the proposed high dimensional dynamic covariance matrices. Asymptotic properties are built to justify the proposed estimation procedure. Intensive simulation studies show the proposed estimation procedure works very well when sample size is finite. Finally, we apply the proposed high dimensional dynamic covariance matrices, together with the proposed estimation procedure, to portfolio allocation. The results look very interesting.

The Riemannian geometry of covariance matrices has been essential to several successful applications, in computer vision, biomedical signal and image processing, and radar data processing. For these applications, an important ongoing challenge is to develop Riemannian-geometric tools which are adapted to structured covariance matrices. This paper proposes to meet this challenge by introducing a new class of probability distributions, Gaussian distributions of structured covariance matrices . These are Riemannian analogs of Gaussian distributions, which only sample from covariance matrices having a preassigned structure, such as complex, Toeplitz, or block-Toeplitz. The usefulness of these distributions stems from three features: 1) they are completely tractable, analytically, or numerically, when dealing with large covariance matrices; 2) they provide a statistical foundation to the concept of structured Riemannian barycentre ( i.e. , Frechet or geometric mean); and 3) they lead to efficient statistical learning algorithms, which realise, among others, density estimation and classification of structured covariance matrices. This paper starts from the observation that several spaces of structured covariance matrices, considered from a geometric point of view, are Riemannian symmetric spaces. Accordingly, it develops an original theory of Gaussian distributions on Riemannian symmetric spaces, of their statistical inference, and of their relationship to the concept of Riemannian barycentre. Then, it uses this original theory to give a detailed description of Gaussian distributions of three kinds of structured covariance matrices, complex, Toeplitz, and block-Toeplitz. Finally, it describes algorithms for density estimation and classification of structured covariance matrices, based on Gaussian distribution mixture models.

Summary The primary purpose of this paper is to propose empirical measures of the structural differences between two communities of plants or animals composed of the same species. Structure is defined to consist of; 1) the species in the community, 2) the pattern of interactions as represented by the covariance or correlation matrix of successive observations on each species, and 3) the mean abundances of each species in each of the two communities. Statistical tests are proposed for testing whether the covariance matrices and the vectors of mean densities for each community are equal and empirical measures of the differences between the covariance matrices and mean vectors are proposed. Given unequal covariance or correlation matrices the factor analysis model is used to derive empirical measures of the degree to which each variable of the ecosystem is responsible for the observed defferences in the pattern of interactions in each community. These tests and measures were applied to data gathered by Hunter (1966) on the abundances of six species of Drosophila censused monthly over a period of approximately two and a half years in two adjacent, but different habitats near Bogota, colombia. The two covariance matrices were significantly different indicating different patterns of interactions in the two Drosophila communities.

Estimating large covariance and precision (inverse covariance) matrices has become increasingly important in high dimensional statistics because of its wide applications. The estimation problem is challenging not only theoretically due to the constraint of its positive definiteness, but also computationally because of the curse of dimensionality. Many types of estimators have been proposed such as thresholding under the sparsity assumption of the target matrix, banding and tapering the sample covariance matrix. However, these estimators are not always guaranteed to be positive-definite, especially, for finite samples, and the sparsity assumption is rather restrictive. We propose a novel two-stage adaptive method based on the Cholesky decomposition of a general covariance matrix. By banding the precision matrix in the first stage and adapting the estimates to the second stage estimation, we develop a computationally efficient and statistically accurate method for estimating high dimensional precision matrices. We demonstrate the finite-sample performance of the proposed method by simulations from autoregressive, moving average, and long-range dependent processes. We illustrate its wide applicability by analyzing financial data such S&P 500 index and IBM stock returns, and electric power consumption of individual households. The theoretical properties of the proposed method are also investigated within a large class of covariance matrices.

Exploring similarity-based classification of larynx disorders from human voice

Upcoming and existing large-scale surveys of galaxies require accurate theoretical predictions of the dark matter clustering statistics for thousands of mock galaxy catalogs. We demonstrate that this goal can be achieve with our new Parallel Particle-Mesh (PM) Nbody code (PPM-GLAM) at a very low computational cost. We run about 15,000 simulations with ~2 billion particles that provide ~1% accuracy of the dark matter power spectra P(k) for wave-numbers up to k~ 1h/Mpc. Using this large data-set we study the power spectrum covariance matrix, the stepping stone for producing mock catalogs. In contrast to many previous analytical and numerical results, we find that the covariance matrix normalised to the power spectrum C(k,k')/P(k)P(k') has a complex structure of non-diagonal components. It has an upturn at small k, followed by a minimum at k=0.1-0.2h/Mpc. It also has a maximum at k=0.5-0.6h/Mpc. The normalised covariance matrix strongly evolves with redshift: C(k,k')~delta(t)^alpha P(k)P(k'), where delta is the linear growth factor and alpha ~ 1-1.25, which indicates that the covariance matrix depends on cosmological parameters. We also show that waves longer than 1Gpc have very little impact on the power spectrum and covariance matrix. This significantly reduces the computational costs and complexity of theoretical predictions: relatively small volume ~ (1Gpc)^3 simulations capture the necessary properties of dark matter clustering statistics. All the power spectra obtained from many thousands of our simulations are publicly available.

Estimation of the high-dimensional banded covariance matrix is widely used in multivariate statistical analysis. To ensure the validity of estimation, we aim to test the hypothesis that the covariance matrix is banded with a certain bandwidth under the high-dimensional framework. Though several testing methods have been proposed in the literature, the existing tests are only powerful for some alternatives with certain sparsity levels, whereas they may not be powerful for alternatives with other sparsity structures. The goal of this paper is to propose a new test for the bandedness of high-dimensional covariance matrix, which is powerful for alternatives with various sparsity levels. The proposed new test also be used for testing the banded structure of covariance matrices of error vectors in high-dimensional factor models. Based on these statistics, a consistent bandwidth estimator is also introduced for a banded high dimensional covariance matrix. Extensive simulation studies and an application to a prostate cancer dataset from protein mass spectroscopy are conducted for evaluating the effectiveness of the proposed adaptive tests blue and bandwidth estimator for the banded covariance matrix.

Radar data suffers from the curse of dimensionality, and with the deluge of data, due to advances in sensors, and technology, it has becomes increasingly challenging to interpret, visualize and process data. In this paper, we assume that the dimensions of the data matrix is large. That is, we assume that the number of variables as well as the number of samples is large, however, their ratio is a finite constant. We further assume a structured covariance matrix, comprised of low rank matrices added to a scaled identity matrix. The behavior of their principal components (PCs) and eigenvalues of the estimated covariance matrix is analyzed using well known tools in random matrix theory (RMT) and almost sure limits are studied. Simulations reveal that the analysis is valid for many distributions, not necessarily Gaussian, provided these samples are statistically independent. The theory is applied to the covariance matrices generated from the KASSPER database.

Summary Ensemble optimization (referred to throughout the remainder of the paper as EnOpt) is a rapidly emerging method for reservoir-model-based production optimization. EnOpt uses an ensemble of controls to approximate the gradient of the objective function with respect to the controls. Current implementations of EnOpt use a Gaussian ensemble of control perturbations with a constant covariance matrix, and thus a constant perturbation size, during the entire optimization process. The covariance-matrix-adaptation evolutionary strategy is a gradient-free optimization method developed in the “machine learning” community, which also uses an ensemble of controls, but with a covariance matrix that is continually updated during the optimization process. It was shown to be an efficient method for several difficult but small-dimensional optimization problems and was recently applied in the petroleum industry for well location and production optimization. In this study, we investigate the scope to improve the computational efficiency of EnOpt through the use of covariance-matrix adaptation (referred to throughout the remainder of the paper as CMA-EnOpt). The resulting method is applied to the waterflooding optimization of a small multilayer test model and a modified version of the Brugge benchmark model. The controls used are inflow-control-valve settings at predefined time intervals for injectors and producers with undiscounted net present value as the objective function. We compare EnOpt and CMA-EnOpt starting from identical covariance matrices. For the small model, we achieve only slightly higher (0.7 to 1.8%) objective-function values and modest speedups with CMA-EnOpt compared with EnOpt. Significantly higher objective-function values (10%) are obtained for the modified Brugge model. The possibility to adapt the covariance matrix, and thus the perturbation size, during the optimization allows for the use of relatively large perturbations initially, for fast exploration of the control space, and small perturbations later, for more-precise gradients near the optimum. Moreover, the results demonstrate that a major benefit of CMA-EnOpt is its robustness with respect to the initial choice of the covariance matrix. A poor choice of the initial matrix can be detrimental to EnOpt, whereas the CMA-EnOpt performance is near-independent of the initial choice and produces higher objective-function values at no additional computational cost.

We consider the estimation of a Gaussian random vector x observed through a linear transformation H and corrupted by additive Gaussian noise with a known covariance matrix, where the covariance matrix of x is known to lie in a given region of uncertainty that is described using bounds on the eigenvalues and on the elements of the covariance matrix. Recently, two criteria for minimax estimation called difference regret (DR) and ratio regret (RR) were proposed and their closed form solutions were presented assuming that the eigenvalues of the covariance matrix of x are known to lie in a given region of uncertainty, and assuming that the matrices H T Cw -1 H and C x are jointly diagonalizable, where C w and C x denote the covariance matrices of the additive noise and of x respectively. In this work we present a new criterion for the minimax estimation problem which we call the generalized difference regret (GDR), and derive a new minimax estimator which is based on the GDR criterion where the region of uncertainty is defined not only using upper and lower bounds on the eigenvalues of the parameter's covariance matrix, but also using upper and lower bounds on the individual elements of the covariance matrix itself. Furthermore, the new estimator does not require the assumption of joint diagonalizability and it can be obtained efficiently using semidefinite programming. We also show that when the joint diagonalizability assumption holds and when there are only eigenvalue uncertainties, then the new estimator is identical to the difference regret estimator. The experimental results show that we can obtain improved mean squared error (MSE) results compared to the MMSE, DR, and RR estimators.

Sensitivity analysis of uncertain nonlinear systems plays an important role in analyzing chemical engineering process models. Even though a variety of different techniques exists for analyzing nonlinear systems, these methods often do not update the parameter distribution using available measurement data. This forms the motivation behind this work as a sensitivity analysis procedure is introduced that deals with nonlinear systems and uncertainty in model parameters, and also incorporates the use of available data. The technique computes two covariance matrices: a covariance matrix that describes the effect that changes of the parameters values, sampled from a prior distribution, have on the outputs, and a covariance matrix that determines the effect of available data on the posterior distribution of the parameters using a Markov chain Monte Carlo method. The information contained in these two covariance matrices is then simultaneously evaluated by balancing the two matrices. The results returned by this technique are the directions in parameter space that have the largest impact on the system behavior according to the given parameter uncertainty as well as the contribution of individual parameters to these important directions in parameter space. The presented technique is applied to three examples.

The rotated delta, which is argued to be a theoretically better-grounded distance measure, has failed to receive any empirical support for its superiority. This study revisits the rotated delta—which is more commonly known as the Mahalanobis distance in other areas—with two different covariance matrices that are estimated from training data. The first covariance matrix represents the between-author variability, and the second the within-author variability. A series of likelihood ratio-based authorship verification experiments was carried out with some different distance measures. The experiments made use of the documents arranged from a large database of text messages that allowed for a total of 2,160 same-author and 4,663,440 different-author comparisons. The Mahalanobis distance with the between-author covariance matrix performed far worse compared to the other distance measures, whereas the Mahalanobis distance with the within-author covariance matrix performed better than the other measures. However, superior performance relative to the cosine distance is subject to word lengths and/or the order of the feature vector. The result of follow-up experiments further illustrated that the covariance matrix representing the within-author variability needs to be trained using a good amount of data to perform better than the cosine distance: the higher the order of the vector, the more data are required for training. The quantitative results also infer that the two sources of variabilities—notably within- and between-author variabilities—are independent of each other to the extent that the latter cannot accurately approximate the former.

This paper develops an improved covariance matrix estimator in the mean-variance optimisation setting. Well-known problems with the sample covariance matrix are that it is singular when the number of observations is less than the number of assets, and can be nearly singular when the number of observations exceeds the number of assets. Therefore, using the sample covariance matrix as an input in mean-variance optimisation can result in unreasonable optimal portfolios and badly biased estimates of Sharpe ratios. We address this problem by imposing structure on the estimated covariance matrix by putting constraints on the Sharpe ratio, asset return variances, and the variance of the global minimum variance portfolio. We show that the constrained maximum likelihood estimator (CMLE) performs better than the sample covariance matrix. Moreover, when the shrinkage approach is applied to the CMLE and single index covariance matrix, it performs better than the shrinkage estimator of Ledoit and Wolf (2004).