Improving the Ensemble-Optimization Method Through Covariance-Matrix Adaptation

Summary The primary purpose of this paper is to propose empirical measures of the structural differences between two communities of plants or animals composed of the same species. Structure is defined to consist of; 1) the species in the community, 2) the pattern of interactions as represented by the covariance or correlation matrix of successive observations on each species, and 3) the mean abundances of each species in each of the two communities. Statistical tests are proposed for testing whether the covariance matrices and the vectors of mean densities for each community are equal and empirical measures of the differences between the covariance matrices and mean vectors are proposed. Given unequal covariance or correlation matrices the factor analysis model is used to derive empirical measures of the degree to which each variable of the ecosystem is responsible for the observed defferences in the pattern of interactions in each community. These tests and measures were applied to data gathered by Hunter (1966) on the abundances of six species of Drosophila censused monthly over a period of approximately two and a half years in two adjacent, but different habitats near Bogota, colombia. The two covariance matrices were significantly different indicating different patterns of interactions in the two Drosophila communities.

Estimating large covariance and precision (inverse covariance) matrices has become increasingly important in high dimensional statistics because of its wide applications. The estimation problem is challenging not only theoretically due to the constraint of its positive definiteness, but also computationally because of the curse of dimensionality. Many types of estimators have been proposed such as thresholding under the sparsity assumption of the target matrix, banding and tapering the sample covariance matrix. However, these estimators are not always guaranteed to be positive-definite, especially, for finite samples, and the sparsity assumption is rather restrictive. We propose a novel two-stage adaptive method based on the Cholesky decomposition of a general covariance matrix. By banding the precision matrix in the first stage and adapting the estimates to the second stage estimation, we develop a computationally efficient and statistically accurate method for estimating high dimensional precision matrices. We demonstrate the finite-sample performance of the proposed method by simulations from autoregressive, moving average, and long-range dependent processes. We illustrate its wide applicability by analyzing financial data such S&P 500 index and IBM stock returns, and electric power consumption of individual households. The theoretical properties of the proposed method are also investigated within a large class of covariance matrices.

Upcoming and existing large-scale surveys of galaxies require accurate theoretical predictions of the dark matter clustering statistics for thousands of mock galaxy catalogs. We demonstrate that this goal can be achieve with our new Parallel Particle-Mesh (PM) Nbody code (PPM-GLAM) at a very low computational cost. We run about 15,000 simulations with ~2 billion particles that provide ~1% accuracy of the dark matter power spectra P(k) for wave-numbers up to k~ 1h/Mpc. Using this large data-set we study the power spectrum covariance matrix, the stepping stone for producing mock catalogs. In contrast to many previous analytical and numerical results, we find that the covariance matrix normalised to the power spectrum C(k,k')/P(k)P(k') has a complex structure of non-diagonal components. It has an upturn at small k, followed by a minimum at k=0.1-0.2h/Mpc. It also has a maximum at k=0.5-0.6h/Mpc. The normalised covariance matrix strongly evolves with redshift: C(k,k')~delta(t)^alpha P(k)P(k'), where delta is the linear growth factor and alpha ~ 1-1.25, which indicates that the covariance matrix depends on cosmological parameters. We also show that waves longer than 1Gpc have very little impact on the power spectrum and covariance matrix. This significantly reduces the computational costs and complexity of theoretical predictions: relatively small volume ~ (1Gpc)^3 simulations capture the necessary properties of dark matter clustering statistics. All the power spectra obtained from many thousands of our simulations are publicly available.

Estimation of the high-dimensional banded covariance matrix is widely used in multivariate statistical analysis. To ensure the validity of estimation, we aim to test the hypothesis that the covariance matrix is banded with a certain bandwidth under the high-dimensional framework. Though several testing methods have been proposed in the literature, the existing tests are only powerful for some alternatives with certain sparsity levels, whereas they may not be powerful for alternatives with other sparsity structures. The goal of this paper is to propose a new test for the bandedness of high-dimensional covariance matrix, which is powerful for alternatives with various sparsity levels. The proposed new test also be used for testing the banded structure of covariance matrices of error vectors in high-dimensional factor models. Based on these statistics, a consistent bandwidth estimator is also introduced for a banded high dimensional covariance matrix. Extensive simulation studies and an application to a prostate cancer dataset from protein mass spectroscopy are conducted for evaluating the effectiveness of the proposed adaptive tests blue and bandwidth estimator for the banded covariance matrix.

High dimensional covariance matrices have attracted much attention of statisticians and econometricians during the past decades. Vast literature is devoted to the research in high dimensional covariance matrices. However, most of them are for constant covariance matrices. In many applications, constant covariance matrices are not appropriate, e.g. in portfolio allocation, dynamic covariance matrices would make much more sense. Simply assuming each entry of a covariance matrix is a function of time to introduce a dynamic structure would not work. In this paper, we are going to introduce a class of high dimensional dynamic covariance matrices in which a kind of additive structure is embedded. We will show the proposed high dimensional dynamic covariance matrices have many advantages in applications. An estimation procedure is also proposed to estimate the proposed high dimensional dynamic covariance matrices. Asymptotic properties are built to justify the proposed estimation procedure. Intensive simulation studies show the proposed estimation procedure works very well when sample size is finite. Finally, we apply the proposed high dimensional dynamic covariance matrices, together with the proposed estimation procedure, to portfolio allocation. The results look very interesting.

The proper selection of the model error covariance matrix and the measurement noise covariance matrix of Kalman filter is an optimization problem. Some scholars have studied this, but there is relatively little research on the selection of the two covariance matrices for Kalman filters with an unknown input. Recently, the authors proposed a modified directed bat algorithm (MDBA) which introduces the best historical location of individuals and the elimination strategy to effectively prevent falling into local optimal solution. So, two methods are proposed in this paper to optimize the model error covariance matrix and measurement noise covariance matrix of Kalman filter with unknown inputs (KF-UI) and extended Kalman filter with unknown inputs (EKF-UI) by MDBA, respectively. The objective functions are constructed using the measurement vectors and the corresponding estimated values, and MDBA is adopted to optimize the two covariance matrices of KF-UI and EKF-UI. To validate the effectiveness of proposed methods, two simple structure examples and a benchmark example are adopted. The influence of structural parameter uncertainties on KF-UI is also considered. The result shows that the MDBA-optimized KF-UI has a strong convergence and can take into account the effect of parameter uncertainties. Then, the effectiveness of the proposed MDBA-optimized EKF-UI method is validated by comparing it with EKF-UI with empirically selected covariance values through trial-and-error. The identification results showed that the proposed methods achieved better identification accuracy and enhanced convergence compared to KF-UI and EKF-UI with empirical covariance values.

Randomized direct search algorithms for continuous domains, such as evolution strategies, are basic tools in machine learning. They are especially needed when the gradient of an objective function (e.g., loss, energy, or reward function) cannot be computed or estimated efficiently. Application areas include supervised and reinforcement learning as well as model selection. These randomized search strategies often rely on normally distributed additive variations of candidate solutions. In order to efficiently search in non-separable and ill-conditioned landscapes the covariance matrix of the normal distribution must be adapted, amounting to a variable metric method. Consequently, covariance matrix adaptation (CMA) is considered state-of-the-art in evolution strategies. In order to sample the normal distribution, the adapted covariance matrix needs to be decomposed, requiring in general Θ(n 3) operations, where n is the search space dimension. We propose a new update mechanism which can replace a rank-one covariance matrix update and the computationally expensive decomposition of the covariance matrix. The newly developed update rule reduces the computational complexity of the rank-one covariance matrix adaptation to Θ(n 2) without resorting to outdated distributions. We derive new versions of the elitist covariance matrix adaptation evolution strategy (CMA-ES) and the multi-objective CMA-ES. These algorithms are equivalent to the original procedures except that the update step for the variable metric distribution scales better in the problem dimension. We also introduce a simplified variant of the non-elitist CMA-ES with the incremental covariance matrix update and investigate its performance. Apart from the reduced time-complexity of the distribution update, the algebraic computations involved in all new algorithms are simpler compared to the original versions. The new update rule improves the performance of the CMA-ES for large scale machine learning problems in which the objective function can be evaluated fast.

We consider the estimation of a Gaussian random vector x observed through a linear transformation H and corrupted by additive Gaussian noise with a known covariance matrix, where the covariance matrix of x is known to lie in a given region of uncertainty that is described using bounds on the eigenvalues and on the elements of the covariance matrix. Recently, two criteria for minimax estimation called difference regret (DR) and ratio regret (RR) were proposed and their closed form solutions were presented assuming that the eigenvalues of the covariance matrix of x are known to lie in a given region of uncertainty, and assuming that the matrices H T Cw -1 H and C x are jointly diagonalizable, where C w and C x denote the covariance matrices of the additive noise and of x respectively. In this work we present a new criterion for the minimax estimation problem which we call the generalized difference regret (GDR), and derive a new minimax estimator which is based on the GDR criterion where the region of uncertainty is defined not only using upper and lower bounds on the eigenvalues of the parameter's covariance matrix, but also using upper and lower bounds on the individual elements of the covariance matrix itself. Furthermore, the new estimator does not require the assumption of joint diagonalizability and it can be obtained efficiently using semidefinite programming. We also show that when the joint diagonalizability assumption holds and when there are only eigenvalue uncertainties, then the new estimator is identical to the difference regret estimator. The experimental results show that we can obtain improved mean squared error (MSE) results compared to the MMSE, DR, and RR estimators.

Sensitivity analysis of uncertain nonlinear systems plays an important role in analyzing chemical engineering process models. Even though a variety of different techniques exists for analyzing nonlinear systems, these methods often do not update the parameter distribution using available measurement data. This forms the motivation behind this work as a sensitivity analysis procedure is introduced that deals with nonlinear systems and uncertainty in model parameters, and also incorporates the use of available data. The technique computes two covariance matrices: a covariance matrix that describes the effect that changes of the parameters values, sampled from a prior distribution, have on the outputs, and a covariance matrix that determines the effect of available data on the posterior distribution of the parameters using a Markov chain Monte Carlo method. The information contained in these two covariance matrices is then simultaneously evaluated by balancing the two matrices. The results returned by this technique are the directions in parameter space that have the largest impact on the system behavior according to the given parameter uncertainty as well as the contribution of individual parameters to these important directions in parameter space. The presented technique is applied to three examples.

The rotated delta, which is argued to be a theoretically better-grounded distance measure, has failed to receive any empirical support for its superiority. This study revisits the rotated delta—which is more commonly known as the Mahalanobis distance in other areas—with two different covariance matrices that are estimated from training data. The first covariance matrix represents the between-author variability, and the second the within-author variability. A series of likelihood ratio-based authorship verification experiments was carried out with some different distance measures. The experiments made use of the documents arranged from a large database of text messages that allowed for a total of 2,160 same-author and 4,663,440 different-author comparisons. The Mahalanobis distance with the between-author covariance matrix performed far worse compared to the other distance measures, whereas the Mahalanobis distance with the within-author covariance matrix performed better than the other measures. However, superior performance relative to the cosine distance is subject to word lengths and/or the order of the feature vector. The result of follow-up experiments further illustrated that the covariance matrix representing the within-author variability needs to be trained using a good amount of data to perform better than the cosine distance: the higher the order of the vector, the more data are required for training. The quantitative results also infer that the two sources of variabilities—notably within- and between-author variabilities—are independent of each other to the extent that the latter cannot accurately approximate the former.

This paper develops an improved covariance matrix estimator in the mean-variance optimisation setting. Well-known problems with the sample covariance matrix are that it is singular when the number of observations is less than the number of assets, and can be nearly singular when the number of observations exceeds the number of assets. Therefore, using the sample covariance matrix as an input in mean-variance optimisation can result in unreasonable optimal portfolios and badly biased estimates of Sharpe ratios. We address this problem by imposing structure on the estimated covariance matrix by putting constraints on the Sharpe ratio, asset return variances, and the variance of the global minimum variance portfolio. We show that the constrained maximum likelihood estimator (CMLE) performs better than the sample covariance matrix. Moreover, when the shrinkage approach is applied to the CMLE and single index covariance matrix, it performs better than the shrinkage estimator of Ledoit and Wolf (2004).

Journal Article Nonparametric estimation of large covariance matrices of longitudinal data Get access Wei Biao Wu, Wei Biao Wu Search for other works by this author on: Oxford Academic Google Scholar Mohsen Pourahmadi Mohsen Pourahmadi Search for other works by this author on: Oxford Academic Google Scholar Biometrika, Volume 90, Issue 4, December 2003, Pages 831–844, https://doi.org/10.1093/biomet/90.4.831 Published: 01 December 2003

A robust algorithm for source number estimation based on the formation of the Hankel covariance matrix is presented. First, multiple data snapshots are taken successively from overlapped subarrays in a way similar to the forward spatial smoothing method to construct the special Hankel covariance matrix and for the total number of subarrays, these special covariance matrices are generated. Then, the average of these matrices is employed in singular value decomposition to generate the corresponding eigenvalues. Finally, the resulting eigenvalues are evaluated via the rule presented in this paper as the Moving Gradient Criterion (MGC) to estimate the number of sources by detection of the largest singular values. The greatest difference between the proposed algorithm and the other conventional methods is the form of the covariance matrix with the observed signal that can handle both non-coherent as well as fully coherent sources. Also, the proposed MGC rule adopted with this form of the covariance matrix is the strength of this work. Numerical simulations demonstrate the high superiority of the proposed approach over the competing methods such as MDL, AIC, SORTE, RAE and MSEE methods, especially in the cases of very closely spaced sources, low SNR values, low sensors number and low snapshots number.

In this paper the model of experiment in which several characteristics are measured or the measurements are repeated in time, is considered. It is assumed that the covariance matrix of the characteristics/time points is unknown, but has the banded Toeplitz structure. The problem of estimating such a covariance structure was considered by e.g. Cui et al. (2016), Filipiak et al. (2018). We discuss two methods that aim to deal with singularity and numerical ill-conditioning of the estimators of covariance matrix under high-dimensional regime. The method of estimation of banded Toeplitz covariance matrix based on a shrinking, described by Ledoit and Wolf (2004) for unstructured covariance matrices is proposed. The estimators obtained with the use of the proposed method are then compared with the estimators obtained by the projection of sample covariance matrix onto the asymptotic cone of the nonnegative definite Toeplitz matrices (Filipiak et al. 2018). For this purpose simulation studies concerning bias and risk are conducted for several sets of parameters.

Support vector regression is a promising method for time-series prediction, as it has good generalisability and an overall stable behaviour. Recent studies have shown that it can describe the dynamic characteristics of financial processes and make more accurate forecasts than other machine learning techniques. The first main contribution of this paper is to propose a methodology for dynamic modelling and forecasting covariance matrices based on support vector regression using the Cholesky decomposition. The procedure is applied to range-based covariance matrices of returns, which are estimated on the basis of low and high prices. Such prices are most often available with closing prices for many financial series and contain more information about volatility and relationships between returns. The methodology guarantees the positive definiteness of the forecasted covariance matrices and is flexible, as it can be applied to different dependence patterns. The second contribution of the paper is to show with an example of the exchange rates from the forex market that the covariance matrix forecasts calculated using the proposed approach are more accurate than the forecasts from the benchmark dynamic conditional correlation model. The advantage of the suggested procedure is higher during turbulent periods, i.e., when forecasting is the most difficult and accurate forecasts matter most.