Improved covariance matrices for point target detection in hyperspectral data

Summary The primary purpose of this paper is to propose empirical measures of the structural differences between two communities of plants or animals composed of the same species. Structure is defined to consist of; 1) the species in the community, 2) the pattern of interactions as represented by the covariance or correlation matrix of successive observations on each species, and 3) the mean abundances of each species in each of the two communities. Statistical tests are proposed for testing whether the covariance matrices and the vectors of mean densities for each community are equal and empirical measures of the differences between the covariance matrices and mean vectors are proposed. Given unequal covariance or correlation matrices the factor analysis model is used to derive empirical measures of the degree to which each variable of the ecosystem is responsible for the observed defferences in the pattern of interactions in each community. These tests and measures were applied to data gathered by Hunter (1966) on the abundances of six species of Drosophila censused monthly over a period of approximately two and a half years in two adjacent, but different habitats near Bogota, colombia. The two covariance matrices were significantly different indicating different patterns of interactions in the two Drosophila communities.

Deep convolutional neural networks can achieve remarkable results for detecting and recognizing large and medium sized objects in visible band color images. However, the ability to detect small objects has yet to achieve the same level performance. Our focus is on applications that require the accurate detection and localization of small moving objects that are distant from the sensor. State of the art object detection and classification networks (such YOLOv3 &#x005B;1&#x005D; and Mask R-CNN &#x005B;2&#x005D;) are not well suited for finding small moving objects in infra-red imagery, nor do they handle temporal information in video sequences. To overcome the limitations of these methods, we propose the Moving Target Indicator Network (MTINet), a new model specifically designed for detecting small moving targets in clutter. Several versions of the MTINet are presented with different refinements (such as spatial pyramid blocks and improved attention mechanism) to increase the probability of detection and reduce false alarm rates. The MTINet is trained to maximize the <i>target to clutter ratio</i> (TCR) metric, which represents the first use of the TCR loss function for detecting moving objects. The further challenge the MTINet, we also evaluated its performance with simulated sensor movement mimicking effects of image stabilization processes. Finally, we also propose a modification of the Reed-Xiaoli detector &#x005B;3&#x005D; (originally developed for anomaly detection in hyperspectral data) to enable it to detect temporal anomalies caused by moving objects, and refer to it as the temporal-ARX (t-ARX) algorithm. We find that the t-ARX algorithm can achieve better probability of detection at lower false alarm rates, while the MTINet is superior at finding difficult targets at higher false alarm rates. We then show that the combination of the two algorithms achieves the best overall performance.

At present there are some worm detection systems, primarily for a single LAN or with hardware router environment, which main use worm propagation characteristics for detection and has high false alarm rate, but it is not applicable for large-scale network for detecting. This paper presents a distributed worm detection technology, which is divided into two parts, client end and the console end program. The system uses the rule-based detection methods to monitor network worms, and the console side manages and coordinates the multiple side work of detecting. Experimental results show that the method can be good applicable for worms conduct surveillance based on a single or multiple local area network and is used for worm alarming, the method has high detection rate and low false alarm rate.

At present there are some worm intrusion detection systems, primarily for a single LAN or with hardware router environment, which are not applicable for large-scale network detection or have high false alarm rate by using only worm propagation characteristics for detection. This paper analyzed worm non-linear propagation models and drew out the worm transmission curves. Then a distributed worm detection technology is designed. The novel distributed worm detection system consists of two parts, client end and console end programs. The system uses rule-based detection method to monitor network worms, and the console side manages and coordinates detection work of the client sides. Experimental results show that the technology is a good solution to worm detection in multiple network environments which can give an alarm with high detection rate and low false alarm rate when the known worm appears.

Estimating large covariance and precision (inverse covariance) matrices has become increasingly important in high dimensional statistics because of its wide applications. The estimation problem is challenging not only theoretically due to the constraint of its positive definiteness, but also computationally because of the curse of dimensionality. Many types of estimators have been proposed such as thresholding under the sparsity assumption of the target matrix, banding and tapering the sample covariance matrix. However, these estimators are not always guaranteed to be positive-definite, especially, for finite samples, and the sparsity assumption is rather restrictive. We propose a novel two-stage adaptive method based on the Cholesky decomposition of a general covariance matrix. By banding the precision matrix in the first stage and adapting the estimates to the second stage estimation, we develop a computationally efficient and statistically accurate method for estimating high dimensional precision matrices. We demonstrate the finite-sample performance of the proposed method by simulations from autoregressive, moving average, and long-range dependent processes. We illustrate its wide applicability by analyzing financial data such S&amp;P 500 index and IBM stock returns, and electric power consumption of individual households. The theoretical properties of the proposed method are also investigated within a large class of covariance matrices.

Hyperspectral images are used for anomaly detection; the improvement over broadband imagery is due to the available spectral information, converting a two-dimensional image into a datacube. This paper deals with subpixel point target detection. The RX algorithm provides a statistical metric for how different an examined pixel is from the background in the data cube. It employs an inverse covariance matrix to estimate and limit the effect of the noise, which is normally estimated from all the pixels in the data cube. Since the background is non-stationary, an improvement in the detection performance can be achieved by segmentation. The first objective of this paper is to quantify in the algorithm the effect of using covariance matrices which are derived from the segments or the more local environments in which the pixel can be found. In addition, practically, pixels may be erroneously assigned to a specific segment though they are influenced by neighboring areas. We will examine different methods of choosing the "pure" pixels of each segment, and the influence of these methods on the probability of detection results.

Computer network defense (CND) protects organizations and individuals against cyber threats by monitoring, identifying, analyzing, and defending network infrastructure from infiltration. Network defenders must maintain high levels of cyber situation awareness (CSA) in order to correctly identify and act on threats to the network. Intrusion detection systems (IDSs) are automated systems designed to assist network defenders in building CSA by sifting through network traffic and flagging potential threats. These systems are plagued by high false alarm rates that inhibit the ability of network defenders to build CSA. More capable IDSs have been developed that are capable of increasing the hit rate and lowering the false alarm rate by analyzing gathered network information. The influence of these IDS technologies on CSA has yet to be explored. 172 San Jose State University psychology students performed a signal detection task for intrusion detection to examine whether integrated automation with a multilayered analysis incorporating both liberal and conservative response criteria leads to better CSA than less-integrated, yet liberally responding automation (high hit rates and high false alarm rates) or conservatively responding automation (with low hit rates and low false alarm rates). The IDS condition was manipulated at three levels (liberal, conservative, both). The reliability of the IDSs was manipulated at three levels (60%, 80%, 95%). This study was unable to observe any differences in task performance or CSA for any of the conditions.

Upcoming and existing large-scale surveys of galaxies require accurate theoretical predictions of the dark matter clustering statistics for thousands of mock galaxy catalogs. We demonstrate that this goal can be achieve with our new Parallel Particle-Mesh (PM) Nbody code (PPM-GLAM) at a very low computational cost. We run about 15,000 simulations with ~2 billion particles that provide ~1% accuracy of the dark matter power spectra P(k) for wave-numbers up to k~ 1h/Mpc. Using this large data-set we study the power spectrum covariance matrix, the stepping stone for producing mock catalogs. In contrast to many previous analytical and numerical results, we find that the covariance matrix normalised to the power spectrum C(k,k')/P(k)P(k') has a complex structure of non-diagonal components. It has an upturn at small k, followed by a minimum at k=0.1-0.2h/Mpc. It also has a maximum at k=0.5-0.6h/Mpc. The normalised covariance matrix strongly evolves with redshift: C(k,k')~delta(t)^alpha P(k)P(k'), where delta is the linear growth factor and alpha ~ 1-1.25, which indicates that the covariance matrix depends on cosmological parameters. We also show that waves longer than 1Gpc have very little impact on the power spectrum and covariance matrix. This significantly reduces the computational costs and complexity of theoretical predictions: relatively small volume ~ (1Gpc)^3 simulations capture the necessary properties of dark matter clustering statistics. All the power spectra obtained from many thousands of our simulations are publicly available.

Estimation of the high-dimensional banded covariance matrix is widely used in multivariate statistical analysis. To ensure the validity of estimation, we aim to test the hypothesis that the covariance matrix is banded with a certain bandwidth under the high-dimensional framework. Though several testing methods have been proposed in the literature, the existing tests are only powerful for some alternatives with certain sparsity levels, whereas they may not be powerful for alternatives with other sparsity structures. The goal of this paper is to propose a new test for the bandedness of high-dimensional covariance matrix, which is powerful for alternatives with various sparsity levels. The proposed new test also be used for testing the banded structure of covariance matrices of error vectors in high-dimensional factor models. Based on these statistics, a consistent bandwidth estimator is also introduced for a banded high dimensional covariance matrix. Extensive simulation studies and an application to a prostate cancer dataset from protein mass spectroscopy are conducted for evaluating the effectiveness of the proposed adaptive tests blue and bandwidth estimator for the banded covariance matrix.

High dimensional covariance matrices have attracted much attention of statisticians and econometricians during the past decades. Vast literature is devoted to the research in high dimensional covariance matrices. However, most of them are for constant covariance matrices. In many applications, constant covariance matrices are not appropriate, e.g. in portfolio allocation, dynamic covariance matrices would make much more sense. Simply assuming each entry of a covariance matrix is a function of time to introduce a dynamic structure would not work. In this paper, we are going to introduce a class of high dimensional dynamic covariance matrices in which a kind of additive structure is embedded. We will show the proposed high dimensional dynamic covariance matrices have many advantages in applications. An estimation procedure is also proposed to estimate the proposed high dimensional dynamic covariance matrices. Asymptotic properties are built to justify the proposed estimation procedure. Intensive simulation studies show the proposed estimation procedure works very well when sample size is finite. Finally, we apply the proposed high dimensional dynamic covariance matrices, together with the proposed estimation procedure, to portfolio allocation. The results look very interesting.

We present two methods to improve three hyperspectral stochastic algorithms for target detection; the algorithms are the constrained energy minimization, the generalized likelihood ratio test, and the adaptive coherence estimator. The original algorithms rely solely on spectral information and do not use spatial information; this usage is normally justified in subpixel target detection, since the target size is smaller than the size of a pixel. However, we found that since the background (and the false alarms) may be spatially correlated and the point spread function can distribute the energy of a point target between several neighboring pixels, the implementation of spatial filtering algorithms considerably improved target detection. Our first improvement used the local spatial mean and covariance matrices, which take into account the spatial local mean instead of the global mean. While this concept has been found in the literature, the effect of its implementation in both the estimated mean and the covariance matrix is examined quantitatively here. The second was based on the fact that the effect of a target of physical subpixel size will extend to a cluster of pixels. We tested our algorithms by using the data set and scoring methodology of the Rochester Institute of Technology Target Detection Blind Test project. The results showed that both spatial methods independently improved the basic spectral algorithms mentioned above, and when the two methods were used together, the results were even better.

Summary Ensemble optimization (referred to throughout the remainder of the paper as EnOpt) is a rapidly emerging method for reservoir-model-based production optimization. EnOpt uses an ensemble of controls to approximate the gradient of the objective function with respect to the controls. Current implementations of EnOpt use a Gaussian ensemble of control perturbations with a constant covariance matrix, and thus a constant perturbation size, during the entire optimization process. The covariance-matrix-adaptation evolutionary strategy is a gradient-free optimization method developed in the “machine learning” community, which also uses an ensemble of controls, but with a covariance matrix that is continually updated during the optimization process. It was shown to be an efficient method for several difficult but small-dimensional optimization problems and was recently applied in the petroleum industry for well location and production optimization. In this study, we investigate the scope to improve the computational efficiency of EnOpt through the use of covariance-matrix adaptation (referred to throughout the remainder of the paper as CMA-EnOpt). The resulting method is applied to the waterflooding optimization of a small multilayer test model and a modified version of the Brugge benchmark model. The controls used are inflow-control-valve settings at predefined time intervals for injectors and producers with undiscounted net present value as the objective function. We compare EnOpt and CMA-EnOpt starting from identical covariance matrices. For the small model, we achieve only slightly higher (0.7 to 1.8%) objective-function values and modest speedups with CMA-EnOpt compared with EnOpt. Significantly higher objective-function values (10%) are obtained for the modified Brugge model. The possibility to adapt the covariance matrix, and thus the perturbation size, during the optimization allows for the use of relatively large perturbations initially, for fast exploration of the control space, and small perturbations later, for more-precise gradients near the optimum. Moreover, the results demonstrate that a major benefit of CMA-EnOpt is its robustness with respect to the initial choice of the covariance matrix. A poor choice of the initial matrix can be detrimental to EnOpt, whereas the CMA-EnOpt performance is near-independent of the initial choice and produces higher objective-function values at no additional computational cost.

We consider the estimation of a Gaussian random vector x observed through a linear transformation H and corrupted by additive Gaussian noise with a known covariance matrix, where the covariance matrix of x is known to lie in a given region of uncertainty that is described using bounds on the eigenvalues and on the elements of the covariance matrix. Recently, two criteria for minimax estimation called difference regret (DR) and ratio regret (RR) were proposed and their closed form solutions were presented assuming that the eigenvalues of the covariance matrix of x are known to lie in a given region of uncertainty, and assuming that the matrices H T Cw -1 H and C x are jointly diagonalizable, where C w and C x denote the covariance matrices of the additive noise and of x respectively. In this work we present a new criterion for the minimax estimation problem which we call the generalized difference regret (GDR), and derive a new minimax estimator which is based on the GDR criterion where the region of uncertainty is defined not only using upper and lower bounds on the eigenvalues of the parameter's covariance matrix, but also using upper and lower bounds on the individual elements of the covariance matrix itself. Furthermore, the new estimator does not require the assumption of joint diagonalizability and it can be obtained efficiently using semidefinite programming. We also show that when the joint diagonalizability assumption holds and when there are only eigenvalue uncertainties, then the new estimator is identical to the difference regret estimator. The experimental results show that we can obtain improved mean squared error (MSE) results compared to the MMSE, DR, and RR estimators.

Sensitivity analysis of uncertain nonlinear systems plays an important role in analyzing chemical engineering process models. Even though a variety of different techniques exists for analyzing nonlinear systems, these methods often do not update the parameter distribution using available measurement data. This forms the motivation behind this work as a sensitivity analysis procedure is introduced that deals with nonlinear systems and uncertainty in model parameters, and also incorporates the use of available data. The technique computes two covariance matrices: a covariance matrix that describes the effect that changes of the parameters values, sampled from a prior distribution, have on the outputs, and a covariance matrix that determines the effect of available data on the posterior distribution of the parameters using a Markov chain Monte Carlo method. The information contained in these two covariance matrices is then simultaneously evaluated by balancing the two matrices. The results returned by this technique are the directions in parameter space that have the largest impact on the system behavior according to the given parameter uncertainty as well as the contribution of individual parameters to these important directions in parameter space. The presented technique is applied to three examples.

The rotated delta, which is argued to be a theoretically better-grounded distance measure, has failed to receive any empirical support for its superiority. This study revisits the rotated delta—which is more commonly known as the Mahalanobis distance in other areas—with two different covariance matrices that are estimated from training data. The first covariance matrix represents the between-author variability, and the second the within-author variability. A series of likelihood ratio-based authorship verification experiments was carried out with some different distance measures. The experiments made use of the documents arranged from a large database of text messages that allowed for a total of 2,160 same-author and 4,663,440 different-author comparisons. The Mahalanobis distance with the between-author covariance matrix performed far worse compared to the other distance measures, whereas the Mahalanobis distance with the within-author covariance matrix performed better than the other measures. However, superior performance relative to the cosine distance is subject to word lengths and/or the order of the feature vector. The result of follow-up experiments further illustrated that the covariance matrix representing the within-author variability needs to be trained using a good amount of data to perform better than the cosine distance: the higher the order of the vector, the more data are required for training. The quantitative results also infer that the two sources of variabilities—notably within- and between-author variabilities—are independent of each other to the extent that the latter cannot accurately approximate the former.