Random Vectors in High Dimensions

We study generalized bootstrap confidence regions for the mean of a random vector whose coordinates have an unknown dependency structure. The random vector is supposed to be either Gaussian or to have a symmetric and bounded distribution. The dimensionality of the vector can possibly be much larger than the number of observations and we focus on a nonasymptotic control of the confidence level, following ideas inspired by recent results in learning theory. We consider two approaches, the first based on a concentration principle (valid for a large class of resampling weights) and the second on a resampled quantile, specifically using Rademacher weights. Several intermediate results established in the approach based on concentration principles are of interest in their own right. We also discuss the question of accuracy when using Monte Carlo approximations of the resampled quantities.

A Monte Carlo method for calculating strength functions in many-fermion systems

Testing Independence for a Large Number of High-Dimensional Random Vectors

The NORTA method for multivariate generation is a fast general purpose method for generating samples of a random vector with given marginal distributions and given product-moment or rank correlation matrix. However, this method has been shown to fail to work for some feasible correlation matrices. (A matrix is feasible if there exists a random vector with the given marginal distributions and the matrix as the correlation matrix.) We investigate how this feasibility problem behaves as the dimension of the random vector is increased and find the problem to become acute rapidly. We also find that a modified NORTA procedure, augmented by a semidefinite program (SDP) that aims to generate a correlation matrix close to the desired one, performs well with increasing dimension.

Brain-inspired high-dimensional (HD) computing represents and manipulates data using very long, random vectors with dimensionality in the thousands. This representation provides great robustness for various classification tasks where classifiers operate at low signal-to-noise ratio (SNR) conditions. Similarly, hyperdimensional modulation (HDM) leverages the robustness of complex-valued HD representations to reliably transmit information over a wireless channel, achieving a similar SNR gain compared to state-of-the-art codes. Here, we first propose methods to improve HDM in two ways: (1) reducing the complexity of encoding and decoding operations by generating, manipulating, and transmitting bipolar or integer vectors instead of complex vectors; (2) increasing the SNR gain by 0.2 dB using a new soft-feedback decoder; it can also increase the additive superposition capacity of HD vectors up to 1.7times in noise-free cases. Secondly, we propose to combine encoding/decoding aspects of communication with classification into a single framework by relying on multifaceted HD representations. This leads to a near-channel classification (NCC) approach that avoids transformations between different representations and the overhead of multiple layers of encoding/decoding, hence reducing latency and complexity of a wireless smart distributed system while providing robustness against noise and interference from other nodes. We provide a use-case for wearable hand gesture recognition with 5 classes from 64 EMG sensors, where the encoded vectors are transmitted to a remote node for either performing NCC, or reconstruction of the encoded data. In NCC mode, the original classification accuracy of 94% is maintained, even in the channel at SNR of 0 dB, by transmitting 10,000-bit vectors. We remove the redundancy by reducing the vector dimensionality to 2048-bit that still exhibits a graceful degradation: less than 6% accuracy loss is occurred in the channel at − 5 dB, and with the interference from 6 nodes that simultaneously transmit their encoded vectors. In the reconstruction mode, it improves the mean-squared error by up to 20 dB, compared to standard decoding, when transmitting 2048-dimensional vectors.

Testing for mutual independence among several random vectors is a challenging problem, and in recent years, it has gained significant attention in statistics and machine learning literature. Most of the existing tests of independence deal with only two random vectors, and they do not have straightforward generalizations for testing mutual independence among more than two random vectors of arbitrary dimensions. On the other hand, there are various tests for mutual independence among several random variables, but these univariate tests do not have natural multivariate extensions. In this article, we propose two general recipes, one based on inter-point distances and the other based on linear projections, for multivariate extensions of these univariate tests. Under appropriate regularity conditions, these resulting tests turn out to be consistent whenever we have consistency for the corresponding univariate tests. We carry out extensive numerical studies to compare the empirical performance of these proposed methods with the state-of-the-art methods.

In statistical pattern recognition, it is important to estimate true distribution of patterns precisely to obtain high recognition accuracy. Normal mixtures are sometimes used for representing distributions. However, precise estimation of the parameters of normal mixtures requires a great number of sample patterns, especially for high dimensional vectors. For some pattern recognition problems, such as face recognition, very high dimensional feature vectors are necessary and there are always not enough training samples compared with the dimensionality. We present a method to estimate the distributions based on normal mixtures with small number of samples. The proposed algorithm is applied to face recognition problem which requires high dimensional feature vectors. Experimental results show the effectiveness of the proposed algorithm.

Copulas are used in finance and insurance for modeling stochastic dependency. They comprehend the entire dependence structure, not only the correlations. Here they are estimated from measured samples of random vectors. The copula and the marginal distributions of the vector elements define a multivariate distribution of the sample which can be used to generate random vectors with this distribution. This can be applied as well to time series. A programmed algorithm is proposed. It is fast and allows for random vectors with high dimension, for example 100.

We consider the problem of estimating sparse vectors from noisy linear measurements in the high dimensionality regime. For a fixed number k of nonzero entries, we study the fundamental relationship between two relevant quantities: the measurement rate, which characterizes the asymptotic behavior of the dimensions of the measurement matrix in terms of the ratio m/log n (with m being the number of measurements and n the dimension of the sparse vector), and the estimation mean square error. First, we use an information-theoretic approach to derive sufficient conditions on the measurement rate to reliably recover a part of the support set that represents a certain fraction of the total vector power. Second, we characterize the mean square error of an estimator that uses partial support set information. Using these two parts, we derive a tradeoff between the measurement rate and the mean-square error. This tradeoff is achievable using a two-step approach: first support set recovery, and then estimation of the active components. Finally, for both deterministic and random vectors, we perform a numerical evaluation to verify the advantages of the methods based on partial support set recovery.

We consider a phase retrieval problem, where we want to reconstruct a $n$-dimensional vector from its phaseless scalar products with $m$ sensing vectors, independently sampled from complex normal distributions. We show that, with a suitable initalization procedure, the classical algorithm of alternating projections succeeds with high probability when $m\geq Cn$, for some $C>0$. We conjecture that this result is still true when no special initialization procedure is used, and present numerical experiments that support this conjecture.

This paper derives central limit and bootstrap theorems for probabilities that sums of centered high-dimensional random vectors hit hyperrectangles and sparsely convex sets. Specifically, we derive Gaussian and bootstrap approximations for probabilities $\Pr(n^{-1/2}\sum_{i=1}^n X_i\in A)$ where $X_1,\dots,X_n$ are independent random vectors in $\mathbb{R}^p$ and $A$ is a hyperrectangle, or, more generally, a sparsely convex set, and show that the approximation error converges to zero even if $p=p_n\to \infty$ as $n \to \infty$ and $p \gg n$; in particular, $p$ can be as large as $O(e^{Cn^c})$ for some constants $c,C>0$. The result holds uniformly over all hyperrectangles, or more generally, sparsely convex sets, and does not require any restriction on the correlation structure among coordinates of $X_i$. Sparsely convex sets are sets that can be represented as intersections of many convex sets whose indicator functions depend only on a small subset of their arguments, with hyperrectangles being a special case.

Dependence within a high-dimensional profile of explanatory variables affects estimation and prediction performance of regression models. However, the strong belief that dependence should not be ignored, based on our well-proven knowledge of low-dimensional regression modeling, is not necessarily true in high dimension. To investigate this point, we introduce a new class of prediction scores defined as linear combinations of a same random vector, including the naive prediction score obtained when ignoring dependence and the Ordinary Least Squares (OLS) prediction score that, on the contrary, fully accounts for dependence by a preliminary whitening of the explanatory variables. Interestingly, the former class also contains Ridge and Partial Least Squares prediction scores, that both offer intermediate ways of dealing with dependence. Through a theoretical comparative study, it is first shown how the best handling of dependence should depend on the interplay between the structure of conditional dependence across explanatory variables and the pattern of the association signal. We also derive the closed form expression of the prediction score with best prediction performance within the proposed class, leading to an adaptive handling of dependence. Finally, it is demonstrated through simulation studies and using benchmark datasets that this prediction score outperforms existing methods in various settings. Supplementary materials for this article are available online.

The volume of random simplices from elliptical distributions in high dimension

In this dissertation we consider a sensor scheduling or resource management problem for a vector Poisson and Gaussian channels. The input is a binary random vector and the output is a set of conditionally independent Poisson or Gaussian random variables. The objective is to design a scaling matrix, which is a linear transformation whose purpose is to entangle the different inputs, under a total given energy/time constraint. The two metrics are adopted to quantify the performance of the designed scaling matrix: mutual information and Bayesian inference. In other words, it is an experimental design problem where the objective is to glean the information about the binary inputs and perform the classification of the input random vector in a fixed time-resource, that is transmitted through a vector Poisson and Gaussian channel, based on the output observations. No optimal solution is claimed in this dissertation for the above problem for either of the Poisson or Gaussian channels; from either of the two perspectives: mutual information or Bayes detection. However, time-symmetry does exist in the said problem. It is further noted that the problem is concave in its domain (i.e. sensing times) from mutual information criterion; and this is based on the observations in the computational results. If this concavity does exist in the problem then together with the time-symmetry result; it can be deduced that the optimal solution has a symmetry too; and that would reduce the exponentially rising dimensionality of the search-space to the linear one (w.r.t dimension of the input random vector). However, concavity of the objective function in the Bayes framework does not exist. Further, it is noted that the classification criterion in the above two channels; and mutual information criterion do not generally lead to the same solution when subjected to the same fixed time constraint and model parameters. It is also noted that the combinatorial explosion is inevitable, that occurs while addressing the problem through computational means, even with exploiting the inherent time-symmetry and the concavity in the objective. This curse of the dimensionality is the main obstacle in exploring the problem for targets greater than four (i.e. for dimension of the input vector greater than 4).

A robust predictive approach for canonical correlation analysis