Eigenvector overlaps of random covariance matrices and their submatrices

This dissertation is concerned with the task of efficiently and accurately tracking the singular values and left singular vectors of a rapidly changing dominant column space of a matrix, in which a column of the matrix is replaced each time step. As part of this task, the dimension of this dominant subspace is determined automatically for each time step. Two methods for determining the singular values and left singular vectors of this dominant columnspace are presented. The first method, which is an exact method, will update all of the singular values and left singular vectors using the rank-two secular function. The derivation of this function, and its properties, are original contributions of this dissertation. This method requires a single O(n3) matrix product to rotate the singular vectors using direct multiplication, and all other computation is O(n2). The second method, the Improved Fast Adaptive Subspace Tracking (IFAST) method, will give accurate approximations of the r largest singular values and corresponding left singular vectors in O( nr2) time. An accuracy analysis of the approximation error of the second method, using the rank-two secular function from the first method, is presented. The block Hankel matrix structure is presented, which can give improved SNR for exponential signals in sensor arrays, at the expense of beam-width. The two rank determination methods of Shah and Tufts, where one is for time-series (Hankel) data and the other is for unstructured sensor array data, are combined into a single general method which works with unstructured, Hankel, and block Hankel matrices. An efficient way to calculate the thresholds used by this method is presented, allowing use in real-time applications.

The aim of a correspondence analysis is the graphical representation of the categories of variables in one frame of reference. This visualization is possible due to the decomposition of the basic matrix with the use of Singular Value Decomposition (SVD). There are three matrices used in the process of decomposition: right singular vectors, left singular vectors, and a singular value diagonal matrix. The aim of this paper is to compare four different approaches and algorithms of SVD methods used in a correspondence analysis. In the literature, four approaches are known to singular value decomposition, defined by: R.A. Fisher (1940), M.J. Greenacre (1984), E.B. Anderson (1991), and J.D. Jobson (1992). Those computational procedures will be presented and compared in this paper. Also, methods of determining the coordinates of the category column and line matrix, as well as the values of inertia will be defined for these approaches. A key problem is to compare the well-known approaches, since in the literature only one approach ‒ proposed by Greenacre ‒ is used for singular value decomposition. The reason of the superiority of this algorithm over the others may be the simplicity and ease of the mathematical calculations. Greenacre’s algorithm is also used in R statistical software, making its availability and popularity growing, however, other algorithms are worth presenting and focusing on.

The problem of blind despreading of short-code direct sequence spread-spectrum signal in asynchronous or synchronous system is considered. A novel blind estimation algorithm for time offset, spreading and information sequences is presented in this paper. Firstly, the received signal is divided into two-information-period-length temporal vectors overlapped by one-information-period and accumulates these vectors one by one to form the accumulated matrix. The algorithm exploits an unbiased estimation of user’s time offset to realize synchronization blindly, and an operation of singular value decomposition is applied to the intercepted matrix, which is intercepted from the accumulated matrix according to user’s time offset. Lastly, the spreading sequence is recovered from a right singular vector blindly; therefore, the shortcomings of traditional algorithms, such as phase ambiguity when make use of two vectors to form the spreading sequence, are avoided. And the information sequence is then recovered blindly from the left singular vector. Simulation results show that the new method is capable of retrieving time offset, the spreading and information sequences with higher accuracy than previous methods.

The compressed beamforming weights feedback is used in the precoded MIMO-OFDM system, e.g., IEEE 802.11n/ac WLAN, to reduce the amount of feedback information. The precoded sub-system at the receiver of this system needs to perform singular value decomposition (SVD) on the estimated channel matrices at all sub-carriers. The computed angles of the right singular vectors are to be feedback to the transmitter. Also the computed singular values and left singular vectors are to be used for designing the optimal detector. Under the exemplary scenario of four antennas at the access point and two antennas at the station, our designed architectures for the precoding sub-system can provide SVD throughput rates 16.8 and 50.4 M matrices/sec for the uplink and downlink transmissions, respectively.

The FORTRAN codes in this Chapter address the question of computing distinct singular values and corresponding left and right singular vectors of real rectangular matrices, using a single-vector Lanczos procedure. For a given real rectangular ℓ × n matrix A, these codes compute nonnegative scalars σ and corresponding real vectors x ≠ 0 and y ≠ 0 such that EquationSource$$ % MathType!MTEF!2!1!+- % feaagCart1ev2aaatCvAUfeBSjuyZL2yd9gzLbvyNv2CaerbuLwBLn % hiov2DGi1BTfMBaeXatLxBI9gBaerbd9wDYLwzYbItLDharqqtubsr % 4rNCHbGeaGqiVu0Je9sqqrpepC0xbbL8F4rqqrFfpeea0xe9Lq-Jc9 % vqaqpepm0xbba9pwe9Q8fs0-yqaqpepae9pg0FirpepeKkFr0xfr-x % fr-xb9adbaqaaeGaciGaaiaabeqaamaabaabaaGcbaaeaaaaaaaaa8 % qacaWGbbGaamiEaiabg2da9iabeo8aZjaadMhaieaacaWFGaGaamyy % aiaad6gacaWGKbGaa8hiaiaadgeapaWaaWbaaSqabeaapeGaamivaa % aakiaadMhacqGH9aqpcqaHdpWCcaWG4bGaaiOlaaaa!4716! $$Ax = \sigma y and {A^T}y = \sigma x.$$$$ ((6.1.1)) Every real rectangular ℓxn matrix, where ℓ ≥ n, has a singular value decomposition, EquationSource$$ % MathType!MTEF!2!1!+- % feaagCart1ev2aaatCvAUfeBSjuyZL2yd9gzLbvyNv2CaerbuLwBLn % hiov2DGi1BTfMBaeXatLxBI9gBaerbd9wDYLwzYbItLDharqqtubsr % 4rNCHbGeaGqiVu0Je9sqqrpepC0xbbL8F4rqqrFfpeea0xe9Lq-Jc9 % vqaqpepm0xbba9pwe9Q8fs0-yqaqpepae9pg0FirpepeKkFr0xfr-x % fr-xb9adbaqaaeGaciGaaiaabeqaamaabaabaaGcbaaeaaaaaaaaa8 % qacaWGbbGaeyypa0Jaamywaiabfo6atjaadIfapaWaaWbaaSqabeaa % peGaamivaaaaieaak8aacaWFGaGaam4DaiaadMgacaWG0bGaamiAai % aa-bcapeGaamiwaiabg2da9iaadIfapaWaaWbaaSqabeaapeGaamiv % aaaak8aacaWGybGaeyypa0JaamysaiaacYcacaWFGaGaamywamaaCa % aaleqabaGaamivaaaakiaadMfacqGH9aqpcaWGjbGaa8hiaiaa-fga % caWFUbGaa8hzaiaa-bcacqqHJoWucaWF9aWaamWaaeaafaqabeGaba % aabaGaeu4Odm1aaSbaaSqaaiaadMeaaeqaaaGcbaGaaGimaaaaaiaa % wUfacaGLDbaaaaa!59A3! $$A = Y\Sigma {X^T} with X = {X^T}X = I, {Y^T}Y = I and \Sigma = \left[ {\matrix {{\Sigma _I}} \\ 0 \\ \endmatrix } \right]$$$$ ((6.1.2)) where Σ is ℓ × n and = diag {σ1,..., σn} with σi, 1 ≤ i ≤ n, the singular values of A. X is a n × n orthogonal matrix, Y is a ℓ × ℓ orthogonal matrix, and the columns of X and of Y are respectively, right and left singular vectors of A. There are many applications for this type of decomposition. Singular values and vectors are discussed in detail for example in Stewart [1973].KeywordsError EstimateSingular VectorMain ProgramVector ComputationReal Symmetric MatrixThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

The problem of simultaneous column and row subset selection is addressed in this paper. The column space and row space of a matrix are spanned by its left and right singular vectors, respectively. However, the singular vectors are not within actual columns/rows of the matrix. In this paper, an iterative approach is proposed to capture the most structural information of columns/rows via selecting a subset of actual columns/rows. This algorithm is referred to as two-way spectrum pursuit (TWSP) which provides us with an efficient solution for the CUR matrix decomposition. TWSP is applicable in a wide range of applications since it enjoys a linear complexity w.r.t. number of original columns/rows. We demonstrated the application of TWSP for joint channel and sensor selection in cognitive radio networks and efficient supervised data reduction.

Iterative computation of the smallest singular value and the corresponding singular vectors of a matrix

The most reliable method for solving linear equations of the least squares principle, which can be used to solve incorrect geodetic problems, is based on matrix factorization, which is called a singular expansion. Some other methods require less machine time and memory. However, they are less effective in taking into account the errors of the source information, rounding errors, and linear dependence. The methodology of such research is that for any matrix A and any two orthogonal matrices U and V, there is a matrix , which is determined as T U AV  . The idea of a singular decomposition is that by choosing the right matrices U and V, you can convert most elements of the matrix  to zero and make it diagonal with non-negative elements. The novelty and relevance of scientific solutions lie in the feasibility of using a singular decomposition of the matrix to obtain linear equations of the least squares method, which can be used to solve incorrect geodetic problems. The purpose of scientific research is to obtain a stable solution of parametric equations of corrections to the results of mea- surements in incorrect geodetic problems. Based on the performed research on the application of the singular decomposition method for solving incorrect geodetic problems, we can summarize the following results. A singular expansion of a real matrix A is any factorization T A U W  of a matrix with orthogonal columns U, an orthogonal matrix W , and a diagonal matrix , the elements of which are called singular numbers of the matrix A, and the columns of matrices U and W — left and right singular vectors. If the matrix A has a full rank, then its solution will be unique and stable, which can be obtained by different methods. However, the method of singular decomposition, in contrast to other methods, makes it possible to solve problems with incomplete rank. Research shows that the method of solving normal equations by sequential exclusion of unknowns (Gaussian method), which is quite common in geod- esy, does not provide stable solutions for poorly conditioned or incorrect geodetic problems. Therefore, in the case of unstable systems of equations, it is proposed to use the method of singular matrix decomposition, which in computational mathematics is called SVD. The SVD singular decomposition method makes it possible to obtain stable solutions to both stable and by nature unstable problems. This possibility to solve incorrect geodetic problems is associated with the application of some limit , the choice of which can be made by the relative errors of the matrix of coefficients of parametric equations of corrections A and the vector of results of geodetic measurements L . Moreover, the solution of the system of normal equations obtained by the SVD method will have the shortest length. Thus, applying the apparatus of the singular decomposition of the matrix of coefficients of parametric equations of correc- tions to the results of geodetic measurements, we obtained new formulas for estimating the accuracy of the least squares method in solving incorrect geodetic problems. The derived formulas have a compact form and allow the easy calculation of elements  and XQ estimates of accuracy, almost ignoring the complex procedure of rotation of the matrix of coefficients of normal equations.

The new common approach for integrity checking of digital images is developed. The new features of formal parameters defining image are revealed, theoretically grounded and practically tested. The characteristics of the mutual arrangement of left and right singular vectors corresponding to the largest singular value of the image’s matrix (block of matrix) and the vector composed of singular numbers is obtained. Formal parameters are obtained using normal singular decomposition of matrix (block of matrix) which is uniquely determined. It is shown that for most blocks of original image (no matter lossy or lossless) the angle between the left (right) mentioned singular vector and vector composed of singular numbers is defined by the angle between the n-optimal vector and the vector of standard basis of the range corresponding dimension. It is shown that the determined feature brakes for the mentioned formal parameters in a non-original image. This shows the integrity violation of the image, i.e. the existence of the additional information embedded using steganography algorithms. So this can be used as a basis for development of new universal steganography methods and algorithms, and one example of the realization is proposed. The efficiency of the proposed algorithm won’t depend on the details of steganography method used for embedding. All the obtained results can be easily adapted for the digital video and audio analysis.

In this paper the static voltage stability index has been identified by minimum singular value of the power flow jacobian matrix. The system security in power system models by the scalar magnitude of a voltage stability index may be a very difficult task. When we are taking account of reactive power generation limits it is very difficult to predict stability index at voltage collapse points. Here a quick method is used to calculate the minimum singular value and the corresponding left and right singular vectors are presented. This developed algorithm is very useful in online because it required small amount of computation time. For different patterns of load and generation are increases, to determine voltage collapse point. For solving the maximum loading problem an optimal power flow approach was used with these points the MSV is calculated and used for training and testing the Neural Network.

We describe a procedure for determining a few of the largest singular values of a large sparse matrix. The method by Golub and Kent which uses the method of modified moments for estimating the eigenvalues of operators used in iterative methods for the solution of linear systems of equations is appropriately modified in order to generate a sequence of bidiagonal matrices whose singular values approximate those of the original sparse matrix. A simple Lanczos recursion is proposed for determining the corresponding left and right singular vectors. The potential asynchronous computation of the bidiagonal matrices using modified moments with the iterations of an adapted Chebyshev semi-iterative (CSI) method is an attractive feature for parallel computers. Comparisons in efficiency and accuracy with an appropriate Lanczos algorithm (with selective re-orthogonalization) are presented on large sparse (rectangular) matrices arising from applications such as information retrieval and seismic reflection tomography. This procedure is essentially motivated by the theory of moments and Gauss quadrature.

An application of Singular Spectrum Analysis(SSA) Method, based on a new elaborated tensorial approach of computation of singular values and left and right singular vectors of arbitrary non-square matrices, for time series is presented. All necessary calculations of singular values and both types (left and right) singular vectors are performed on the base of elaborated tensorial approach. It is showed that non parametric SSA can be efficiently used as a universal filter to separate Low and High frequencies components in long signals and time series.

In this paper, a novel pair-matching algorithm is developed for two-dimensional (2-D) direction-of-arrival (DOA) estimation. In the proposed algorithm, firstly we take the singular value decomposition (SVD) of the cross-correlation matrix of two uniform linear arrays (ULAs). Next we extract the square root of the singular values. Lastly we multiply the left singular vectors and right singular vectors by the square root of the corresponding singular values respectively and obtain the new singular vectors which are called the modified singular vectors. After this processing, there exists a special relation between the transition matrices from the modified singular vectors to the steering vectors and the autocorrelation matrix of the signals. Exploiting this relation, the pair-matching procedure of the azimuth and elevation angles is carried out effectively. The simulation results are presented to validate the effectiveness of the proposed algorithm.

The design of the precoder the maximizes the mutual information in linear vector Gaussian channels with an arbitrary input distribution is studied. Precisely, the precoder optimal left singular vectors and singular values are derived. The characterization of the right singular vectors is left, in general, as an open problem whose computational complexity is then studied in three cases: Gaussian signaling, low SNR, and high SNR. For the Gaussian signaling case and the low SNR regime, the dependence of the mutual information on the right singular vectors vanishes, making the optimal precoder design problem easy to solve. In the high SNR regime, however, the dependence on the right singular vectors cannot be avoided and we show the difficulty of computing the optimal precoder through an NPhardness analysis. © 2009 IEEE.

We study the eigenvalue distribution of random matrices pertinent to the analysis of deep neural networks. The matrices resemble the product of the sample covariance matrices, however, an important difference is that the analog of the population covariance matrix is now a function of random data matrices (synaptic weight matrices in the deep neural network terminology). The problem has been treated in recent work [J. Pennington, S. Schoenholz and S. Ganguli, The emergence of spectral universality in deep networks, Proc. Mach. Learn. Res. 84 (2018) 1924–1932, arXiv:1802.09979] by using the techniques of free probability theory. Since, however, free probability theory deals with population covariance matrices which are independent of the data matrices, its applicability in this case has to be justified. The justification has been given in [L. Pastur, On random matrices arising in deep neural networks: Gaussian case, Pure Appl. Funct. Anal. (2020), in press, arXiv:2001.06188] for Gaussian data matrices with independent entries, a standard analytical model of free probability, by using a version of the techniques of random matrix theory. In this paper, we use another version of the techniques to extend the results of [L. Pastur, On random matrices arising in deep neural networks: Gaussian case, Pure Appl. Funct. Anal. (2020), in press, arXiv:2001.06188] to the case where the entries of the data matrices are just independent identically distributed random variables with zero mean and finite fourth moment. This, in particular, justifies the mean field approximation in the infinite width limit for the deep untrained neural networks and the property of the macroscopic universality of random matrix theory in this case.